Troed SångbergMar 7

Yeah, LLM coding tools can help in reverse engineering completely novel and proprietary systems.

I get the whole anti-AI stuff prevalent on Mastodon, I really do. The code quality when using them for development is really horrible, while you'll get something that fulfills the requirements the tech debt - and security holes - is through the roof.

But.

There's actual gains to be had from using them as well. I just tried using Devstral 2 to analyse some flash dumps I'm definitely sure has not been in any training data ever and yeah, that saved me time. The analysis looks exactly like what I would have come up with myself after about 10x the time the model needed.

If you still believe LLMs "can only repeat what's in their training data" you're simply wrong.

Show threadFrançois GaleaMar 8
@troed Not sure anyone seriously argued LLMs "can only repeat what's in their training data".
The "stochastic parrot" metaphor introduced by Bender et al. is not about repeating training data, but producing output based on a Markov model that results from training data. This does not prevent the production of seemingly "new" content, but that content still does come exclusively from training data.
Show threadTroed SångbergMar 8

@zerkman Oh there are many making that exact claim :)

Humans also produce output that comes exclusively from our training data. That doesn't seem to be a limitation since inferences still can be drawn in completely novel ways.

Show threadFrançois GaleaMar 8
@troed sure but our "training data" is far from being text only!
Also we don't need millions of examples to understand basic concepts.
Immediately getting the key ideas in every new piece of information is something we are good at. We can produce whole new concepts from simple initial ideas.
In the opposite, LLM word-salad mixing thousands of different sources is just obfuscated plagiarism.
Show threadTroed SångbergMar 8

@zerkman Oh we're not LLMs - agree. I think there are are far more similarities than the anti-AI crowd are able to internalize though.

As far as science is concerned, humans are also large neural networks producing output from input.

Show threadDonal FellowsMar 8
@troed @zerkman LLMs approximate (poorly, inefficiently, without live feedback) part of what the brain does (the bits that do associative recall principally). Much of the rest of the brain is doing signal processing and motor control (gotta control that meat body), but some is doing much higher order processing, and LLMs don't do any of that.
Show threadTroed SångbergMar 8

@dkf

Absolutely - the current "train once" paradigm regardless of long contexts will never approach what human brains do.

That wasn't the point.

@zerkman

Show threadDonal FellowsMar 8

@troed Not claiming it was the point. The real point is that LLMs are a partial model, and one of the things that's missing is the "decide which conflicting thing to choose part". Currently that's bolted on from outside (the probabilistic model to choose the next word from the distribution) and that's an inferior technique.

I think that part's also crucial (in mammals at least) for developing a sense of self. No proof.

Show threadTroed SångbergMar 8

@dkf Again, agree. I believe the level of "intelligence" in animals depends on the number of if-then simulations run by brains to anticipate and effect desired outcomes.

(My main influences are Susan Blackmore and Douglas Hoftstadter)

Show threadDonal Fellows

@troed It's not so simple. Much projection of the future is LLM-style, pattern-matched against experience. But that produces conflicts, between memories and with what is sensed. The critical part is the "decide which to believe" circuits, centred on cortical pyramidal cells (which have the right behaviour in their dendritic trees).

I think that, in a recurrent loop (Hofstadter-style), induces the sense of self.

Mar 8 at 12:31pmWeb
Show threadDonal FellowsMar 8
@troed But it's all really complicated. Everything about the brain is dynamic and overlapping. We compute with time and timing patterns, not value, so that's another way in which LLMs aren't the real thing.
Show threadTroed SångbergMar 8

@dkf I doubt we're going to even try to invent "brains". Evolution has surely resulted in our brains being optimized for low energy usage, which would explain why our memory is so "bad" compared to what can easily be accomplished by computers.

Without the same external pressure we're more likely going to try to make "better" brains, which then won't be as "fuzzy" and so they're never really going to be fully alike in how they work.

As you, I also believe an introspection loop to be the source of self. It's interesting that LLMs went that way (the "thinking"/"reasoning" models) quickly. I doubt it actually accomplishes much when feedback doesn't change the actual training weights though.