first of all, when i began i was quite skeptical on commercial AI.

this exercise has only made me more skeptical, for a few reasons:

first: you actually can hit the "good enough" point for text prediction with very little data. 80GB of low-quality (but ethically sourced from $HOME/logs) training data yielded a bot that can compose english and french prose reasonably well. if i additionally trained it on a creative commons licensed source like a wikipedia dump, it would probably be *way* more than enough. i don't have the compute power to do that though.

second: reasoning models seem to largely be "mixture of experts" which are just more LLMs bolted on to each other. there's some cool consensus stuff going on, but that's all there is. this could possibly be considered a form of "thinking" in the framing of minsky's society of mind, but i don't think there is enough here that i would want to invest in companies doing this long term.

third: from my own experiences teaching my LLM how to use tools, i can tell you that claude code and openai codex are just chatbots with a really well-written system prompt backed by a "mixture of experts" model. it is like that one scene where neo unlocks god mode in the matrix, i see how all this bullshit works now. (there is still a lot i do not know about the specifics, but i'm a person who works on the fuzzy side of things so it does not matter).

fourth: i built my own LLM with a threadripper, some IRC logs gathered from various hard drives, a $10k GPU, a look at the qwen3 training scripts (i have Opinions on py3-transformers) and few days of training. it is pretty capable of generating plausible text. what is the big intellectual property asset that OpenAI has that the little guys can't duplicate? if i can do it in my condo, a startup can certainly compete with OpenAI.

given these things, I really just don't understand how it is justifiable for all of this AI stuff to be some double-digit % of global GDP.

if anything, i just have stronger conviction in that now.

@ariadne To me it's a question of sufficient output quality, the strongest models available just barely function enough to do a little bit of general purpose instructed information processing unreliably. That will improve over time but the current stuff is very early.

The reason I'm a bit skeptical of a proliferation of weekend-sized models is that that size sacrifices the key ingredient enabling the whole LLM craze: the magical-looking ability to run plain language instructions.

@mirth i mean, i don't think that necessarily holds *if* you have the ability to build whatever you need with legos.

in many cases simply translating natural language to a specification for an expert system is enough

@ariadne
Yeah, one thing I've wondered is how much simpler a system that, instead of processing code, took the plain english "refactor this to blah blah" and just processed the language and figured out what to tell the IDE and etc for everything else, could be.

Run a calculator instead of being one - and you have a much simpler problem to solve.

Could the reliability and ethical problems all be solved -- maybe, i dunno, but - yet another case of "tech could be cool if the harmful parts go away..."

@mirth

@pixx @mirth i think small LLMs do not really have an ethical problem: i trained a 1.3B parameter LLM off of my own personal data in my apartment by simply being patient enough to wait. no copyright violations, no boiling oceans, just patience and a professional workstation GPU with 96GB RAM.

the ethical problem is with the Big AI companies who feel that the only path forward is to make bigger and bigger and bigger monolithic prediction models rather than properly engineer the damn thing.

that same ethical problem is driving the hoarding, because companies are buying the hardware to prevent their competitors from having it IMO.