now that i am... writing my own agentic LLM framework thing... because if you're going to have a shitposting IRC bot you may as well go completely overkill, i have Opinions on the state of the world.
openclaw, especially, seems to be hot garbage, actually, because i was able to teach my LLM (which i trained from scratch on the highest quality artisanal IRC logs, 2003 to present, so i can assure you it is not a very good LLM) to use tools in the context of my own framework quite easily.
first of all, when i began i was quite skeptical on commercial AI.
this exercise has only made me more skeptical, for a few reasons:
first: you actually can hit the "good enough" point for text prediction with very little data. 80GB of low-quality (but ethically sourced from $HOME/logs) training data yielded a bot that can compose english and french prose reasonably well. if i additionally trained it on a creative commons licensed source like a wikipedia dump, it would probably be *way* more than enough. i don't have the compute power to do that though.
second: reasoning models seem to largely be "mixture of experts" which are just more LLMs bolted on to each other. there's some cool consensus stuff going on, but that's all there is. this could possibly be considered a form of "thinking" in the framing of minsky's society of mind, but i don't think there is enough here that i would want to invest in companies doing this long term.
third: from my own experiences teaching my LLM how to use tools, i can tell you that claude code and openai codex are just chatbots with a really well-written system prompt backed by a "mixture of experts" model. it is like that one scene where neo unlocks god mode in the matrix, i see how all this bullshit works now. (there is still a lot i do not know about the specifics, but i'm a person who works on the fuzzy side of things so it does not matter).
fourth: i built my own LLM with a threadripper, some IRC logs gathered from various hard drives, a $10k GPU, a look at the qwen3 training scripts (i have Opinions on py3-transformers) and few days of training. it is pretty capable of generating plausible text. what is the big intellectual property asset that OpenAI has that the little guys can't duplicate? if i can do it in my condo, a startup can certainly compete with OpenAI.
given these things, I really just don't understand how it is justifiable for all of this AI stuff to be some double-digit % of global GDP.
if anything, i just have stronger conviction in that now.
@ariadne To me it's a question of sufficient output quality, the strongest models available just barely function enough to do a little bit of general purpose instructed information processing unreliably. That will improve over time but the current stuff is very early.
The reason I'm a bit skeptical of a proliferation of weekend-sized models is that that size sacrifices the key ingredient enabling the whole LLM craze: the magical-looking ability to run plain language instructions.
@mirth i mean, i don't think that necessarily holds *if* you have the ability to build whatever you need with legos.
in many cases simply translating natural language to a specification for an expert system is enough
@mirth back in the earlier AI wars, these were called "expert systems"
my idea is basically SLMs for I/O with other small models and tools governed by a user-generated expert system
@ariadne @mirth sorry to reply to you in separate threads but I missed these original posts.
You are correct about this. I worked in a tertiary way on something with formal verification and some other sorcery involved doing pretty much exactly this pre-public LLM but way after expert systems (which were before my time).
Thereβs fruit there that pays off, they sold the implementation.
@ariadne
Yeah, one thing I've wondered is how much simpler a system that, instead of processing code, took the plain english "refactor this to blah blah" and just processed the language and figured out what to tell the IDE and etc for everything else, could be.
Run a calculator instead of being one - and you have a much simpler problem to solve.
Could the reliability and ethical problems all be solved -- maybe, i dunno, but - yet another case of "tech could be cool if the harmful parts go away..."
@pixx @mirth i think small LLMs do not really have an ethical problem: i trained a 1.3B parameter LLM off of my own personal data in my apartment by simply being patient enough to wait. no copyright violations, no boiling oceans, just patience and a professional workstation GPU with 96GB RAM.
the ethical problem is with the Big AI companies who feel that the only path forward is to make bigger and bigger and bigger monolithic prediction models rather than properly engineer the damn thing.
that same ethical problem is driving the hoarding, because companies are buying the hardware to prevent their competitors from having it IMO.
@pixx @mirth @alwayscurious and it being not copyrightable is a problem because not all jurisdictions have "public domain".
and "public domain" is also a risk to OSS licensing.
yes, but i think natural language an an *interface* is still useful, and so SLMs are useful here because you can do things like
"please turn off the lights" --> {"action": "lighting-control", "state": "off"}
and I think weekend-sized models are perfectly fine for that.
there are of course other paths for that which don't require models, but using a model to process natural language and translate the intent to structured data seems like an obvious path to ensuring consistency across different languages
verses say, manually looking for specific keywords in the text to infer intent, but that requires maintaining large sets of keywords and so on and so forth and it turns into a nightmare.
that a model can guess what the intent is and represent it as structured text with a confidence score is useful.
@ariadne @mirth Call me old-fashioned, but I don't like to talk to my house. I'm lazy and I like to control the lights from the sofa, that's why I've got a few remote controlled switches. A few more lights are controlled by a Raspberry Pi via GPIO, and I use simple command line tools to control those, I wrote them in Python without any LLM, and I access them from any device on the LAN via SSH.
led-bar-rgb 192 128 96
...and the LED bar switches to a nice pastel orange for a nice summer sunset feeling.
*you* do not, but many people i know, including myself, want a libre voice assistant.
a voice assistant requires the ability to process arbitrary natural language and make a reasonable guess as to what the user wants.
hence the need for a model.
@ariadne @pinskia @mirth
What they are doing is forcing competitors to do more with less. Smaller models with a clever architecture, not huge monoliths trained by brute force. Might come back to bite them sooner or later.
I'd like to see more hybrid models, where the LLM largely sticks to being the language module, and other models (possibly not even NN) specialize in other functions.
@jannem @pinskia @mirth yes, this is what i eventually want to build. a set of libre building blocks for building ethical, libre and personal agentic systems that are self-contained.
the shit Big AI is doing is not interesting to me, but SLMs and other specialized neural models legitimately provide a useful set of tools to have in the toolbox.
today, however, I just want to prove the ideas out by shitposting in IRC ;)
@ariadne A shitpost bot trained on IRC logs?
Holy fucking shit you found a valid use for "AI".
@dvshkn i generated a bunch of examples of valid and invalid JSON document fragments and then prompted it with "reply in JSON" and then a spec on what it can do.
the hardest thing has been convincing it to shut the fuck up actually.
L1Q πΊπ¦
Ariadne Conill π°
gaytabase
ΗΚΙp
mirth
Allie!
pixx
Demi Marie Obenour
Erin π½β¨
Else, Someone
Lord Caramac the Clueless, KSC
yak
LisPi
Wouter Verhelst
Janne Moren
Thom, exceedingly pure
alys
Dan Ports
Henri
beaiouns 