Matthew MaybeLast year the Swiss AI initiative released Apertus, a huge gift to those of us still wishing for a version of LLMs that isn't rooted in theft. It joined a handful of base models able to claim training on permissively licensed data alone. There were even instruct variants of both the full 70B model and an 8B version you can run locally. But most folks who would do that shrugged it off, since it hadn't yet been fine-tuned for reasoning and tool use. I now think I may have closed those gaps. (🧵)
At first, I tried the straightforward approach, and fine-tuned some LoRA adapters for the Apertus 8B variant on popular tool use and reasoning datasets. The results were pretty meh, not competitive. I had almost given up until Claude Code was leaked. It then became clear to me that the emperor truly had no clothes and if the king of agentic LLMs was doing tool use and reasoning in different ways in different places, there isn't actually a right way to build those features! (2/🧵)
One of the criticisms of Claude Code was that it showed signs of struggling to produce valid JSON, and wasted inference brute-forcing that requirement to be met. Well, I knew that StarCoder is good at producing JSON, and can be taught to do a lot else with in-context learning. Could it serve as the tool use decision-maker? Yes, quite capably, as it turns out. (3/🧵)
Next up, I tried to hand off "reasoning" to the 70B version of Apertus (hosted on the Public AI inference utility), prompting it to produce the familiar "inner monologue" that models like DeepSeek-R1 exposed to users. Again, imitating their approach produces pretty "meh" results, not worth sharing. What works is what I call a "society of mind" approach, letting the 8B model talk to its big brother, the 70B version, with slightly different prompts. This is shockingly effective. (4/🧵)
I'm not sharing eval metrics because I don't have any, not because they're bad. I only have vibes, but part of those vibes is knowing that the thing which is helping me automate and organize stuff isn't based on theft. It's also running, largely, on my workstation at home, powered by a solar co-op, not transmitting private info to a power-hungry data center. If I'm getting 90% of the productivity benefit without compromising on those ideals, I'm fine with the missing 10%. (5/🧵)
My question to Mastodon is: does anyone care? Or have we all sorted ourselves into anti-genAI and pro-genAI absolutists with no room for discussion about right and wrong ways to implement such systems? When I was working on AI detection, I was shocked by how many AI critics scorned my efforts because I trained an image classification model--a kind of AI--to catch generative AI output. It wouldn't surprise me if the set of people interested in my "middle way" approach is small. (6/🧵)
If there's interest, I'll share my little ChatGPT/Claude/Gemini/DeepSeek clone that wasn't trained on stolen data and doesn't require massive energy usage. Honestly, it was largely vibe-coded anyway (yes I used Claude to build its own replacement). But if nobody cares, I'll just keep using it for the things I'm using it for, and skip the emotional labor of cleaning it up and putting it out there to be attacked from all sides. At least I won't feel quite so guilty about using AI at all. (end 🧵)
udo m. rader ☕ 🇪🇺 🇺🇦 🐧@matthewmaybe can only talk for myself, but this sounds very interesting. So if you can, I'd be more than interested and compare it to my own local way of doing things (currently mostly based on Qwen3.5)