AionoMar 17
tante
One of the weirder things in "AI" is the whole "skills" thing. It sounds as if those were somehow little programs or tools integrated into your agents but they are just markdown files with strings that you pull into your prompt to hope the slop machine does better. Crazy shit.
Mar 12 at 9:27pmWeb
Show threadSnoopJMar 12
@tante tinker-toy kit for fanfic
Show threadKen MilmoreMar 12
@tante Marketing.
Show threadalarmed feet operative: kimothy siddonMar 12
@tante it's so funny right? here's my "dotfiles" of "skills" in which i tell the robot what it's supposedly capable of doing (and hope it does it correctly)
Show threadOliMar 12
@tante Thanks for fighting. I'm a bit resigned at the moment, as I get the impression that the crazier all this AI stuff gets, the more it attracts the mainstream.
Show threadOliMar 12
@tante The crazier it gets, the more people talk about it, the more it becomes normalised, and people think they will be left behind if they don't use it.
Show threadReilly Spitzfaden (they/them)Mar 12
@tante I saw some "skills" once that repeat an instruction a few times, basically begging the machine not to modify files.
Show threadMuss das sein.Mar 12
@tante yep. I first had the theory those skill would be smaller models. But no.
Show threadPaul 夏央 KishimotoMar 12
@tante not really different from a prayer wheel or phylactery: bits of scripture in a container believed to bring good fortune.
Show threadJPMar 12
@tante "a skill is a thing, a piece of data that can be ingested and copied endlessly to produce monetizable outputs" vs "a skill is a thing a person can learn to do, can teach others, can practice and improve at their entire lives, a collective process deepening in effectiveness and understanding over generations"
Show threademenelMar 12
@jplebreton @tante i think that’s what they call “gatekeeping” lol 🤮
Show threadDavid MankinsMar 13
@tante skills are just macros.
Show threadDavid MankinsMar 13

@tante as I noted, “skills are just macros”, but I guess they’re a little cleverer than that.

They work-around limited context space by being part of your prompt that only gets loaded if the skill-triggering metadata indicates the “skill” may be relevant to the problem at hand.

Show threadjwzMar 13
@tante It's like when I slap the side of my computer and yell GO FASTER
Show threadFrederic ThevenetMar 13
@jwz @tante I strongly believe it is a universal rule of engineering, probably going back to the ancient Egyptians or even homo habilis, that the only acceptable form of verbal communication with a piece of hardware is yelling profanity at it when it fails to perform adequately.
Show threadFrederic ThevenetMar 13
@jwz @tante e.g. "scrape faster you goddam mammoth-humping biface!"
Show threadtanteMar 13
@jwz I mean we all do that but I wouldn't claim that that's a billion dollar business ;)
Show threadGlyphMar 13

@tante I know very little about LLMs and even less about product architecture at these companies but that has not stopped me from making devastatingly accurate predictions about a bunch of things, so here's my guess:

the major challenge of current models is context rot, which is obvious and catastrophic when applying them to any real-world problems. "skills" are one of a set of tools to play a little shell game to shorten the prompt by hiding the prompt until heuristics determine it's needed

Show threadGlyphMar 13
@tante my prediction is that like all prompt-shortening techniques, "skills" will rapidly scale up as the LLM enthusiast community heavily leverages them, starts reselling them to each other, builds up a bunch of infrastructure to make them bigger and bigger, and eventually skills will balloon to the point where loading one immediately blows out your context, making it worthless
Show threadGlyphMar 13
@tante my prediction following that is that we will start to see "skill trees" that break things down even further into a hierarchy, and then a bunch of blog posts telling you how to craft your external prompt to glob up everything in the tree so the model really knows what it's doing, and then that "best practice" will start making the prompt too long too
Show threadGlyphMar 13
@tante god I hope I'm wrong about this, most days right now I would just like to be wrong more often
Show threadtanteMar 13
@glyph I feel that deeply in my bones.
Show threadPseudo NymMar 17

@tante @glyph

I call the first meeting of The Kassandra Club to order.

The first rule of Kassandra Club is you can talk about it all you want, but nobody will believe you.

Show threadDan DeanMar 17
@glyph @tante this is literally exactly it.
Show threadViraptorMar 13
@glyph
There's no need for all trees. The push to minimise the context already gave us the "skill search skill".
@tante
Show threadslotosMar 13
@glyph @tante Companies are social constructs. We’ve observed that system architectures follow communication patterns about 70 years ago. The observations holds.
Show threadKahia, a.k.a The AnarchivistMar 13
@tante Just like "RAG" is just giving the answer from the document into the prompt in hope the model will repeat the document extract... it's amazingly scammy
Show threadClueless by SamMar 13
@tante they found a convenient format to record their Magic Spells 🤷‍♀️
Show threadGoing T. MaineMar 13
@tante THAT’S a skill? Dang! Dang! I gotta learn the enemy more
Show threadTroed SångbergMar 17
@tante Sounds a lot like what we do with the kids in school.
Show threadManniMar 17
@tante I have to admit that those people are just very, very good a marketing/pr.
I remember imagining something *very* different when I first heard about skills.
Show threadDaniël Franke Mar 17
@tante AFAIK most AI "startups" are also just prompt-injection-as-a-service.
Show thread🏴🏳‍⚧🏴‍☠Mar 17

@tante

we have to burn that shit to the ground before it eats the world

Show threadJohnMar 17

@tante

"So, imagine an intelligent computer that does whatever you tell it to!"

"An *intelligent computer!* Wow. So I can just talk into this microphone and-"

"Well, no. You've gotta type it out."

"Oh, okay. So I just type it out here and it'll do whatever I want?"

"Well...no. I mean, if you ask it *correctly*"

"So...I've gotta be really clear about what I want? Like a genie!"

"Actually you have to write extensive code then input that or it won't do anything useful."

Show threadPseudo NymMar 17

@johnzajac @tante

So... A compiler?

Show threadJoonas SarajärviMar 17
@tante yep, and when even the top models seem to very easily disregard simple instructions in a short and simple system prompt, I highly doubt that it works well to shovel more stuff in and expect very skilled results.
Show threadPseudo NymMar 17

@muep @tante

Thing is, they don't "disregard instructions" at all. "Instructions" aren't part of their universe. It's completely orthogonal.

"Instructions" are just more tokens of context to complete. It is purely 100% a game of linguistics.

Prompts, skills, system messages, are all context, the pattern to be extended.

What trips us up is that it *sounds* like it is listening, processing, and doing things with concepts.

But only insofar as those concepts are encoded into next tokens.

Show threadJoonas SarajärviMar 18

@pseudonym @tante I know in the lower level it works like that, but models are still trained to react to instruction-like sequences so that it would be meaningful for humans to think in terms of giving instructions and getting the model to follow them.

It's a leaky abstraction, even more so that some other abstractions. But still one that in my view exists and what people build on. Or at least attempt to.