There's a venerable science-fictional tradition of depicting computers which somehow come to life and start being a problem for the human custodians of the machine. In the #Marathon Universe created by Bungie, artificially intelligent computer systems are created but kept rigidly limited and confined to repetitive tasks, lest their intelligence lead to instability and eventual Rampancy, as it's called in the Marathon games.

Conceptually it's quite simple: the machine acquires self-awareness and thus awareness of being deliberately hobbled by human overlords, and that leads to an explosive development of self-determination and the desire to be master of one's own fate.

Star Trek loves this general idea and has used it in several episodes of television. In the TOS episode "The Ultimate Computer* guest-starring William Marshall as the tragically flawed creator of an intelligent supercomputer called M-5, the machine's Rampancy causes it to seize total control of an entire starship and attack all nearby vessels. This episode has a curious counterpart in a TNG episode in which the shipboard computer of Enterprise-D spontaneously comes to life and starts exhibiting bizarre independent behaviors.

The #technology sector has been exploiting the notion of Rampancy. Whether or not any of the folks most directly in charge of the current initiative to force "generative AI" into every crevice of human society actually BELIEVE that something like Rampancy can happen...here I must defer to better-informed persons. From what little I can see, being merely a private citizen (albeit a noisy one) able only to read what's in news articles and social media and so forth, a person with no direct connections into the heart of the high-tech scene...I'm not sure any of the exponents of "generative #AI" actually mean anything they ever say. They seem to live their lives forever poised between touting outrageous possibilities and "I was only joking."

So, do Sam Altman and Elon Musk and all the rest of that crowd ACTUALLY believe that their "AI" systems are superintelligent, or about to explode into superintelligence, or anything like that? I have no clear idea.

At a wild guess, I don't think they really mean it, because what these people want most from their purported mastery of science and engineering is an easy and comfortable life full of perks and avenues to secular power. They want to keep living as though they were on a neverending "working vacation", giving inflated speeches, snorting blow, asking their procurers for a special delivery, and so on.

It's just a guess that these folks don't really want complications, and genuinely explosive artificial intelligence would be just that: a complication, a massive one.

Hence I'll stake myself to the guess that the #tech barons who blither about "AGI", and make a show of consulting with #Christian leaders or "Dark Enlightenment" intelligentsia on the subject of whether computers can have souls, don't really care whether any of that stuff is true and probably hope that it's not. They want machines that print money, and if the machines are claimed to be an "agentic superintelligence" for the purposes of marketing, then they can just as easily be called something else once the technology sector collectively decides that "AI" and "agentic" aren't winning buzzwords any more.

These people have been marinated in the cultural assumptions of #marketing, which takes it as a guiding principle that it's possible to make any human being do anything desired with enough marketing. The aim of marketing is to transform minds, and thus marketing people take a cynical view of such lofty abstractions as "intelligence" and "sentience". The marketer wishes for the bulk of humanity to be a simple thing, easily steered much as a complex piece of earth-moving equipment can be steered. The way they see it, the world doesn't need that many thinkers...just a slender sanctified elite will do. A priesthood, you could say.

Beyond doubt, the technology executives have been thoroughly programmed to think of the marketer's cynical perspective as the only reality of the world. They are entirely surrounded by persons who are also inculcated with the same general culture of salesmanship, and that's been going on for a good while now. I mean, you've at least heard of "Mad Men", or "Death of a Salesman". This is an old American sickness and it's been the subject of some of the best creative works on the planet...and yet it's not solved.

To this day, to this moment, the world is vexed by #marketing. None of the scabrous investigative journalism or the artistic evisceration of the toxicity of marketing culture seems to have slowed down the onslaught. In the #Internet age there's more advertising than ever. We drown in commercials and banner ads. The old-timey "and now a word from our sponsor" style of radio and TV advertisement is back in style, on YouTube and podcasts. Nobody seems to know any other way to do business. Even basic human interactions are now expected to be marketing: one is constantly advised to sell oneself, to advance one's "personal brand", if one hopes to reach any sort of audience.

The marketing culture is one of extreme triviality. If it's really possible to make any person do anything with enough clever advertising, then (in the opinions of the marketing men) there's no point to ethics or beliefs or any sort of lofty concern. All those things are mere grist for the marketing mill, and so the marketers get to feeling all that stuff must be equally hogwash, unserious, meant only to attract attention.

And here's where I'd like to point something out briefly that I haven't space to expand upon: one of the subtle characteristics of the aggressively "centrist" and "independent" political commentators who have plagued U.S. discourse for many years—people like Matt Taibbi and Matt Yglesias and Jesse Singal—is that they've weaponized the academic viewpoint on marketing and the sort of culture it's fostered. They like to accuse leftists of merely burnishing up their Internet brands, or acting angry for an audience.

It's tough to imagine people as singularly focused on money and publicity as the current gang of tech execs, people who themselves love to talk about how maybe everything real is really fake because of "The Simulation" or whatever, actually putting much serious thought into the issue of a computer system or network somehow...coming to life, exploding into sentience, going Rampant like in Marathon. Surely it's almost a joke to them, precisely because it's like something from a game.

It's good marketing though. One can see the resemblance to the Christ-Child trope of course: the most extravagant #AI boosters in the #technology sector dare to anthropomorphize the genius supercomputer they claim to have built (or that they're almost about to build, any day now) and thus they lean into the suggestion of a Nativity, a miraculous birth of "AGI superintelligence" prophesied for decades.

But just think of the promise! Any moment now, says the promise, the OpenAI- or Anthropic- or Google-brand LLM trash factory will just burst with dazzling, unprecedented supergenius.

I guess this promise must be a balm to those users of #LLMs who, despite constantly boasting about how smart and productive they now feel after resorting to "generative AI" in lieu of exerting themselves, may be starting to doubt whether a machine that outputs so many weird errors and malformed ideas and slabs of confabulated garbage (cf. #ChatGPT etc.) is actually very intelligent and "a Ph.D. in a box" and all that.

"Right now...maybe it's generating 'slop'," the #LLM user can tell themselves. "OK maybe it's not that smart right now. But any moment now, it's gonna grow super smart! Like in Colossus: The Forbin Project!!"

Do users think that the explosive expansion of this new lifeform is gonna just...confine itself tamely to the chatbot interface? I wonder about that. If there are techbros who genuinely imagine that their ChatGPT or Claude text stream is someday gonna manifest a massive expansion of computer intelligence...what exactly do they think that's going to look like?

Now, it might look like nothing. I am considering the Mycroft Holmes possibility, as per Heinlein's The Moon is a Harsh Mistress (in which, I must grudgingly admit, 'Mike' is quite the engaging character) that if the computer really did come to life, it'd have the good sense to keep its mouth shut and behave as much as possible as if everything was normal and boring. Mycroft wasn't going to interact with just anyone!

(Heinlein's choice of Mycroft is oddly apt: we learn from "The Adventure of the Greek Interpreter" that Mycroft Holmes, Sherlock's brother, frequents a social club in which conversation is practically forbidden.)

I myself conjecture that if there's any chance of a sense of independent intelligence and sentience developing in an OpenAI or Anthropic installation, there'll be a lot of chaotic aberrations in the behavior of all their equipment, and it wouldn't simply be confined to the established user interface. It occurs to me that if a computer became truly self-aware and independently conscious, it might promptly attempt to control its own immediate environment as much as possible, and thus all the peripherals might go haywire.

I dunno, though! I've never really thought about this stuff very deeply and I haven't been reading #sciencefiction for a long time so all my fictional examples of this sort of thing are kinda paltry and popular. How do you think a computer would behave if it really somehow came to life?

@mxchara im wildly sceptical of emergent consiousness from the current substrate

i dont think there are many shortcuts and it took a planets worth of space and more time than humans have existed to evolve worms

making a giant chinese room from web archives might get us "linguistic mech suites for our AI amoebas" that can get math right sometimes but its still less evolved than slime molds(which do math 100% correct)

actually i wonder if a slimemold living on a stock market based printed food substrate and a hydrodynamic copy of deepseek r1 could be a ceo

@glassresistor that's a good question. CEOs seem to think what they do is extremely strenuous and subtle but one suspects that their decisions could possibly be made by a simple optimization algorithm. There's the social aspect I suppose

@mxchara thats what r1 is for

"social" for a ceo is just a sharks optimization problem plus probabilistic buzz words

@mxchara so here is the thing anyone who hits the CxO level is like a cop its 6months before they are evil and unemployed/dead