CassandrichAI is a lot like fossil fuel industry. Seizing and burning something (in this case, the internet, and more broadly, written-down human knowledge) that was built up over a long time much faster than it could ever be replenished.
I think a lot of ppl who are skeptical of the criticisms here really don't understand how it's *burning* anything because they haven't thought about how value is derived from provenance.
Suppose you have a $100M batch of medicine but an inside saboteur has put poison in a few bottles and you can't determine which ones. How much is your inventory worth? $0 - or less since you have to dispose if it too. This is because the value depended on the provenance - having a reasonable basis to believe it's what it appears to be.
For a lower stakes example, why is the organic produce in the grocery store more expensive? If you took the labels off and mixed it all together with the rest, it wouldn't be. The value at sale time is derived from a meticulous record keeping process that makes faking provenance comparable in cost to just doing it right.
Provenance of human written knowledge comes from a lot of places. Just because something was written by a human doesn't make it accurate or non-garbage. But the labor cost in producing misinformation that's hard to distinguish from meaningful writing in the same domain, together with a lot of systems we have in place, makes evaluating provenance a tractable problem.
The ability to produce unlimited amounts of plausible-looking garbage at essentially no cost, and to crowdsource that kind of vandalism to millions of randos by disguising it as something fun, destroys that capability. It's a DDoS attack on written knowledge.
I've seen folks arguing that good and accurate info can come out of "AI" too, so we can't dismiss it as garbage.
This misses the point entirely.
Even if "AI" "says" something 100% accurate, the provenance is still garbage. It's like a broken clock. It's like waiting for the nazi to say something non-offensive and saying "wow they're at least right about some things".
And quite coincidentally, all this comes right at the moment Elon burned down the greatest tool we had for realtime vetting of information across a vast range of knowledge domains through a shared domain-knowledge trust graph. We're still a long way from rebuilding that here.
So how do we move forward? We can't entirely put this shit back in the shitter. The models are large but tractable for bad actors to keep and continue using even if we somehow banned them.
But there's a lot we can do...
We can recognize the people pushing this shit as charlatans, enemies of humanity, rather than the geniuses they want us to see them as. We can stop falling for their scam of the day. We can organize to tear down their power, devalue their wealth, deliver them consequences.
We can preserve and strengthen our standards for provenance of knowledge. In particular, open and community based projects that deal with knowledge can clearly and unequivocally ban model-generated bullshit and users who try to sneak it in.
A lot of the problems with provenance of knowledge didn't begin with LLMs. LLMs just greatly accelerated the threats that fashy political aspirations and capitalism/adtech already incentivized. LLMs make the defender's role a lot harder, but not fundamentally different.
jz.tuskYou say "provenance", I say "trust".
Every time I see people creating new social network apps, I ask "how can I find entities to trust, how can I remember them? how do I know which entities I can't trust, and more importantly for these, how do I recognize them as distrusted when I see them again?"
It seems *so* damn basic to me, and yet on-line VC project, heck, the fascist project, seems to be to destroy our networks of trust.
Prof. Emily M. Bender(she/her)@dalias Do you have this thread as a blogpost anywhere? I think these are really crucial points and I'd like to be able to point people to them...
@emilymbender No, but they're public and I use microblogging as a permanent medium. Linking/quoting/archiving this thread is fine.
@dalias Thanks -- though I'd still encourage you to put them somewhere with a friendlier UI....
Moritz Negwer@dalias Nice summary. This is far outside my expertise, so I only have to offer a crude analogy:
The trope used to be that data is the new oil, so by that analogy LLMs would be the distilled gasoline?
(Note how this analogy implies that the result powers a huge speedup in e.g. commerce and warfare. Plus brutal battles over control of the oil fields / data warehouses, often ending with autocrats in charge.)
Christian Schwägerl@moritz_negwer @dalias Per @emilymbender publication with Chirag Shah, the proper analogy for LLMs is an oil spill in a fragile ecosystem
https://dl.acm.org/doi/10.1145/3649468#fn30
https://dl.acm.org/doi/10.1145/3649468#fn30
Techronic9876@dalias this is a decent point about LLMs and AI but it’s going to be solved within the year from the research labs, then probably another 6 months rolled into the FOSS/commercial AI tools
There’s already been decent work into figuring out where LLMs got the info from, the next step is understanding why it used those sources, then training it how to discern on which sources to value
Darrel PlantWho's got the time for all that, though? And what about the fact that the well of information future AIs draw from is forever polluted by the previous generations?
More importantly, why wasn't the lack of sourcing seen as an issue before the fact, rather than afterward? Every authoritative source in history had footnotes, references, etc. In the digital realm, even Wikipedia has references. So why did the big brains developing AI not take provenance into account?
@darrelplant @dalias because researchers didn’t know LLMs would be able to chat. This was an emergent capability. They weren’t trying to build a chat bot, they were trying to build special-purpose sentiment/analysis/grammar/translation tools, and chatting took everyone by surprise. LLMs were essentially an accident
Now that they know LLMs can do zero-shot and one-shot learning, they’re working very hard on the provenance/explainability/alignment questions
Pretty sure they knew they would be able to chat before they released products with names like "ChatGPT".
I've been watching attempts at chatbots develop since the late 70s. If the people building tools to write text based on language data had no inkling that their tools could fake holding a conversation, then they are very, very stupid people.
@darrelplant @dalias OpenAI releasing chatgpt was hugely controversial — & still is — by the people who actually discovered LLMs. OpenAI didn’t build chatgpt they just commercialized it. But once something is published research anyone can use it
Looking at how industries & governments reacted, I don’t think anything would have stopped someone from commercializing LLMs before they were ready. Best we can do now is harass/regulate new entrepreneurs to not repeat that
@Techronic9876 @darrelplant LLMs were not "discovered". All this was known over 50 years ago. They just lacked access to the volume of text and GPUs to implement it, which capitalist asshats "solved" by scraping everyone's stuff without license or consent.
@Techronic9876 @darrelplant That LLMs do what they do is not surprising at all to anyone with a basic understanding of probability and statistics, and really shouldn't have been a century ago, either.
I don't know, after all the science-fiction I read and watched, I'm really kind of surprised at how bad they are. It's 2023! Where's my jetpack?
@darrelplant @Techronic9876 I mean if you know how they work it's not surprising.
It's also why sci-fi authors never envisioned "AI" as LLMs - because they're such a ridiculously dumb, obviously "fake" way to do AI, with no intelligence whatsoever.
The programming and data storage details of the positronic brains of my youth were never really specified. I mean, we were still working with punch cards. It was assumed it was going to be something more sophisticated than punch cards and glossed over. "Big, dumb database search" wasn't a thing.
@darrelplant You could clearly see from how it was depicted that it was parsing language and applying logical rules, not behaving like a glorified markov bot.
@dalias
Considering that LLMs can't even stick to the rules expressed in a 19-word, one-sentence instruction, I'm not sure I'd really want to trust one "governed" by Asimov's Three Laws.
Considering that LLMs can't even stick to the rules expressed in a 19-word, one-sentence instruction, I'm not sure I'd really want to trust one "governed" by Asimov's Three Laws.
The provenance issue isn't even related to chatbots specifically, though. A traditional search engine basically provides its reference data by way of links to those references. It doesn't verify their validity, but they are provided, just a like a freshman college student writing a paper.
Even if an LLM was incapable of chat, and its function was to summarize a topic, or to generate articles to put human writers out of work, IT NEEDS TO PROVIDE REFERENCES.
That's Doctor Foreigner 🛂@Techronic9876 @dalias solved in a year? That's an excellent joke! Maaaybe some good detection techniques would be viable/scalable in that sort of time range, but this is a social problem not just a technical one. Look at the whole cultural around fact checking and misinformation labelling and suppression. It wasn't doing that well with human scale misinformation and bullshit, nevermind the scale possible with the current generation of LLMs, and that BS had no trouble finding an audience so...
@alsothings @dalias there’s already really good work on arxiv on identifying which documents an LLM output comes from, and other work on letting LLMs know the probability of tokens explicitly, and then other work on the output being a system of agent LLMs
If you put all this together you have an AI that can explain itself and explain other things, down to the sources & other possibilities
I’ll be surprised if someone doesn’t have a working demo of this by fall, & an OSS project by next spring
@Techronic9876 @dalias yeah, I don't doubt your timing on the narrow technical issues you're talking about. The thing that compelled me to reply in the first place, was the bit of (probably unintentional) rhetorical fun you did where you declared the problematic deployment and use of modern LLMs would be solved in about 18 months, rather than just _providing_ a means to opt in to better systems.This is, as I said, hilariously far away from 'solving' the systematic problems enabled by these tools
@Techronic9876 @dalias In contrast, I think the systematic problems around information and knowledge sharing/ discovery that we are all now faced with are _generational_ in timescale, but would just love to be wrong
@Techronic9876 @alsothings "An AI that can explain itself and explain other things, down to the sources & other possibilities"?
No you do not. You have a probability model that tells you, for particular word soup, which sources and explanations are most likely, within that model, to have some correlations with the word soup that can plausibly be interpreted by a reader as agreeing with it.
@dalias @alsothings it’s not a word soup, otherwise this would have worked two decades ago with the n-gram models
It’s a multi thousand dimensional vector space where the model regresses each dimension into some concept, then maps the tokens at the intersection of all possible concepts it represents
This means the model can infer new concepts by interpolating between points, or extrapolating to new points in the space
@Techronic9876 @alsothings No it does not mean that. Thinking it does is magical thinking by someone whose salary depends on not understanding what they're doing.
@Techronic9876 @dalias what do you mean exactly when you say 'worked'? LLMs are at heart, a big refinement of various sequential models that came before (n-grams based things markov models etc). But you're just describing the precision of the word soup, not some way in which that sort of modelling stragegy would qualitatively shift epistemic modes from stochastic mimicry (at potentially very high levels) to operational knowledge?
@alsothings @dalias a single LLM would not, but that’s where the systems previously mentioned that are currently being researched will come in
Next time you’re in public and are eaves dropping on people’s casual conversations, tell me it doesn’t sound like two chat GPT agents 99% of the time; people under-appreciate what a really good precise interactive word soup can actually do
@Techronic9876 @alsothings That's literally the cargo-cult fallacy.
@dalias @alsothings it’s a functionalist perspective
But the power of a belief system is in its predictive ability. You predict LLMs will stagnate and continue to have poor explainability and reasoning
I predict in about a year and a half, consumer AI will have mostly solved the explainability problem and continue to get better beyond that
I hope whoever is wrong then will update their belief system to reflect what actually happened
@Techronic9876 @alsothings It's *not* a functionalist perspective unless the function you have in mind is convincing people to believe something. Which is generally a malicious function.
In your example of overhearing a conversation, the difference is that it corresponds to a vast network of consistent facts unknown to you but that could become known later, and the GPT garbage doesn't. Having these appear similar to you is a problem not an achievement to celebrate.
@Techronic9876 @alsothings If you think the explainability problem is something that can be solved, you don't understand the problem.
@dalias @alsothings the technical aspect can be solved, humans will always fight to use something for good or evil
good people need to keep using and building good AI
Coach Pāṇini ®
Asta [AMP]@paninid @Techronic9876 @dalias to the infinite surprise of nobody who wasn’t a fucking hype booster
d@nny disc@ mc²@aud @dalias @paninid @Techronic9876 github copilot landing page stated day 1 that attribution was "coming soon!" it made me so upset that is a flat lie
@hipsterelectron @dalias @paninid @Techronic9876 lmao seriously. I genuinely doubt they thought that was true, too.
Jed Brown@hipsterelectron @aud @dalias @paninid @Techronic9876 Yup. The value proposition all along was plausible deniability of copyright infringement. As soon as you provide ready access to attribution, you destroy that plausible deniability as well as credibility of the anthropomorphizing "AI vernacular" being used for lobbying and legal defense.
@jedbrown @hipsterelectron @dalias @paninid @Techronic9876 That plausible deniability extends to basically all "AI" uses: you bake in the bias, then get to say "computer says no", as they say.
@aud @dalias @paninid @hipsterelectron @Techronic9876 Indeed, and that's why "AI is replacing human jobs" is misleading. More like "AI is doing things that would break existing law if a human were to do the same things". This is obviously valuable because crime pays.
@jedbrown @aud @paninid @hipsterelectron @Techronic9876 Wow this is such a perfect explanation and ALSO explains why it's the exact same people behind it as cryptocurrency.
@dalias @jedbrown @paninid @hipsterelectron @Techronic9876 "you couldn't charge an NVIDIA GPU with a crime... not outside of Texas, anyway!"
@aud @paninid @hipsterelectron @jedbrown @Techronic9876 Please please please charge the nvidia gpus with a crime and apply civil forfeiture to them.
@dalias @paninid @hipsterelectron @jedbrown @Techronic9876
computer says no,
think that's not a crime?
now the GPU's doing science
on the company's dime
computer says no,
think that's not a crime?
now the GPU's doing science
on the company's dime
@dalias @paninid @hipsterelectron @jedbrown @Techronic9876 oh my god: having to donate compute time to the public good would be a lovely punishment for these tech fuckers.
LisPi@dalias We could also make a habit of making knowledge verifiable from first principles.
Kevin Leecaster@dalias
Whether or not we can organize in the face of their efforts to divide their opposition is my biggest concern.
Whether or not we can organize in the face of their efforts to divide their opposition is my biggest concern.
Raven Onthill@dalias there are uses for the technology, I think, but, as you say, the generation of spam is an appalling problem. Beyond that is the complete irresponsibility of the people who've let this loose. They apparently really believe they're going to create artificial people and enslave them. I don't think the project is likely to be a success and if it is a success it would be a repugnant act.
Globaltom@dalias the thing Twitter (the company) never understood/capitalized on was that Twitter (the platform) brings together people with shared interests.
They seemed to think it was all people following celebrities, broadly defined. The original checkmarks were mostly limited to those who were 'celebrities' in their field - hence why the criteria were about number of followers.
Falcon Darkstar@dalias we may never rebuild it here. Fediverse users were so intent on enforcing existing community norms around CWs and alt text that many of my friends in the infosec crowd for example had a few posts here, discussions about almost nothing other than that, and then with little other engagement, in many cases they continued using Twitter.