CassandrichAI is a lot like fossil fuel industry. Seizing and burning something (in this case, the internet, and more broadly, written-down human knowledge) that was built up over a long time much faster than it could ever be replenished.
I think a lot of ppl who are skeptical of the criticisms here really don't understand how it's *burning* anything because they haven't thought about how value is derived from provenance.
Suppose you have a $100M batch of medicine but an inside saboteur has put poison in a few bottles and you can't determine which ones. How much is your inventory worth? $0 - or less since you have to dispose if it too. This is because the value depended on the provenance - having a reasonable basis to believe it's what it appears to be.
For a lower stakes example, why is the organic produce in the grocery store more expensive? If you took the labels off and mixed it all together with the rest, it wouldn't be. The value at sale time is derived from a meticulous record keeping process that makes faking provenance comparable in cost to just doing it right.
Provenance of human written knowledge comes from a lot of places. Just because something was written by a human doesn't make it accurate or non-garbage. But the labor cost in producing misinformation that's hard to distinguish from meaningful writing in the same domain, together with a lot of systems we have in place, makes evaluating provenance a tractable problem.
The ability to produce unlimited amounts of plausible-looking garbage at essentially no cost, and to crowdsource that kind of vandalism to millions of randos by disguising it as something fun, destroys that capability. It's a DDoS attack on written knowledge.
I've seen folks arguing that good and accurate info can come out of "AI" too, so we can't dismiss it as garbage.
This misses the point entirely.
Even if "AI" "says" something 100% accurate, the provenance is still garbage. It's like a broken clock. It's like waiting for the nazi to say something non-offensive and saying "wow they're at least right about some things".
And quite coincidentally, all this comes right at the moment Elon burned down the greatest tool we had for realtime vetting of information across a vast range of knowledge domains through a shared domain-knowledge trust graph. We're still a long way from rebuilding that here.
So how do we move forward? We can't entirely put this shit back in the shitter. The models are large but tractable for bad actors to keep and continue using even if we somehow banned them.
But there's a lot we can do...
We can recognize the people pushing this shit as charlatans, enemies of humanity, rather than the geniuses they want us to see them as. We can stop falling for their scam of the day. We can organize to tear down their power, devalue their wealth, deliver them consequences.
We can preserve and strengthen our standards for provenance of knowledge. In particular, open and community based projects that deal with knowledge can clearly and unequivocally ban model-generated bullshit and users who try to sneak it in.
Techronic9876@dalias this is a decent point about LLMs and AI but it’s going to be solved within the year from the research labs, then probably another 6 months rolled into the FOSS/commercial AI tools
There’s already been decent work into figuring out where LLMs got the info from, the next step is understanding why it used those sources, then training it how to discern on which sources to value
That's Doctor Foreigner 🛂@Techronic9876 @dalias solved in a year? That's an excellent joke! Maaaybe some good detection techniques would be viable/scalable in that sort of time range, but this is a social problem not just a technical one. Look at the whole cultural around fact checking and misinformation labelling and suppression. It wasn't doing that well with human scale misinformation and bullshit, nevermind the scale possible with the current generation of LLMs, and that BS had no trouble finding an audience so...
@alsothings @dalias there’s already really good work on arxiv on identifying which documents an LLM output comes from, and other work on letting LLMs know the probability of tokens explicitly, and then other work on the output being a system of agent LLMs
If you put all this together you have an AI that can explain itself and explain other things, down to the sources & other possibilities
I’ll be surprised if someone doesn’t have a working demo of this by fall, & an OSS project by next spring
@Techronic9876 @dalias yeah, I don't doubt your timing on the narrow technical issues you're talking about. The thing that compelled me to reply in the first place, was the bit of (probably unintentional) rhetorical fun you did where you declared the problematic deployment and use of modern LLMs would be solved in about 18 months, rather than just _providing_ a means to opt in to better systems.This is, as I said, hilariously far away from 'solving' the systematic problems enabled by these tools
@Techronic9876 @dalias In contrast, I think the systematic problems around information and knowledge sharing/ discovery that we are all now faced with are _generational_ in timescale, but would just love to be wrong
@Techronic9876 @alsothings "An AI that can explain itself and explain other things, down to the sources & other possibilities"?
No you do not. You have a probability model that tells you, for particular word soup, which sources and explanations are most likely, within that model, to have some correlations with the word soup that can plausibly be interpreted by a reader as agreeing with it.
@dalias @alsothings it’s not a word soup, otherwise this would have worked two decades ago with the n-gram models
It’s a multi thousand dimensional vector space where the model regresses each dimension into some concept, then maps the tokens at the intersection of all possible concepts it represents
This means the model can infer new concepts by interpolating between points, or extrapolating to new points in the space
@Techronic9876 @alsothings No it does not mean that. Thinking it does is magical thinking by someone whose salary depends on not understanding what they're doing.
@Techronic9876 @dalias what do you mean exactly when you say 'worked'? LLMs are at heart, a big refinement of various sequential models that came before (n-grams based things markov models etc). But you're just describing the precision of the word soup, not some way in which that sort of modelling stragegy would qualitatively shift epistemic modes from stochastic mimicry (at potentially very high levels) to operational knowledge?
@alsothings @dalias a single LLM would not, but that’s where the systems previously mentioned that are currently being researched will come in
Next time you’re in public and are eaves dropping on people’s casual conversations, tell me it doesn’t sound like two chat GPT agents 99% of the time; people under-appreciate what a really good precise interactive word soup can actually do
@Techronic9876 @alsothings That's literally the cargo-cult fallacy.
@dalias @alsothings it’s a functionalist perspective
But the power of a belief system is in its predictive ability. You predict LLMs will stagnate and continue to have poor explainability and reasoning
I predict in about a year and a half, consumer AI will have mostly solved the explainability problem and continue to get better beyond that
I hope whoever is wrong then will update their belief system to reflect what actually happened
@Techronic9876 @alsothings It's *not* a functionalist perspective unless the function you have in mind is convincing people to believe something. Which is generally a malicious function.
In your example of overhearing a conversation, the difference is that it corresponds to a vast network of consistent facts unknown to you but that could become known later, and the GPT garbage doesn't. Having these appear similar to you is a problem not an achievement to celebrate.
@Techronic9876 @alsothings If you think the explainability problem is something that can be solved, you don't understand the problem.
@dalias @alsothings the technical aspect can be solved, humans will always fight to use something for good or evil
good people need to keep using and building good AI
@Techronic9876 @alsothings You are so deep in your dreams to make a career out of this and so out if touch with mathematical reality.
@dalias @alsothings I think I have a very clear grasp of the mathematical reality, having done a hundred hours of course work on the topic, thousands of hours of reading, and several days of conference presentations
Saying it’s just a “word soup” demonstrates being out of touch with the mathematical reality