jonny (good kind)RE: https://infosec.exchange/@hacks4pancakes/116192434654015384
The only use case for AI is culpability laundering.
The US military has infinite resources and could have hired infinite people to draw up target lists, and they could have made a mistake in those target lists. Military error is not unique to AI, even if it intensifies it and mechanizes it. Previously they would have blamed bad intel, fog of war, but in any case that would be an admission of culpability and error that resides within the military.
Notice how the mere existence of AI serves to launder culpability here: by refusing to confirm or deny the use of AI in targeting, we are left to imagine a vast unknowable cybernetic military such that AI and humans can no longer be disentangled. The creation of a sense of "it is impossible to know" is the product. If they did use Claude to target bombs, you get the literal deflection of culpability - AI did it, not us, but even if they didn't, the amorphous integration of AI into military systems renders the same result: the AI may not have picked the targets, but it did provide the intel, hired the analysts, and so on.
Laundering culpability works in both directions: externally, to us, but also internally, to the people within the machine. It generates a new category of casualty, algorithmic casualties, that are immediately normalized. The machine can function by providing a plausible lie to the people that operate it, that they are just doing what the provided tools told them to do.
Gadfly (-booq-)@jonny Yes but also: https://en.wikipedia.org/wiki/Weapons_of_Math_Destruction
That can be and has been done for years-to-decades even without LLMs.
The new capability now (which isn't insignificant) is how much easier and more broadly it can be done.
@gaditb
Really, there has been a technology that one could pose any question to and receive plausible-seeming responses propped up by billions of capex capable of laundering culpability for both user and purveyor for decades now? I must have missed that and failed to consider how nothing was new except scale
Really, there has been a technology that one could pose any question to and receive plausible-seeming responses propped up by billions of capex capable of laundering culpability for both user and purveyor for decades now? I must have missed that and failed to consider how nothing was new except scale
@jonny There was, it just had a strong piece of it that was a social technology and wasn't purely technical.
It was "outsourcing to a private company which advertizes expertise to develop an opaque proprietary automated bureauocratic tool".
Look at COMPAS.
https://en.wikipedia.org/wiki/COMPAS_%28software%29
More info about its history:
https://www.propublica.org/article/machine-bias-risk-assessments-in-criminal-sentencing
Each templated question (e.g. "How likely will ___ re-offend if released?") required long time and a lot of development, advertizing, and corporate capture to bring it into use,
but absolutely:
@jonny
(a) gave plausible-seeming responses (an opaque numerical answer that often fit biases)
(b) was and is propped up by I-assume-billions-of-capex (this is my weakest claim, but I don't think the particular scale of money uses in advertizing and defending it is your most core argument)
(c) launders culpability for the user -- as far as I know no court has been held liable for using it, and it is still in use
(d) launders culpability for the purveyor -- Northpointe is still in business and has as far as I know never been charged either for the harm caused nor even for false advertizing after it was shown to be both racially biased and less accurate than group consensus of people completely lacking amy expertise
(a) gave plausible-seeming responses (an opaque numerical answer that often fit biases)
(b) was and is propped up by I-assume-billions-of-capex (this is my weakest claim, but I don't think the particular scale of money uses in advertizing and defending it is your most core argument)
(c) launders culpability for the user -- as far as I know no court has been held liable for using it, and it is still in use
(d) launders culpability for the purveyor -- Northpointe is still in business and has as far as I know never been charged either for the harm caused nor even for false advertizing after it was shown to be both racially biased and less accurate than group consensus of people completely lacking amy expertise
@jonny I don't want to say that there's nothing new here. Quantity and speed have a massively significant quality of their own, and it means that we cannot necessarily plan to frame and fight this new development the same way as before.
But so, this is not entirely unprecedented -- there are cases of the same playbook being run in the same ways before that we can learn from and use the knowledge gained to fight this significant threat better.
@jonny One thing that imo it teaches is that the specific technology, LLMs here, and its specific mechanisms and affordances might be less significant than the framing and opacity it is allowed in its social presentation.
Notably, black-box analysis of the COMPAS algorithm (despite it de facto being a part of many states' sentencing laws and practices, it is proprietary and protected from civil inspection) shows that it is probably a very simple and very describable, comprehensible, and dissectable (racist) iirc-possobly-even-linear regression model.
It was the /opacity/ that gave it its laundering power, not any inherent internal /complexity/.
This lines up with other cases where opacity allowed people to project deeper meaning and trust, e.g. the ELIZA bot.
@gaditb
That's sort of exactly what I'm saying. But the two are not possible to decouple. The framing is possible because of the technology. Prior generation quant policing is still possible to interrogate as a model that is designed for a specific aim, and its possible to articulate a resistance to a particular application and deployment of a narrowly focused product. Its not so possible to articulate an opposition to the magic everything juice sloshing as lubricant through the whole machine. I dont find "its just more of qualitatively the same thing" to be true or useful for understanding what's happening.
That's sort of exactly what I'm saying. But the two are not possible to decouple. The framing is possible because of the technology. Prior generation quant policing is still possible to interrogate as a model that is designed for a specific aim, and its possible to articulate a resistance to a particular application and deployment of a narrowly focused product. Its not so possible to articulate an opposition to the magic everything juice sloshing as lubricant through the whole machine. I dont find "its just more of qualitatively the same thing" to be true or useful for understanding what's happening.
@jonny So, whether "it's significantly more (and therefore has new implications) of nevertheless qualitatively the same thing" is true feels to me like splitting hairs -- which I will gladly do if you're interested, I like it, but tends to frustrate people --
but I do think it is /useful/.
In particular, I think a tactic that was found to work for public outreach [citation needed] for previous fights which I believe were similar,
was a two-pronged approach [again, citation needed] that
@jonny (a) contextualized the usage and configuration of the algorithms/ai/models, to make it clear that both building and using them are human/organizational decisions made by particular people, attacking the attempt to assign agency, and therefore responsibility/blame, to the model itself thst should go to the decision-makers,
and
(b) removed the mystique and hazy claims of authority from the model through combinations of accurate reporting, black-boxing and analyzing results, and reverse-engineering from the output. That attacks the ability to appeal to some real knowledge or authority encoded in the model itself,
and between the two of them -- at least in the court of public opinion, and sometimes [citation needed] in policy responses -- recenters the laundered responsibility back to those responsible.
@jonny The particular commonality here that I think I see, that is why I think this comparison is worth bringing up, is that in comparison to actual money laundering the sort of laundering happening here is INCREDIBLY shallow. Like, you absplutely don't have to hand it to money launderers, but when tracing their laundering you have to bring in forensic accounting, trace subtle flows of cash, credit, goods, debts, obligations, and gambling. It's a multi-layered maze.
Whereas this -- this is just the people doing the thing and then putting on an LED-bedazzled mask and arguing The Robot Did It.
And I think that shallow laziness can have tactical implications for opposing it, and I think there are other previous cases of similar ahallowness that we can look at for inspiration.