Prof. Emily M. Bender(she/her)There's a lot that's alarming in this article, but perhaps the most alarming part is the NYC spokesperson assering that the problem can be fixed via upgrades:
>>
https://www.thecity.nyc/2024/03/29/ai-chat-false-information-small-business/
It seems to bear repeating: chatbots based on large language models are designed to *make shit up*. This isn't a fixable bug. It's a fundamental mismatch between tech and task.
Also, it's worth noting that RAG (retrieval augmented generation) doesn't fix the problem. See those nice links into NYC web pages? Not stopping the system from *making shit up*. (Second column is chatbot response, third is journalist's report on the actual facts.)
>>
Finally, as is usual and *completely unacceptable* the public does not have information about the training data used to build this thing, just the info that Microsoft made it.
Stone Bear (has moved)@emilymbender From where I sit, that's more than sufficient information. It's AI, and MSFT made it. 
❗ NEXT!!
❗ NEXT!!
Eric LawtonA Canadian court just found Air Canada liable for its chatbot's lies, which makes perfect sense. They're just publishing it on their website so are responsible for its content.
There's no "AI exception" in any laws, anywhere, that I know of.
Hollie@emilymbender @gregtitus The alt text seems different than the screenshot?
@hollie @gregtitus Oh thanks for catching that! I've edited the alt text to reflect the right part of the table I screen capped.
Resuna@emilymbender It's the end result of decades of writing code with an eye to "beating the turing test", where "beating the turing test" explicitly requires fooling the judges. It's not a surprise that they have gotten really good at writing code to fool the judges.
When you're writing code to fool people, fooling people isn't a bug, it's a design feature.
dragfyre@resuna @emilymbender A THOUSAND TIMES this.
rhempel@emilymbender As I'm sure you know already, it's worth reminding everyone that LLMs are ALWAYS making stuff up - ALWAYS.
It's just that very often most people can't tell, and somehow politicians and captains of industry really, really, really want to believe that they can replace humans with AI.
They can't - and until it hits them in the wallet they won't care.
soc@emilymbender What is "count tips toward minimum wage requirements" if not "take a cut of your worker's tips"?
Perhaps the real fault of the bot is that it hasn't been exposed to years of corporate propaganda teaching it that the current practice is benign and totally unobjectionable?
Ergative AbsolutiveMin wage in NYC is $15 an hour. If employees are tipped, employers can pay them $10.35 an hour and assume that the tips make up the deficit. So the employer can make $4.65 an hour in saved wages, which is taking effectively a 'cut' of the tips, yes.
However,taking a 'cut of the tips' more typically means something like 'Employer gets X% of tips'. So if an employee makes $100 in tips during a dinner hour, the employer cannot legally take, say, 10% ($10) of it.
Also, if an emplyee does not make minimum wage from tips, the employer must make it up. So if an employee only makes, say, $3.00 in tips, not only is the employer not entitled to 10% of it, they must actually shell out an addition $1.65 to make up the employee's pay to the full minimum wage of $15 per hour.
(source: https://dol.ny.gov/minimum-wage-0)
And anyway, allowing the tips to count towards minimum wage is only for hospitality workers. Employers can't touch tips otherwise.
RenéWiersma@ergative @emilymbender Which part of
> years of corporate propaganda
> teaching that the current practice
> is benign and totally unobjectionable
did you struggle with, perhaps I can help?
Björn@emilymbender @fps_gbg IMHO the final issue with any issue is that if the people in charge are able to
see issues in things they and can adjust. If they want this and don't care of the repercussions any logical argument is lost on them, the only move is to
replace them. There are some many people with authority that ignore scientific consensus for ideological reasons that is has become really dangerous
in for us.
#politics #politicsandtechnology
see issues in things they and can adjust. If they want this and don't care of the repercussions any logical argument is lost on them, the only move is to
replace them. There are some many people with authority that ignore scientific consensus for ideological reasons that is has become really dangerous
in for us.
#politics #politicsandtechnology
Qybat@emilymbender A tangential point: RAG and similar technologies are in demand because it's becoming impossible to find desired information through conventional search, due to the resurgence of SEO methods, the heavy dependence of most websites on advertising, and the optimisation of websites to this aim. Search for a simple question now and you may have to wade through pages of fluff and padding (perhaps AI generated) to make sure you see enough ads to earn your answer.
@emilymbender I've noticed that LLM bots are very, very reluctant to ever admit they are wrong about anything. I wonder if this is because they are trained upon crawled text from general internet conversation - could it be that statistically it's very rare for anyone on the internet to actually admit they made a mistake, and LLMs reflect this? They have learned to imitate human stubbornness and pride.
jonny (good kind)@emilymbender
Ya totally. RAG just makes the situation worse by giving plausible cover to companies who dont really care if the thing actually works or not to say it does for as long as people are buying.
Ya totally. RAG just makes the situation worse by giving plausible cover to companies who dont really care if the thing actually works or not to say it does for as long as people are buying.
Peter Stone
Lucy Weir@emilymbender I don't understand: the Chatbot's response is internally incoherent. Yes, you can take tips, it says, but then it says they must report tips (which is not the same) and then it repeats that 'therefore, you are allowed to take ... tips'. Which is not what's implied by having to report them. It's gobbledygook.
wobweger 
@emilymbender
indeed, makes me wonder 🤔
what does it take people to recognize #SALAMI is flawed, fawlty tech by design and most certainly so far from intelligent as imaginable,
there is no tweak - no update - no new version - no nothing to make a marketing stunt come true 🤔
evidence is piling up 🤔
what does it take
#StochasticParrots
indeed, makes me wonder 🤔
what does it take people to recognize #SALAMI is flawed, fawlty tech by design and most certainly so far from intelligent as imaginable,
there is no tweak - no update - no new version - no nothing to make a marketing stunt come true 🤔
evidence is piling up 🤔
what does it take
#StochasticParrots
fmc01@emilymbender NYC spokespersons and their minders know that chatbots are just computer programs that make stuff up - but when you are looking at the world through a pile of money from cost savings anything goes. Chat bots like humans tell truths and tell lies but only humans know the difference.
Lala@emilymbender it will be fixed via Upgrades the same way as self driving cars are fixed and will totally make zero mistakes and harm no one.
Oh, did I kill that pedestrian? Yeah, i'll try better next time. And I passed 99 others without hitting them, thats 99% accuracy
Oh, did I kill that pedestrian? Yeah, i'll try better next time. And I passed 99 others without hitting them, thats 99% accuracy
Kee Hinckley@emilymbender Faced with complains about this, the government agency says, “But thousands of people were helped!” Sure, but how many weren’t lied to?
I feel like this is yet another instance of our general tech+automation+capitalism problem of ignoring the heavy cost of false positives. The systems are at a scale that makes manual review expensive, and the companies have zero incentive to discuss it, because the customer isn’t likely to notice most of them.
Police departments love image recognition because companies claim they catch N% of crooks. They never talk about the percentage of innocent people caught.
Fraud finding algorithms found tons of fraud in postal offices in England. Nobody looked at how many times they caught the wrong people until it was far too late, and the company hid it when they found out rather than lose the contract. People went to prison, lost jobs, committed suicide. But hey, the system caught fraud.
Facebook lauds the number of accounts banned for hate speech but never publishes the number banned incorrectly. Like most companies selling “AI”, they don’t actually know that number, it requires expensive manual review they don’t want to do. When I was in that org I was told that every quarter they put it on the “ought to do” list, but it never made the cut. In large part, I suspect, because, because developers and teams are rewarded based on how they increase profit, and Facebook have no way to measure what they lose by banning the wrong person. Getting developers there to work on something that doesn’t result in promotion is hell.
Jesse@emilymbender @billseitz NY should upgrade their mayor
Gabriel N@emilymbender upgrade = ENHANCE!
Riley S. Faelan@emilymbender "Our magic eight-ball has delivered thousands of timely, accurate answers! Who cares about thousands of timely, inaccurate answers that it might also have delivered?"