Lynne
Prof. Emily M. Bender(she/her)Hey Philly folks --- SEPTA is doing a pilot of a system supposedly uses "AI" to call the cops when the "AI" detects a gun.
This is terrifying. What kind of civil oversight do you all have going on out there?
>>
@alex is quoted raising key points. There's zero transparency about how the system is evaluated and it's pretty predictable what harms are going to happen --- and to whom.
>>
And can you spot the GLARING omission in this evaluation plan? (Answer in next post, for those who aren't sure.)
>>
Absolutely key point of evaluation: How many times does the system send the cops in to a situation where no violence was occurring, but all primed to think that there is?
This is maybe most urgent right now for Philly, but we've all got work to do making sure our electeds aren't setting up this nonsense in our own towns.
Denial 🐔 Shown@emilymbender so while the concerns raised by Alex are absolutely valid, so are the concerns of people facing gun violence in their communities.
I'm not sure I've seen anything described in the system that isn't any different from a zealous camera-watching security guard. They often don't get bias correcting training. What is the AI equivalent of a job interview, a background check, and a 6-month new employee evaluation period? It's at least potentially possible to audit outcomes of the AI equivalent of an anti-bias training, whereas doing that for a human training is often more challenging.
Should there be an audit? Yes, thank you for calling for it. Perhaps ZeroEyes would be willing to hire or allow DAIR to do one. Perhaps SEPTA would do it.
Keeping humans in the loop and at least as skeptical of the tech as they would be of an unknown person making the same claim seems vital. Perhaps SEPTA would do let DAIR give a quick training to those people.
How can we advance the AI ethics along with the exigencies of living amongst rampant gun violence. Giving critiques, even valid ones, of the efforts to address problems without simultaneously offering viable paths to do better is problematic in it's own ways.
Also, the movie project mentioned in the article looks amazing.
Wayne Werner@DenialShown @emilymbender it's depressingly ironic that we actually already have a tool in our arsenal that's effective when used correctly.
Kansas City police stops actually work when focused on neighborhoods where gun violence happens.
@ketmorco @emilymbender is that a KC specific program, or a style of police stops? I'm unfamiliar with the details. Can you point to something with more details and analysis of how/why it works?
I'd actually say that's a note for hope. It's easy to become depressed if there are no viable solutions, being stuck between a rock and a hard place, so to speak. When viable alternatives are known, there is at least a compass needle pointing at where to go, even if there is no map for how to get there.
@DenialShown @emilymbender i say ironic, because now they're (mis)used as tools of predjudice and oppression (surprise!)
I can't find the write up, but the gist is that violent crime tends to be narrow - that is, it's not (usually) random.
So finding where the targets/sources are, stopping for almost any pretext and searching the cars has an impact.
But then they bragged about it and everyone missed the measure part and just stop & kill PoC, mostly.
@ketmorco @emilymbender reminds me of the "healthy diet" examples that @nntaleb talks about. A diet in a culture with long, healthy lives is not just about what is eaten. What is avoided, how much, and when the food is eaten is probably more important. The patterns and practices around the activity are what is key. It takes a holistic analysis instead of a reductionist analysis to truly understand and translate what works.
There are no shortcuts or free lunches.
@DenialShown Yes, the concerns of people living through a gun violence epidemic are valid and extremely important. But I don't believe that this proposal (scaled, automated surveillance) is coming from those people. It's coming from others who want to make a buck by selling AI Snake Oil.
And yes it is worse that people watching CCTV: scale for one thing, automation bias for another.
@emilymbender So then, do smart and insightful people with understanding of the potential problems and complications of even the most well-intentioned systems have an ethical obligation to engage directly with the civic leaders who seem to be taking a cautious approach to testing a system, probably with little guidance other than their own limited understandings and the voices of the company that is selling the tool (whatever their motivations)?
@DenialShown Yes, we have an obligation as citizens to engage with our elected leaders and to bring our expertise to this.
I *don't* think that the approach described is a "cautious approach to testing". It's a live beta of a system that will impact life & death decisions....
@emilymbender Then it sounds to me like the folks at DAIR have an ethical obligation to get a meeting with the SEPTA leaders calling the shots, and really work with them to address these issues. To use the press attention thus far as a foot in the door to direct engagement rather than a media-mediated wedge between world-views would be a good course of action. Obviously just my humble take. Thank you for being engaged.
@DenialShown No, the folks at DAIR get to decide what their work is. @alex has done her duty in talking to the press. I'm working to raise awareness by posting on social media, hoping to get word to folks in Philly.
You don't get to assign tasks to DAIR. DAIR isn't responsible for chasing down every last one of these in the world.
@emilymbender @alex Definitely not trying to assign tasks. Good luck with your work. It's important work.
SpeakerToManagers@emilymbender
Aside from all the issues of reliability of detection, overuse of the system in non-white/lower class neighborhoods, and potential harms of AI algorithms, systems currently in operation have been used to record voices, and the recordings have been used in court (per EFF). This is clearly unconstitutional overreach.
Aside from all the issues of reliability of detection, overuse of the system in non-white/lower class neighborhoods, and potential harms of AI algorithms, systems currently in operation have been used to record voices, and the recordings have been used in court (per EFF). This is clearly unconstitutional overreach.
Leszek KarlikNo of evaluation of false positives in a machine learning system is just flabbergasting.
Jeremy Kahn@Leszek_Karlik @emilymbender OMG the First Rule of Metrics is to outline your counter-metric
(said differently: figure out how to Monkey's Paw perversely game your metric, measure to be sure you're not doing that)
@Leszek_Karlik @emilymbender and the obvious Monkey's Paw move here is to call in the guns EVERY TIME
that is the homicidal logic we see from "mad AIs" in SF all the time
"i will reduce human suffering by removing all the humans"
@trochee @Leszek_Karlik @emilymbender based on what i can tell these are not CV/AI experts who necessarily know all of the best practices. These are domain experts who have built a tactical tool, like building a tourniquet out of sticks and a t-shirt. It has a very old-school tinkerer/hacker vibe in a way. They are building something to solve a problem they are familiar with and probably cobbled together just enough coding and ML to make it work; from the complete other direction of a computer scientist who understands the intricacies of the tools and techniques they are using.
Maybe they would be willing to hire you for some consulting.
@DenialShown @Leszek_Karlik @emilymbender it's not a matter of best practices.
I'm objecting to the application of AI for these purposes altogether, especially because the existing AI systems are widely known to be amplifiers of cultural, race, gender, and language bias.
More like building a garrote than a tourniquet TBH.
@trochee @Leszek_Karlik @emilymbender i understand what you are saying (at least i hope i do). I don't necessarily disagree, but I'm not fully baked enough to go hard in any direction. What I'm saying is both a claim about gaining a richer understanding of the path that led to them doing what they did as well as an analysis of the rhetoric.
In terms of the former, these are people who are trying to address a real, known, and tangible problem. They are attempting to bring innovative solutions, and working in the adjacent possible space of technologies that are immediately available. They are thinking at a very immediate and tactical level.
Along comes a group of academics throwing stones at their work, yet offering no proposals to address the original problem. No evidence has been presented of harm from the innovation, only speculation and accusations. "How dare they look down on us from their ivory towers, when we are at least trying to solve real, known, and tangible problems!?" I'm reminded of a Benjamin Sisko line from ST:DS9 "Well, it's easy to be a saint in Paradise…".
Again, the original concerns about the system under test might be valid. My only feedback, offered humbly is that as i see it the question is largely being framed as "better the devil you know than the one you don't". As far as i can tell it's not a binary choice, though. Engaging with the creator and/or the transit agency offers a third or fourth way where the tool is made better or the folks in charge can call off its use if it's truly bad. To stretch that field tourniquet turned garrote metaphor, teach them how to build the former instead of the latter, or observe the dressing closely enough to tell the field commander when to order "stop!" Telling someone in the next foxhole over that tourniquets are not good is generally not well received when they've got people already dying all around them.
@DenialShown @Leszek_Karlik @emilymbender I'm not convinced i understand everything you're saying, but I'm pretty sure that your perspective is not taking existing systems of power and control into account.
Legislative & "criminal justice" systems in the US and Europe (at least) are already full of friction and loopholes that reinforce existing power dynamics.
So do most AI systems -- and they punch down along the same axes.
@DenialShown @Leszek_Karlik @emilymbender
These "tinkerers/hackers" are not "punching up", they're playing "on easy mode" -- rich people can already get off parking tickets by throwing lawyers at the court until everybody backs down -- it's why SBF isn't currently awaiting trial behind bars
@DenialShown @Leszek_Karlik @emilymbender
I'd be interested in applications of AI that were _counter_ to those systems of oppression, but i don't believe — under the current provenance of the LLM training data and curatorship of the models and their training -- that this is possible.
I don't share your optimism that these are scrappy underdogs punching up against entrenched power. That was the "disrupt" line that sold us into the hands of Uber and the "gig economy"
KatLS@trochee @Leszek_Karlik @emilymbender and the first law of humans is to see where the incentives lie. Are incentives for finding error ? Or just incentives for not finding errors? And of course, follow the money.
@KatLS @Leszek_Karlik @emilymbender yep. Understanding the reward surface is key to understanding system behavior, even if the system is people
Julian@emilymbender I am sure that dataset is well ballanced, and not biased at all with regard to who carries weapons... Those false positives are going to be interesting...
ron salaj@emilymbender oh dear… so, they really expect a bloody AI to “detect” a gun?? already there are plenty of empirical facts that AI does really damages the society, particularly those from vulnerable conditions, in most spheres: education, emplyment, “crime prediction”, public services and welfare, etc.; and despite this more support is given to some hypothetical tests!! #abolishAI
VAWeisman@emilymbender JFC. This is a DREADFUL idea. Might as well start bombing homes. Again! 😱😩🤬
sharp@emilymbender We don't have much and unfortunately SEPTA's leadership is largely made up of suburbanites who are easily grifted.
Worst case we're all counting down the days until someone gets automatically SWATted. Or best case this will be like the shotspotter systems that don't have any affect on homicides, violence, or arrests but create busywork and run the city's budget dry.
@sharp @emilymbender thus. 😑🫤👆
InfoSecSherpa@emilymbender Thanks for posting.
Finnley Dolfin@emilymbender
I propose a test of that system. Everyone should have gun-shaped objects which aren’t guns and see how many false positives they have to follow-up on.
Tree branches kind of shaped like guns, a hot dog falling out of a hot dog bun, holding your fingers in a gun shape pointing at someone while saying “here’s to you, kid”, magazines and newspapers with pictures of actual guns…
The list goes on.
Martin Vermeer FCD@emilymbender Funny thing, Helsinki does not have this problem 🤔
freshfruit4rottingbrains@emilymbender I just happened to see this and live in a septa region. I had the same exact initial reaction especially when at least years ago the bus drivers and conductors all said septa policy was to allow people to exercise their constutional rights on their busses and trains so long as they didn't break leather.