Robert Mar 10
Christine Lemmer-Webber

It turns out GenAI code changes are causing serious incidents and outages at Amazon with "high blast radius" https://arstechnica.com/ai/2026/03/after-outages-amazon-to-make-senior-engineers-sign-off-on-ai-assisted-changes/

Junior / middle engineers no longer allowed to push GenAI code to production without senior engineer review

(HT @KimPerales )

EDIT: Better link above than before. Old one is here:
https://www.ft.com/content/7cab4ec7-4712-4137-b602-119a44f771de

After outages, Amazon to make senior engineers sign off on AI-assisted changes

AWS has suffered at least two incidents linked to the use of AI coding assistants.

Ars Technica
Mar 10 at 2:27pmWeb
Show threadChristine Lemmer-WebberMar 10
Here's a different article about the same topic https://www.tomshardware.com/tech-industry/artificial-intelligence/amazon-calls-engineers-to-address-issues-caused-by-use-of-ai-tools-report-claims-company-says-recent-incidents-had-high-blast-radius-and-were-allegedly-related-to-gen-ai-assisted-changes
In wake of outage, Amazon calls upon senior engineers to address issues created by 'Gen-AI assisted changes,' report claims — recent 'high blast radius' incidents stir up changes for code approval

Amazon says it's a routine meeting

Tom's Hardware
Show threadChristine Lemmer-WebberMar 10
And even another link https://arstechnica.com/ai/2026/03/after-outages-amazon-to-make-senior-engineers-sign-off-on-ai-assisted-changes/
After outages, Amazon to make senior engineers sign off on AI-assisted changes

AWS has suffered at least two incidents linked to the use of AI coding assistants.

Ars Technica
Show threadkatMar 10

@cwebber lol.

lmao.

Show threadThe Blue WizardMar 10
@cwebber I suspect there will be a lot of rejection of AI-assisted changes, which will make the managers ask why there are so many rejections, making senior engineers look, um, idiotic? Ugh...not a pretty scene.
Show threadAlexander 😷Mar 10
@cwebber Can't wait for what happens when all their senior devs are burnt out from non-stop slop reviews. 🍿
Show threadGeorge SaichMar 10
@alxndr @cwebber
This! Exactly.
...and when they burnout and leave where is the bench to backfill?
Show threadHydrian6d ago
@alxndr @cwebber Or when all the senior devs age out and industries as a whole havn't been cultivating coding skills for years/decades.
Show threadSemitones4d ago
@alxndr @cwebber New band name: non stop slop
Show threadTimo the timoMar 10

@cwebber "While generative AI does have its uses, especially in specialized fields like medical research [...]"

surely they did not mean generative AI? certainly they mean the more traditional neural networks and other machine learning approaches that have already been used successfully before the LLM hype took off?

Show threadLichoMar 10
@timotimo the AI hype people are very deliberate about mixing the two up. That's why the umbrella term exists. To give people an impression LLM chatbots have superhuman abilities, stealing the achievements from machine learning like they stole the human work for training @cwebber
Show threadnoodleMar 10
@timotimo @cwebber Me quietly pottering away with support vector machines 
Show threadAaron Caskey-DemaretMar 10
@cwebber re. Tom's Hardware article: Why must every story about how genAI sucks end with "While GenAI is definitely useful in some circumstances..." or something similar? Stories about how a Coal plant poisoned a river don't end in "While it's true that coal plants generate useful electricity..." as though that is something that needs reinforcing.
Show threadI Voted for McGovernMar 10
@cwebber
Who could have predicted this?
Show threadJocelynephiliac Mar 10
@cwebber is there a link thats readable without paying for a subscription?
Show thread^[[200~MichaelMar 10
@twipped @cwebber https://archive.is/wXvF3
Show threadStéphanieMar 10
@twipped @cwebber Does this work? https://archive.is/hLd8X
Show threaddisorderlyfMar 10
@Stephanie @twipped @cwebber stuck in an infinite captcha wall with these links. It seems to silently fail and ask me to fill another.
Show threadCM ThiedeMar 10

@Stephanie @twipped @cwebber Yes! Gets more granular with the internal shit storm brewing with this one too: https://archive.is/gYG86

I've been there at various companies as they try to expand at all costs, being punished for being human is a good time to say fuck off, let your robots do the work.

Show threadChris AdamsMar 10
@twipped The ArsTechnica link has ads but does not require a login: https://arstechnica.com/ai/2026/03/after-outages-amazon-to-make-senior-engineers-sign-off-on-ai-assisted-changes/
After outages, Amazon to make senior engineers sign off on AI-assisted changes

AWS has suffered at least two incidents linked to the use of AI coding assistants.

Ars Technica
Show threadHazelnootMar 10
@cwebber if only we could have predicted this ​​
/sarcasm
Show threadThe Witch of Crow BriarMar 10
@hazelnoot @cwebber totally unforeseeable consequence of replacing the light switch with a giant machine that sometimes spits out candles and sometimes spits out candle-shaped sticks of dynamite!
Show threadCharles U. FarleyMar 10
@cwebber Did they seriously think using AI meant that they could just skip code review?
Show threadT Alex BeamishMar 10

@freakazoid @cwebber This. Getting an AI to write your code is equivalent to a) an insanely over-confident intern or b) a seasoned developer has gotten heavily into the psychedelics. Both choices are questionable, both need serious code review.

And in the end, are you really ahead?

Show threadKwaze KwazeMar 10
@talexb Just not reviewing anything (read: glancing and saying "looks good") is how any of these people are able to say with a straight face this stuff makes them faster.
Show threadPhil Burg (he/him)Mar 10

@kwazekwaze @talexb I listened to a podcast guest two weeks ago who was very pleased to say his company now does exactly that.

"We weren't doing real code reviews before, and now all our code is written by Claude our QA team can't keep up - so I let them all go" or words to that effect.

I'm sure it'll turn out well for their customers.

Show threadT Alex BeamishMar 11

@Standard_Phil @kwazekwaze

Oh .. my goodness. There are so many ways that this could end badly for this company.

And letting all of QA go? That's the last line of defense for a software company. Crazy.

Show threadPhil Burg (he/him)Mar 11
@talexb @kwazekwaze I don't often find myself yelling at the car stereo when I'm listening to a podcast, but on this occasion I had some things to say (nobody was there to benefit from my insight, sadly)
Show threadCharles U. FarleyMar 10
@talexb As someone with ADHD who frequently stalls out any time work gets tedious, yes, I'm definitely ahead. Would love to be able to work on stuff that's always so fun I don't stall out on it, but sadly I need to actually get paid so I can keep my family fed, clothed, and housed.
Show threadT Alex BeamishMar 10
@freakazoid Delighted to hear that it works for you - I'm just commenting that it's imperative to check work produced by an AI.
Show threadRussellMar 10

@freakazoid It sounded like the change is to requiring multiple senior reviews instead of just one. There's so much pressure on and chaos in the process that the only way to seem remotely productive is "LGTM 🤞".

They probably are using LLM agents to supplement the review process itself (they were at my now previous place, and people who can override human review required checks have started doing so). So, AI creates, AI reviews, and a human (gotta have a scape goat) clicks "merge".

Show threadCharles U. FarleyMar 10
@zimzat Yeah, I'm definitely seeing the pressure. AI -> more review requests + more pressure to be productive -> rubber stamps. We're in a world where reviewing a PR now often takes longer than writing it. And of course it's the person who wrote it who gets the credit. Which is going to need to change; in AI-world, author and reviewer should really get equal credit.
Show threadCharles U. Farley6d ago
@zimzat Oh, also, in my experience, AI is worthless for review of AI-generated code; it will only catch the same things that it would have caught had the PR author used the review agent themselves, which will in turn be using the PR description the agent probably wrote, and thus will miss the same sort of mistakes the agent would have made to begin with. The result is that if the author is doing the bare minimum that they should be, the review agent won't find any issues.
Show threadENIGMATICO  Mar 10
@cwebber @KimPerales For nobody's surprise.
Show threadRich Puchalsky ⩜⃝Mar 10

@cwebber

Just have senior engineers write all the code. Probably easier than reviewing AI code.

If that gets to be too much work, just declare that everyone is now a senior engineer.

Show threadPaul_stilgarMar 10

@richpuchalsky @cwebber

Ask a ai to write a program in any language ( cobol, java , python , c,…) which divide two inputted numbers. All of them check if the second number is 0 to avoid a division by zero but NONE checks if the inputted strings are numbers . And this is a basic control .

#ai

Show threadderptronMar 10

@cwebber @KimPerales

Why am I not surprised that Amazon is just now creating a policy of senior code review?

Show threaddeechMar 10
@cwebber @KimPerales I'll bet $100 the incidents would have happened even with sign off. No one wants to review slop.
Show thread"Musty Bits" McGeeMar 10
@deech @cwebber @KimPerales I never know what the actual technical name for it is but I swear there's an attention blindness or fatigue that comes with this sort of thing - like FAA employees deal with. You can't be vigilant constantly or actually critical when 95% of it is fine
Show threadZack WeinbergMar 10
@arichtman @deech @cwebber @KimPerales Yeah I can't remember the official psychology term either but it's real. Same phenomenon as it being hard to notice typos. And, in fact, same phenomenon as drivers not seeing bikers and walkers, which may be a better way to highlight the safety concerns here.
Show threadBadly-read literary snobMar 10
@zwol @arichtman @deech @cwebber @KimPerales sounds like what Ironies of automation is about. Very short paper, not about programming but relevant
Show threadWouter LindenhofMar 10

@arichtman @deech @cwebber @KimPerales

There are a few terms:

- Attention fatigue (due to long exposure)
- Vigilance decrement
- Habituation

But the funniest one is the “autopilot effect” or “cruise control effect”.

Show threadParker MorseMar 10
@DevWouter @arichtman "Automation bias"
Show threadDec [{()}]6d ago

@flashesofpanic @DevWouter @arichtman
"Children of the Magenta Line"

https://www.aopa.org/news-and-media/all-news/2023/march/flight-training-magazine/always-learning-magenta-line

https://www.pbpt.org/captains-corner-19/

https://airfactsjournal.com/2020/09/stepping-down-in-automation-the-real-lesson-for-children-of-the-magenta-line/

https://vimeo.com/groups/364219/videos/159496346

Show threadsabikMar 10

@DevWouter @arichtman @deech @cwebber @KimPerales
There's also

- Automation bias — a tendency to trust the computer

- Anchoring effect — once one option is seen, it's harder to come up with and evaluate other options

Show threadTed LemonMar 10

@deech @cwebber @KimPerales

Ah, but then there would have been a senior engineer to blame, and it would no longer be the LLM looking bad...

Show threadWoke Leftist TrashMar 10

@[email protected] @deech @cwebber not a matter of “if” it would happen in my mind, but when.

Human review is a limited control gap in the system of pushing changes to a production system and has limited utility in preventing errors.

Most empirical studies I’ve read seem to agree that human code review has a marginal effect on error rates that disappears of the reviewer had read a couple hundred SLOC in the last hour.

CEOs aren’t engineers and most software developers aren’t either, IMO. Liability is very screwy in our industry.

Making developers sign off on changes they can’t even hope to vet is a disaster waiting to happen.

Show threadBeing Left Behind EnjoyerMar 10

@agentultra @deech @cwebber

Software developers aren't engineers unless they're also actual engineers in another field (e.g. civil engineering) that requires certifications. Fields that come with requirements such as engineering ethics.

Show thread😀🚲Mar 10
@thomasfuchs @agentultra @deech @cwebber and yet we have traffic Engineering, a licensed profession using cars to kill people.
Show threadWoke Leftist TrashMar 10
@enobacon @deech @cwebber @thomasfuchs true. The better ones, like those in Amsterdam, try to avoid that.
Show thread😀🚲Mar 10
@agentultra @deech @cwebber @thomasfuchs I don't believe traffic Engineering is a regulated profession is the Netherlands, but maybe someone signs and seals the plans, or they just have better design manuals and don't let politicians meddle in the lane widths or number of lanes. What we have in the US is a sham profession.
Show threadWoke Leftist Trash4d ago
@thomasfuchs @deech @cwebber there are countries with professional engineering organizations that do license software engineers (Canada, maybe some EU countries).
Show threadMarkus TackerMar 10
@deech @cwebber @KimPerales But do you want to get fired because of it?
Show threadcraignicolMar 10
@deech @cwebber @KimPerales I'll bet they were signed off. The audit said they were reviewed...
Show threadAlberto de MurgaMar 10
@cwebber @KimPerales None saw this coming 🙄
Show threadSandrockcstm, in space!Mar 10
@cwebber I love that Amazon just admitted that high-risk changes were pushed to production by Jr Engineers with no review by Senior staff. There's so much to unpack here.
Show threadarceuthobiumMar 10
@Sandrockcstm @cwebber why does it sound like “Amazon has begun requiring basic due diligence”
Show threadJennyFluff  💾Mar 10
@cwebber @KimPerales AmazonSlopService