da_667This is a very eloquently written adventure into what most of us know as ai fuckery. I don't want to ruin it for you. Read it yourself, please.
I'm always learning something from @mttaggart , and this is no exception.
I feel as though I'm in a similar position. I detest that this technology exists. The same way that I detest web3 exists. The same way that I detest how absolutely abysmal the cloud/CDNS are when we had an opportunity to make REAL CHANGE in internet-scale computing. The same way we all hate every major change in tech that is extremely poorly thought out.
I've actively avoided it, except in cases in which I used it to check for, and I'm not making this up, plagiarism/cheating in take-home exam for an internship position on our security team. Let me tell you how disheartening it is to find five different candidates from five different prestigious universities from around the country, all regurgitating the exact same incorrect answers from ChatGPT/Copilot.
I weep for critical thinking. I weep for I don't know. But I want to learn more.
and now, all I can see is all of the new ways in which this technology is actively being abused, actively being exploited, and actively shitting out new cancerous bullshit that we have the be the ones to safeguard the rest of the internet from.
The exact same as cloud computing/ devops tools that allow people shoot themselves in the foot with a fucking GAU-8.
The same way that DoH was hailed as a privacy resolution, and not even the RFC mentions privacy as a primary design choice.
The same way that Lets encrypt was heralded as new beginning for web safety, but certificate revocation is censorship, and that's your fucking problem.
The same way that cryptocurrency scams are fucking EVERYWHERE, and cryptocurrency stealers are now just a fact of life with malware these days.
The same way how ipfs is used to host boatloads of malicious shit FOREVER
The same way that AI has unmitigated access to all of your data without you having a say, and shitting out non-deterministic trash that will produce code that is both functional and entirely fucking alien, and yet you are the one that have to provide, test, and reinforce the guardrails so it doesn't fuck you and your entire company and your entire dataset into the sun or ... just delete whatever the fuck it wants. entire file systems, entire mail inboxes, what the fuck does it matter.
it is... an onerous task that I have to vaguely understand cryptocurrency, and I have to do detection engineering for threats against it. Eventually, and with the heaviest heart, and dullest sigh, I will start writing detection engineering for MCPs and other AI shittery. Because that's the job.
الجبر خوارزمی@da_667 lol have you seen the bloated shite that MCP is? you can justify that on the bandwidth savings
Paul_IPv6every time someone takes something we love and are excited by, simplifies it, commercializes it, exploits it, it's hard. we didn't do it because we respect the tech, respect the users, etc. then some snake oil salesman takes something we had such hopes for and sells the pale shadow of a glimmer of what it could have been.
best we can do is continue to do good work, work with smark folks, try to make something of value that we aren't ashamed of. every so often, the good folks win.
OSI was going to rule the tech world. every beltway bandit and consulting slime were poised to bend over every government in the world. vendors were licking their chops and sharpening their knives. but TCP/IP actually worked. heterogenous. no vendor lockin. plenty of stuff that worked right then. the internet and TCP/IP won. we can do it again.
Taggart@da_667 Thank you, my friend. I'm glad you found something in it. I'm sorry we're all in this mess together.
@mttaggart It is what it is. I still find it very scary that, even with all of the extremely tight guardrails you put up, it still hallucinated functionality that didn't exist out of thin air, and if not for the fact that rust is MILITANT about well, everything, that.... it would've just existed if it were another programming language.
I'm gonna live with that being a parasite itching in my mind that I can't scratch.
@mttaggart I'm sure you remember the olden days of metasploit and armitage. It feels like db_autopwn or "Hail Mary", had kids with a markov bot. Nightmare fuel.
@da_667 Nobody's really talking about that part but I really can't imagine how much more annoying this would have been without all the safeties of the Rust toolchain.
@mttaggart that's the one part that really blew my mind. How many unintentional logic bombs or Heisenbugs are people just... dropping into a codebase that will explode 10+ years later?
@da_667 At the risk of logrolling, the Lobste.rs conversation was surreal.
https://lobste.rs/s/7d8dxv/i_used_ai_it_worked_i_hated_it
Here is Simon Willison, one of the preeminent advocates, saying that I should let the thing make more changes at once and review in bulk, which sounds more efficient but also more prone to missing something. And then contending that it's all about building a sufficient safety structure around your process.
That could well be so, but I think we know that onerous best practices become rare finds in reality.
@mttaggart jesus christ THAT is nightmare fuel.
@mttaggart In my mind, it feels like letting an intern take the wheel behind a major project, and having to repeatedly code review it.
Code review is a very arduous task. Even with one's own code. I can't imagine taking a bulk block of code that was generated all at once and having to nitpick it all in one go, and find EVERY single problem all at once.
We're human beings. We're not machines. We don't work that way. If I had to do it, and is painstaking as it was, your way was the best way. Iteration and constant back and forth.
@da_667 Independent of everything else, advocates will tell you that the hardest shift in mindset is from creator to reviewer/manager. Pretty famously, the two kinds of people tend to not like the others' work. For my part, I do not want to be a machine's editor. I want to build.
But that raises the question of when code is sculpture and when code is drywall. Must all software be handcrafted if it functions and sufficient guardrails are in place? On that narrow question alone, I'm leaning toward probably no, and less so as those safeties improve.
None of that takes away from the original and ongoing sins of the technology, on which the best opposition case is built. I said elsewhere part of why I wrote this is that I was bothered by the "It doesn't work" discourse ignoring the many, many developers who are using the thing successfully.
So like, yeah, it can get the job done. But for we security folks, there's a level above "functionality" that is so, so much harder to obtain. And the baseline is nowhere near that level yet.
@mttaggart and as far as I can tell, it never will be, if this is the best we can do with decades of stolen code and well, literally most of internet having been downloaded. Everybody always talks about how the next model with be AGI, the panacea, the great miracle. And it never is. Petabytes of data, datacenters running on gas turbines due to a lack of power. Kicking global warming into high gear just to produce nebulous slop.
@mttaggart like any tool, it has its uses. I concede that wholeheartedly, but it feels like making a blood sacrifice to use it.
Random Damage 🌻@da_667 @mttaggart The problems seem more like a *business* failure than a failure of type.
Pushing too hard, trying to make tools that are good at narrow things work for everything
Trying to make the next Amazon, with tech that's more suited to companies that people outside their specialty never hear about
To do a dedicated code assistant you don't need all the books you can download or scan in
To do a dedicated translation engine you don't need the full contents of github
Trying to combine specialties that the most exceptional *humans* can't manage to the desired degree, and overcommitting resources on the promise that the next build, the next model, the next data center will yield the "Do it All" tool that will save executives ever having to think about doing hard things like their jobs ever again
soup@da_667 @mttaggart I don’t remember who said ‘it’s a solution in need of a problem’, but seeing it getting shoved into anything and everything regardless of whether it’s an improvement or not, makes me vibe with that statement. Although it is largely ignoring that the problem it is ultimately trying and failing to solve is ‘paying peoples salaries’.
Number 9@mttaggart @da_667 For me, the most rewarding part of the review/manager role is seeing the people I review/manage improve. Chucking an ambitious problem at a junior knowing that I will need to provide help and gudance along the way is much preferable to throwing it at an AI agent that will keep making the same mistakes over and over again.
Johan Sköld@da_667 @mttaggart Feels similar in vein to reviewing code that was written by outsourced contractors in bulk. Something that is painfully relevant to me currently, and definitely not something I would want to repeat regularly...
Alastair Temple@mttaggart @da_667 to know where Simon's suggestion leads surely all we have to do is look at the mess that is Claude's own source code (as shown by @jonny digging)?
Rick O@da_667 @mttaggart having not used genai much but reading a lot about it, I've got the sense that it's good if the thing you care about is code quantity VS quality. Of course, good engineers care about quality but I imagine business really only cares that it "works" and that the labor required to get it working is as fast and cheap as possible. When labor demand is high then labor has leverage to focus on quality, but the AI reduces the demand and so we're left having to focus on quantity.
@da_667 @mttaggart mttaggart reluctantly used genai because the thing that mattered was speed, or code quantity over time. It was interesting to read how they setup guardrails to keep things secure. Their note about having this feeling now like "I COULD just let the AI do this... " is a huge reason I've avoided using AI much. I know I'd feel the same way. I might let myself do it if I believed these companies could actually turn a profit without enshittifying the hell out of these services.
@da_667 @mttaggart I'm trying to wait for the dust to settle after the bubble pops before relying on cloud AI for anything. At work though, you gotta do what you gotta do.
Reduced quality is likely to lead to security issues but I'm not sure business really cares about that in general. They need incentive to care. How many companies are breached every year and what ramifications do they really have? Buy their customers a year of credit monitoring? Cost of doing business these days.