void_ptrApr 13, 2023
requiem šŸ¦«

Twice now Iā€™ve experienced the fallout of bugs in my coworkers code and when I looked into it the bug was introduced by Copilot.

Think about that for a second.

Iā€™m trying to accept that everyone I talk to at work about these systems (I wonā€™t dignify them by using the term ā€œintelligenceā€) ignores my warnings and treats me like a fool for refusing to use them, but now I have to clean up the mess others make by trusting these things.

This isnā€™t sustainable.

Apr 12, 2023 at 4:30amWeb
Show threadmav Apr 12, 2023
@requiem I don't disagree but at the same time, humans engage in a truly shocking number of completely unsustainable practices that we then sustain for decades or centuries, despite them being deeply unsustainable
Show threadrequiem šŸ¦«Apr 12, 2023
@mav very true
Show threadRenƩeApr 19, 2023

@mav We generally do this by throwing cheaper resources (tooling, workarounds, docs, more rigid QA that slows things down) at the problem and discounting the opportunity costs of fixing it / overemphasizing the sunk cost of the original work.

@requiem

Show threadSatsumaApr 12, 2023
@requiem youā€™re not the only person noticing this, unfortunately: https://arxiv.org/pdf/2211.03622.pdf (TL:DR, study participants who wrote AI assisted code wrote code which contained more security vulnerability in tests *and* self-assessed their code as more secure, in comparison to participants who wrote their code independently)
Show threadrequiem šŸ¦«Apr 12, 2023
@Satsuma I knew this would be bad, but I didnā€™t think it would get this bad this fast.
Show threadRenƩeApr 19, 2023

@requiem The grey-gooifying of the internet Commons will happen shockingly fast now.

@Satsuma

Show threadmjfgatesApr 12, 2023
@Satsuma @requiem Some classic Dunning-Kruger there. The machine is 100% confident!
Show threadAn Inhabitant of CarcosaApr 12, 2023
@Satsuma @requiem Dunning-Kruger Effect As A Service
Show threadclacke: exhausted pixie dream boy šŸ‡øšŸ‡ŖšŸ‡­šŸ‡°šŸ’™šŸ’›Apr 18, 2023
@Satsuma @requiem Did anyone counter that the real problem was not applying enough AI-assisted debugging to the AI-assisted writing of bugs?
Show threadrequiem šŸ¦«Apr 19, 2023
@clacke lol
@Satsuma
Show threadPhilippa CowderoyApr 19, 2023

@clacke @requiem @Satsuma Make it AI-assisted verification and we can at least talk seriously...

...about making sure we understand what properties we specified!

Show threadRenƩeApr 19, 2023
@clacke šŸ’€šŸ‘Œ@requiem @Satsuma
Show threadRiley S. FaelanApr 20, 2023

@Satsuma: I dub this effect the Artificial Dunning & Kruger Phenomenon, or AD&KP.

@requiem

Show threadPatrick LamApr 21, 2023
@Satsuma @requiem I was talking to a journalist from Nature Technology and said that I wouldn't trust Copilot-generated code further than I can throw it...
Show threadAndy CooperApr 12, 2023
@requiem they have one purpose. creating the boilerplate for docstrings.
Show threadTimeo_danaosApr 12, 2023
@requiem that reminds me of the opponents to tyres at the beginning of the automobile era because some of them were exploding.
Show threadrequiem šŸ¦«Apr 12, 2023
@timeo_danaos how so?
Show threadLisPiApr 12, 2023

@timeo_danaos @requiem They were right too.

Steel wheels on steel rails work much better, are more efficient and don't have nearly as many catastrophic failures.

Instead of accepting the blatantly obvious, we instead made this ridiculous #CarCentric society.

Show threadRenƩeApr 19, 2023

@lispi314 Yeah this could be a great analogy. By pandering to an oligarch who had an interest in a corporation we paved the commons at taxpayer expense for private gain, ruined the countryside, damaged the health of generations with leaded fuel, skewed community planning for the next hundred years, and now discover what a disaster it was for the potential survival of the environment that sustains us.

Like that, but with information.

@timeo_danaos @requiem

Show threadwoozong šŸ……Apr 12, 2023
@requiem it has entered the 'incubating' status on our corporate tech radar... this is not going to end well.
Show threadRileyApr 12, 2023
@requiem AI IS GOING TO TAKE OVER EVERYONES JOBS
Show threadrequiem šŸ¦«Apr 12, 2023
@dakota5488 only if we work for people who are ok with buggy, non-secure software šŸ¤£
Show threadbenbrown (old account)Apr 12, 2023
@requiem fixing problems with LLM output is going to be every job soon
Show threadrequiem šŸ¦«Apr 12, 2023

@benbrown it is becoming my primary objective to prevent this.

Hopefully, by providing compelling alternatives but alternatively, by any means necessary.

Show threadbenbrown (old account)Apr 12, 2023
@requiem the gold rush mentality has already taken over
Show threadTrƶglƶdĆætApr 12, 2023

@benbrown @requiem

it's not about that, it's about the copilot litigation, eu "ai" regulation that's being hashed out and so on

they don't care about money at this stage, only to undermine legal proceedings and get free labour out of the less educated

Show threadbenbrown (old account)Apr 12, 2023
@troglodyt @requiem when has undermining regulation and abusing labor not been about money
Show threadSoftwarewolfApr 12, 2023

@requiem Using LLMs to generate code like this is just going to take all the fun out of programming and leave developers with having to troubleshoot even more code.

Troubleshooting someone else's code sucks, and it's even worse when you can't sit down with them and ask them what their thought process was, because they aren't a sentient being.

Show threadrequiem šŸ¦«Apr 12, 2023

@faoluin This is one of my primary concerns. I've seen other trades and skills die through patterns like this, and in some cases they were replaced by something superior, but in as many (or more) they were replaced by something that's worse in every way except making a few people rich.

Programmers who rely on this will become worse programmers, and people who use the work will suffer. Those programs will then get consumed into these models and the results will get even worse.

Garbage in, garbage out.

Show threadDave MackeyApr 12, 2023
@requiem I don't think LLM's as dev tools are going away but I do think there is a lot of growth of all of us in understanding how to use these tools in a responsible and ethical manner.
Show threadrequiem šŸ¦«Apr 12, 2023
@davidshq I would argue that it's impossible to use them in an ethical manor. They are built from the bodies of other programmers, and will continue to consume us and displace us while yielding worse software exponentially as they consume their own output.
Show threadDave MackeyApr 12, 2023
@requiem In what way are they built from the bodies of other programmers? Do you mean the use of open source code to feed the LLM's? If so, I agree to some extent. It seems to me that allowing one's code to be consumed by an LLM should be opt-in rather than opt-out...and I suspect we will see many open source licenses updated to include clauses forbidding the use of the code within LLM contexts or something similar.
Show threadrequiem šŸ¦«Apr 12, 2023

@davidshq precisely.

I created a license expressly for this reason a year or so ago, but everyone thought I was nuts šŸ¤£

Show threadclacke: exhausted pixie dream boy šŸ‡øšŸ‡ŖšŸ‡­šŸ‡°šŸ’™šŸ’›Apr 18, 2023
@requiem @davidshq Not nuts, just ineffectual, as so far nobody seems to be challenging their legal theory that training data does not create derivatives in the legal sense.
Show threadwijnšŸ³ļøā€āš§ļøApr 12, 2023
@requiem @kieran my manager is constantly talking about how great it is, but heā€™s a manager and all heā€™s using it for is cleaning up bad bash he writes for reports. I imagine thatā€™s like itā€™s one useful, safe application.
Show threadHugo Slabbert āš ļøApr 12, 2023
@requiem Once again...
Show threadrequiem šŸ¦«Apr 12, 2023
@hugo excellent
Show threadArcturax šŸ¦‡Apr 17, 2023
@hugo @requiem not to mention this thing will try to do exactly what you said. It won't ask questions like "does the client really want X this way? The ask is unclear" or "I think we need to add Y to this code, let's talk to the BSA or client". It can't question what its doing it just does it with no fore, after or ongoing thought.
Show threadJorge StolfiApr 18, 2023

@hugo @requiem

It is worse than that. The program may be the right solution to the wrong problem, or completely wrong, or terribly inefficient, or overly complicated...

Show threadHugo Slabbert āš ļøApr 18, 2023
@JorgeStolfi 100%
Show threadMatt CampbellApr 12, 2023
@requiem OK, so your coworkers should have reviewed the Copilot-generated code. That doesn't necessarily mean they shouldn't use it at all.
Show threadThe Long TailApr 12, 2023
@matt @requiem "Not ALL AI code generators"...
Show threadrequiem šŸ¦«Apr 12, 2023
@sleepyfox this @matt
Show threadCaleb MaclennanApr 12, 2023
@requiem I've seen this too. Several projects I make use of have seen a proliferation of boilerplate looking code that seems to work at first but really just makes a steaming mess out of an otherwise working system. The thing somebody wanted to fix should have been 3 lines but tooling is leading people to add 30 that they don't fully understand.
Show threadJon WoodApr 12, 2023
@requiem My hope is that weā€™ll see things settle down in the near future with people not blindly trusting LLMs so much. Thereā€™s a space for them being used to automate boilerplate, but Iā€™ve seen way too many stupid errors and imagined APIs from Copilot to trust it.
Show threadNicola RomanĆ²Apr 12, 2023
@requiem It seems to me like you are looking at it from the wrong point of view. I am sure you have found many other human-introduced bugs as well, and have also found good code written with the aid of Copilot. The problem is not Copilot (which works well in many cases), it is people misusing it and trusting the code without actually checking (or not being able to check) that it works.
Show threadDavid WeiserApr 12, 2023
@requiem In these instances, does Copilot also introduce unit/integration tests as part of the code change?
Show threadrequiem šŸ¦«Apr 12, 2023

@weiser not that I'm aware of, but these engineers are also using it to write tests generally.

Having a machine you don't understand write your safety net seems to me like asking for it.

Show threadDavid WeiserApr 12, 2023

@requiem I agree with you. I'm just trying to explore what being responsible with a New Fancy Tool looks like.

E.g. If I see a PR with no unit tests, but a lot of code changed, I might push back on approving the PR until there are useful tests. Might a related softeng discipline keep "blind trust of copilot code" out of main?

Show threadjnsApr 12, 2023

@requiem It's vastly disappointing how many people (including here) misunderstand both the problems associated with AI and the capabilities of AI in of itself.

* The current capabilities of AI are over-hyped and over estimated. It's fancy pattern recognition. It is by no means intelligent.

* Corporations are abusing it to steal code, art, and thus get rid of jobs.

* AI output is error prone, and always worse than what a skilled human would produce, but bad quality has never stopped a corporation from cheaping out in order to profit.

It is a multiplier in the race to the bottom. Artists, writers, etc,... are all getting massively screwed by having derivatives of their work stolen while at the same time job offers for the more simpler tasks vanish. As if creative people needed another kick while down. And their customers are being screwed by getting worse products in the end.

We're not "scared" of AI because we think it might go skynet on us. It ain't that clever. It's problematic because it gives corporations another way to exploit us. On a massive scale.

And sorry, but "Should have reviewed the code" is a lame excuse. We all know it's harder and slower to properly and thoroughly review code than to write it from scratch, especially for the trivial stuff AI would be used for at this time.

By using AI you're feeding more data to the companies running them which they can assimilate into their models. By using the tools you are accelerating the problem and actively making the world worse.

There is just no reason and no excuse to use AI. Just don't.

Show threadrequiem šŸ¦«Apr 12, 2023

@jns this has been my argument for years: Iā€™m not afraid of AI getting too smart and hurting us, Iā€™m afraid of giving dumb AI enough power to hurt us.

We crossed that line at least 5 years ago when we started letting AI dispatch men with guns.

Show threadpotatoxelApr 17, 2023
@jns @requiem i run local llm models for fun. they silly sometimes. and generate top tier comedic code.
Show threadMakerGeekApr 17, 2023
@jns @requiem I'm curious whether you think there is any place for such systems? I agree that there are many problems in the theft of IP. And certainly the output of anything complex can be questionable at the moment. Though perhaps not more so than the average Google search brings up.. But as an interactive teaching system, that does a good job of recognising what you're saying/asking and producing some explained output. I think it has great potential. At least I've found value in it.
Show threadjnsApr 17, 2023

@makergeek @requiem I think that's asking the wrong question. One can always dream up use cases, but the question we should be asking is does the benefit we get out of those use cases weigh up against the potential (and real) harm.

For instance, I don't work on AI projects, not because I can't think of any use cases, but because it would be endorsing the current hype surrounding it. Companies who are currently using AI for profit rely on that hype in order to get more funding.

So the question is, ultimately, is what I'm doing, ultimately benefiting or harming society. And when it comes to using or developing AI projects, any time I make that balance, it heavily shifts towards harm.

Show threadMakerGeekApr 17, 2023
@jns @requiem interesting. Such a broad topic I guess I can see what you're saying. I tend to be something of a techno-optimist. My dabbling with the LLMs has so far been quite positive. But I do see the dangers. Either way I'm not sure the genie is going back in the bottle now.
Show threadrequiem šŸ¦«Apr 17, 2023

@jns very well said.

This is the same reason I stopped designing weapons systems in middle school, because I donā€™t want a world with more ways to kill people, even if thereā€™s a lot of money in it and the engineering is interesting.

@makergeek

Show threadMakerGeekApr 18, 2023
@requiem @jns I think the moral case in weapons systems is somewhat more clear cut. But tools that can be used for good or ill are tricky. I personally don't fly because of the environmental impacts. But should flying be off limits for all usage? Lots of tools have been developed that did a great deal of harm and a great deal of good. Are we better off leaving the development of such tools to people that don't fret about the harms side of the equation?
Show threadFrancis šŸ“ā€ā˜ ļø GulottaApr 18, 2023

@makergeek @requiem @jns I think yes itā€™s better to let it flame out on its own but Iā€™m not sure.

ā€œThe greatest minds of my generation are working on spam filtersā€ - William Gibson

and then it was ad networks and then it was crypto, and now maybe itā€™s AI. Iā€™ve had engineers quit to go join boiler room crypto ventures.

Show threadFrancis šŸ“ā€ā˜ ļø GulottaApr 18, 2023
@makergeek @requiem @jns And itā€™s all still out there but much of it has ā€œfailedā€ and faded away. The technological shift however is slow and remains despite the cash grabs. We now have blockchain config managers, and data science enjoys what was made for spam and clicks. I donā€™t pretend that the money didnā€™t influence the technological progression but I donā€™t think I couldā€™ve changed itā€™s direction with my involvement.
Show threadrequiem šŸ¦«Apr 19, 2023

@reconbot I hope youā€™re right about it being another fad or whatever term we should use for these things.

@makergeek @jns