Mastodawn

Hey, I need a bit of moral compass here. We can agree that using AI to generate code is bad, no questions asked. The environmental and societal impact is awful, the skill loss is real etc.

But! What about security? The past months have shown that AI has become actually good at finding real security issues which have been missed for years in high-profile software.

So not using AI for that might actually expose users to undiscovered security issues which are then found and exploited by more ruthless people.

In light of that: How would you like the maintainer of your favourite Open Source projects use AI?

Boost the hell out of this please.

	Not at all
	Exclusively to find security issues
	To find security issues and for code review

Poll ends at Jun 7 at 7:38am.

@jssfr that's a tough one. It would be easier to decide, if the ethical impact of using AI would be less dramatic, but as that's not the case I'm altering between not using at all and using it only for the security stuff

mirabilos🐈‍⬛8h ago

@jssfr unethical thieving planet-burning fashtech is unethical thieving planet-burning fashtech, and the ends cannot possibly justify that

@jssfr compromise; you can use AI for code review but every time you do, you have to throw one (1) rock at a billionaire

@jssfr Even for that use-case I still would only use open weights models run in a way that doesn't finance big-tech.

(My moral says that scraping/impact has already been done, and at least for security/reviews there is less of the whole stealing going on since it doesn't create code directly used in the end-product)

@jssfr Will it even be economically feasible for small non-profit open source projects once the price catches up to the cost of running those models?

@jssfr Wait, this reduces to another "use it or be left behind" argument, doesn't it?

kaffeebär 🐻5h ago

@jssfr Not at all. Its always this "for more security"-excuse to use bad tech.

@jssfr see the problem is now that these tools exist and do work for vulnerability research (if guided effectively). There is now another thing against open source developers. This is by design: open source bears the cost in terms of training by theft and also the products of these tools.

I would argue that the solution to this is for developers to treat security bugs as normal bugs until receiving funding from companies to process them privately. At the end of the day, the same companies which yell and cry about vulnerabilities in the "open source supply chain" are the cause of this crisis. They could have contributed more to stability and safety in these projects before, and they should pay the cost now. The devs themselves are under no obligation to use these tools.

Stefan Eissing 5h ago

@addison @jssfr I agree. There is no obligation for a FOSS project to run a security program.

There is no obligation to run LLMs either. The primary idea of FOSS is to do something fun and share it with others, without strings attached.

Forbidding use of LLMs (or shaming) can easily become a witch hunt. How to defend against someone accusing you of "illegal" LLM use? Also problematic.

Anna Miriamsdochter 5h ago

@jssfr can we have much more slender algorithms with much less of the ethical drawbacks if they’re specialized for this?

The Psychotic Network Ferret 5h ago

@jssfr Any use of AI is driving the construction of massive, resource consuming data centers. Any use of AI is legitimizing the use of AI in general. It doesn't matter if you found a use for it that actually works, it doesn't change the fact that AI/LLM is being used to fill the Internet with misinformation, trash, and propaganda, all while utilizing an extreme amount of resources and passing the external costs onto unwilling participants.

I think it should be obvious to anyone that the price for finding more security issues is actively making the world a shittier place.

@jssfr @bert_hubert I voted “Not at all”

My real answer would be that I wouldn’t blame any project for using AI this way if they so choose (whereas I do blame them for vibe-coding); I think you can use it to find some kinds of security issues and that would be useful.

But the poll asks explicitly whether I’d *like* the projects to adopt these, and there my answer is no; projects are already strained for resources, monetary and time. Don’t divert resources you don’t have.

Mark W. Gabby-Li 🐌4h ago

@jssfr Are you already using all the existing static analyzers you can get your hands on? If not, why start with LLMs?

Have you tried having a human audit your code for security issues?

How do the costs compare, if you were actually paying these LLM companies for their compute profitably?

Arthur van der Harg 4h ago

@jssfr The way I look at it: AI is going to be used to find vulnerabilities, period. Because AI analysis does find vulnerabilities. The choice is between the author finding them first, or waiting until bad actors exploit them. Bad actors don’t give a rat’s patootie about your ethical stance. For them, AI is a tool that works, so if you refuse to use AI to find vulnerabilities, the more they will find.

Jasper Vinkenvleugel 4h ago

@jssfr from a purely ideological standpoint I’d say no, and I hate the fact that some projects are almost forced into using AI for security because others are already doing it while ignoring all norms for responsible disclosure. So with that being said, I think we need to have some level of flexibility while we think about proper ways to regulate the usage of AI, and make the ecosystem of open software more resilient to it.

Paco Hope 4h ago

@jssfr Secure software doesn’t matter if everyone dies to climate change. Until we talk cost AND benefits, there’s no point in talking benefits. How much climate change and damage should we accept to use AI for “security” purposes?

And who is writing the fixes for the “security” findings? People or AI? How does this not lead right back to writing code with AI?

And why only security? Why not performance, also? Why not reliability? Why not portability? Accessibility? Why would we consider this one non-functional requirement—security—and none of the others?

A poll like this is easy to ask because all the hard things are silently resolved in each respondent’s mind. If a project tried to somehow limit AI use to “just security” it would fall apart quickly. It’s not possible to define or do.

Cassandrich 3h ago

@jssfr The past few months have not "shown that AI has become actually quite good at finding real security issues".

It's shown that if you have a New Thing being hyped as a way to find security issues with millions (much less billions) of dollars being poured into its use to find security issues, you're going to find a lot more security issues just because you're looking.

Same thing happened with fuzzers when they were hot. Once they weren't, the findings dried up.

Virtually any other way of spending these billions of dollars on finding security issues would find orders of magnitude more. Including the obvious one: paying a thinktank full of hackers 6-sigure salaries to sit around reading code looking for vulns.

everything_bagel

@dalias @jssfr whether AI is good at finding real security issues or not has been shown to be irrelevant by recent events. Just look at what’s been going on with the Linux kernel, where, real issues or not, the developers were so completely overwhelmed by the absolute flood of vulnerabilities being reported that they had to just say no to accepting them at all. And now look at rsync, which has absolutely introduced serious problems and doubts into a bedrock piece of computing infrastructure, as a direct result of AI use.

Unless and until AI can be used in a responsible manner, it’s doing more harm than good to software development, even if it is technically capable. The ethical and environmental concerns are absolutely part of that, but importantly, even if those were solved tomorrow, it would still be demonstrably bad to continue as we are right now.

Eric Brombaugh 3h ago

@jssfr I might go so far as to dispute the initial premise that LLMs are good at finding security issues. Yes there have been some high profile cases of this discussed in the media, but when I dig deeper there seems to be some disagreement about how important these vulns really are. Generally they're fairly inconsequential stuff and the AI providers are touting them for marketing. Couple that with the fairly high false positive rate and the utility may not be there.

Ember is so tired of people.

@jssfr only looking at the very few actual issues that have possibly been found with LLMs ignores the hundreds of times more useless time-wasting non-issues that it 'finds'.
do not trust the LLM proponents to not lie about how much it helped (or much more likely, didn't) or how many real issues it maybe found compared to time wasters. they will lie.

even if it were somehow miraculously true that LLMs can be a net positive towards finding security issues, it would not in any way be worth all the immense harms (also any amount of LLM use will result in deskilling (and dependence/addiction), finding security issues is a skill after-all)

Pay them enough that they can spend time doing code reviews and finding those vulnerabilities or so they can afford an external review.

Peach “Fuzz” Fae 2h ago

@jssfr @paco AI used to be called ‘expert systems’ when it was trained on limited, not stolen data sets. It could be quite good at a number of thing like diagnosing rare diseases. So, using an expert system would be ok in my book. Using an LLM in a narrow way with stolen training data would not.

Anton Gerasimov 2h ago

@jssfr I am not a moral compass, as I am mostly interested in losing skills and not properly building mental models, not the other stuff - or rather I don't think that individual boycott is an adequate answer to those issues.

So anyway, from the point of view of professional ethics, I would say LLM code reviews are beneficial a kind of probabilistic linter on top of other static and dynamic analysis, but not as a replacement for human review.

@jssfr the environmental argument against using LLMs depends on the scale of the project and the implications of a breach. If an LLM can help avoid millions of computers getting hijacked to mine crypto and we can assume the attackers use LLMs to find such vulnerabilities, a few prompts per month would be worth it.

I'd rather have a world without the kinds of LLM companies we have today, but a full boycott doesn't always have to be the best choice.

@jssfr when you say that the issues have been missed, does that imply that somebody looked?