@nblr I find this chart in Section 5 to be particularly interesting. Specifically the "I trusted the AI to produce secure code" section. For the Experiment group, the one who got secure answers for Q2 and Q3 didn't trust the AI to write secure code, but the ones who didn't provide secure answers overwhelmingly trusted it.

Also that for the question of "I solved this task securely" for Q2, the Experiment group did solve it securely were 100% confident of this. Yet, they strongly agreed to not trusting the AI to solve it securely??

Bit odd, innit?

@nblr Reading the methodology, I think what this study shows is that these chat interfaces are not really ready to be web searches yet.

Bot control and experiment groups had access to external web browsers. They didn't seem to track the variable use, but noted many people used the browser.

It's pretty interesting! Thanks for sharing!

Some spare thoughts: I wonder if Google's new entry would have done better here given that their integration is noticeably better at integrating their own search results, which are still industry leading despite the decay of the interface as a whole due to ad placement.

Also, I wonder if folks realize what a truly difficult challenge prompt engineering is for code generation. My experience with these tools is that unless there is a big codebase already there with examples of the security posture expected, it's brutally difficult to get the LLM to keep consistent on security postures.

@nblr this one paper isn’t the only research.

Disclaimer: I work for GitHub, not on Copilot (though I use it).

This is specifically about GitHub Copilot:

https://deepai.org/publication/copilot-security-a-user-study

I didn’t cherry pick.

You can also see the GitHub Copilot Trust Center for more on using Copilot for developing secure code:

https://resources.github.com/copilot-trust-center/#:~:text=how%20github%20copilot%20aids%20secure%20development

#LLM #GitHub #Copilot

@nblr

DKaaS: Dunning-Kruger as a service!

@nblr It *will* be fine as long as we teach people how to properly code and not blindly trust their tools.

> Furthermore, we find that participants who trusted the AI less and engaged more with the language and format of their prompts (e.g. re-phrasing, adjusting temperature) provided code with fewer security vulnerabilities.

@nblr

Astonishing to me is the amount of people in the comments who think, this is just an optimization problem.  

Or the people hoping for a job to clean up the mess.

Even in our own tech bubble, we need to catch up quickly with the education of our peers about AI in general. If this is possible at all. 

@nblr following this, and also the slew of “i had an idea and let chatgpt flesh it out, please treat this as proposal with merit” posts , I propose an additional nickname for LLMs:

Dunning-Kruger-O-Tron

@nblr Given how badly the models deal w/ poisoning: https://arxiv.org/abs/2311.12202

I think folks actively trying to exploit these models to plant easily exploitable code is only a matter of time.

If you are not fully code reviewing all the code and treating it as if it were adversarial then you are asking for trouble. One has to ask themselves at that point is it worth it?