Mastodawn

cchelberg 4h ago

✧✦Catherine✦✧

i'm at a loss of words after reading a paper about reformatting code using an ML model that has a measured statistical quantity A_c which says how often the reformatted code behaves the same as the original

the "ideal" (their choice of words) case is 64.2%

edit: this got popular without me really intending to, so here's why i'm reading research: i want a semantic style transfer tool that can automatically format a patch "the same as the rest of the file / rest of codebase is formatted" without the rigidity involved in black or rustfmt that i find so hostile to my workflow that i refuse to use them. obviously, i want a tool that generates semantically equivalent code 100.0% of time (ignoring source locations or reading from __file__)

✧✦Catherine✦✧12h ago

this isn't satire, this is real research published by IEEE/ACM

disorderlyf 12h ago

@whitequark So let me get this straight, IEEE thinks you should count it as a win if rewriting your code by vibing it has less than 15% better odds than a literal coinflip of reproducibility?

edited for clarity and to fix a typo

Urix Turing 4h ago

@disorderlyf @whitequark IEEE and ACM don't do the research nor they think you to do things, they are publishers that own journals and conferences where researchers publish their work

✧✦Catherine✦✧4h ago

@urixturing @disorderlyf yeah. there are other issues with their models but this isn't one

Deborah Preuss, pcc 🇨🇦12h ago

@whitequark @danlyke so … by "reformatted" I assume you mean aesthetically tidied up, with no change in functionality required?

If I got that right: wtf?

✧✦Catherine✦✧12h ago

@deborahh @danlyke this is what a reasonable person would understand to be "code style", yes

Ken Butler 10h ago

@whitequark @deborahh @danlyke ie, the sort of thing a linter does?

Henning Paul DC4HP 5h ago

@nxskok @whitequark @deborahh @danlyke to be fair, according to the paper, replacing for with while loops and vice versa and the like was also the goal

@deborahh @whitequark @danlyke

No.

"there is no existing work that performs full stylization on an arbitrary piece of code. The most common methods are rule-based linters, formatters, which are limited to a few pre-defined style rules"

✧✦Catherine✦✧5h ago

@mrkeen @deborahh @danlyke I do think that stretching the definition of what "code style" could reasonably refer to until it fits the shape of the research product is a part of the problem here. (Consider that the introduction explicitly refers to the gotofail bug as something the research is supposed to help with, whereas it is plainly evident that it would make that problem only worse.)

🌸 Ⓐnarkitty lumi-nhac

@whitequark is this satire? this sounds crazy

✧✦Catherine✦✧12h ago

@lumi no it's published in the proceedings of IEEE/ACM https://upload.whitequark.org/1774306843-Duetcs_Code_Style_Transfer_through_Generation_and_Retrieval.pdf

🌸 Ⓐnarkitty lumi-nhac

@whitequark i was more so asking if the paper was satire, but i guess looking at it answers my questions as well

@whitequark i can imagine a few cases where reformatting a code could change behavior (mostly related to language constructs that capture source locations) so I think I would be willing to accept as low as 99.99%

✧✦Catherine✦✧12h ago

@porglezomp you'll love Fig. 6

✧✦Catherine✦✧12h ago

@porglezomp there's explanatory text that says the issue with the identifier "found" is that it's rarely used

@whitequark I really love “not changing endl to \n” listed as a style issue when that changes buffer flushing behavior.

✧✦Catherine✦✧12h ago

@porglezomp did you notice that one of the If tokens is capitalized in the output

@whitequark hahaha I did not

jonathankoren™10h ago

@whitequark @porglezomp also cout << “\n”; is not equivalent to << endl; endl flushes the stream. “\n”; does not. They’d need to write << “\n” << flush; to get the same behavior. Which is annoying to write, which is precisely why endl exists!

@jonathankoren @whitequark @porglezomp I just thought I’d take a moment to say that I thought that i was barely coping with things today, but thanks to figure 6 I clearly am not.

I’m not even going to ask about a use case. Taa for now I’m off to disintegrate.

Cassandra is only carbon now 12h ago

@whitequark @porglezomp I'm spitting out my drink at j++ → j--. Holy shit.

Icewind 11h ago

@xgranade
I think the right is the output from running the model on the right code (center being the "desired output"). So it's not changing the semantics of the loop, just not not changing the loop order to match their desired outcome.

Given that loop order can have behavioral impact (and I would never trust an LLM to be able to tell if it did), that seems like the correct behavior to me though
@whitequark @porglezomp

✧✦Catherine✦✧11h ago

@robin @xgranade @porglezomp oh you're right

@xgranade @whitequark @porglezomp
I think reversing the `j` for loop is actually wanted by them? It's labelled "ground truth", and it is a potential valid optimisation

Inga stands with 🇺🇦 🇵🇸2h ago

@sabik @xgranade @whitequark @porglezomp but they also changed the boundaries! "Input" checks all values from 2 to i+2 inclusive; but "ground truth" just trows i+2 iteration out.

@IngaLovinde @xgranade @whitequark @porglezomp
`i` starts from 1 in the "ground truth" version

Inga stands with 🇺🇦 🇵🇸2h ago

@sabik @xgranade @whitequark @porglezomp ah I see, so the new i is just the old one + 1

fuzzyfuzzyfungus 12h ago

@whitequark @porglezomp This looks like it could join the current crop of "DLSS5 off/DLSS5 on" memes.

Ring <3 Rootkitty 10h ago

@whitequark "If" right next to "if"

Lucie / minute 10h ago

@whitequark @porglezomp long live the new flesh

Lona Theartlav 12h ago

@whitequark Huh. Were they actually trying to make it work, or trying to show that it's a bad idea to try to use ML for that task?

✧✦Catherine✦✧12h ago

@theartlav former

Brandon 12h ago

@whitequark “surely it just means differing output structure to accommodate the formatting, right?”

No, it just produces code that won’t compile. In a refactoring tool.

(Haha, didn’t see the post a minute before mine with the exact same snip)

Daniel Carosone 12h ago

@whitequark you know where there's a ready source of additional words? you surely will not regret sourcing additional words.

matthewcroughan 12h ago

Got a link?

✧✦Catherine✦✧12h ago

@matthewcroughan https://upload.whitequark.org/1774306843-Duetcs_Code_Style_Transfer_through_Generation_and_Retrieval.pdf

@whitequark not a paper *deliberately* about genetic algorithms, then?

fuzzyfuzzyfungus 12h ago

@whitequark "Code style generally does not interfere
with the code semantics and executability"; but we present novel methods for it to do so!

Irenes (many)12h ago

@whitequark compare and contrast the Extreme Programming philosophy, in which a code change doesn't count as "refactoring" unless all observable behavior is identical

Peter Brett 12h ago

@ireneista TIL that my philosophy is the same as the Extreme Programming philosophy

Irenes (many)11h ago

@krans @whitequark it was a nice name for a movement, it did a good job of conveying that the goal was radical change

at the time, from what we can tell, none of the people saw it as a labor movement specifically, which is too bad... that might have prevented it from being watered down by successive cycles of consulting and renaming

✧✦Catherine✦✧12h ago

@ireneista i like how it starts with this (left) and ends with "here is a variable we think would be good here. Do you like this" (right)

✧✦Catherine✦✧12h ago

@ireneista starting with "gotofail bad" and ending with making the problem significantly worse, apparently without ever reflecting on this

Irenes (many)11h ago

@whitequark because "the thing we're promoting is incredibly dangerous, and not in fun ways" is not really the thing anyone wants to be cited for

Geoff Wozniak 10h ago

@ireneista @whitequark Now, show me the numbers on the effort to make a rule-based style file compared to this. Because I'm sure that A_c is 100.0 in that case.

✧✦Catherine✦✧10h ago

@GeoffWozniak @ireneista so the problem i'm solving is that while for C++, you have tools like clang-format which are nice and flexible, for Rust you have rustfmt which is rigid and makes your code look like ass. I do not like my code looking like ass but I am also receptive to the idea that introducing as many knobs as clang-format has into rustfmt would make it unmaintainable

Geoff Wozniak 10h ago

@whitequark @ireneista I have not had to deal with rustfmt yet. For clang-format, I work in existing projects and use (very) mildly tweaked variants of the base style for the project.

At the risk of instigating the canonical bikeshed discussion, I am a conformist formatter and have not concerned myself with modifying style all that much. But I agree that clang-format has some bizarre knobs to tweak.

✧✦Catherine✦✧10h ago

@GeoffWozniak @ireneista I view code as art so I find strongly canonicalizing formatters like black to be actively destructive. right now I use Ruff with a 300-line configuration for some of the Python code and I think there's gotta be a better way to approach this that isn't destructive

Irenes (many)10h ago

@whitequark @GeoffWozniak that's our view as well

✧✦Catherine✦✧9h ago

@ireneista @GeoffWozniak based on a discussion with someone who has worked on this problem before we want to try building a diffusion model that captures the whitespace between code tokens and is then able to inject it into a given parsetree, which appears to be a fairly efficient and unproblematic way to do this

✧✦Catherine✦✧9h ago

@ireneista @GeoffWozniak and everything that is best done on a parsetree (import ordering for example) will be done in the parsetree because it ain't broken

Irenes (many)9h ago

@whitequark @GeoffWozniak yeah this is a recurring research topic for us, we've talked with several of our friends about it over the years. just making a parser/generator that properly round-trip whitespace and comments is already a ton of work, alas...

✧✦Catherine✦✧9h ago

@ireneista @GeoffWozniak there's tree-sitter nowadays which I believe should do that (and I think it should be failure-tolerant considering its fairly wide use in editors: nvim, zed, etc)

✧✦Catherine✦✧9h ago

@ireneista @GeoffWozniak my literal first Python project was making a Python parser that fully captures source spans (which wasn't upstream at the time--in 2014 or so), so i'm quite familiar with the topic by now :p

Geoff Wozniak 9h ago

@whitequark @ireneista This sounds a lot like XSLT (or XSLT-adjacent).

kouhai, of the health issues 5h ago

@whitequark @ireneista @GeoffWozniak ~~ah, so python indentation~~

Geoff Wozniak 9h ago

@whitequark @ireneista I very much respect that.

I view code like writing and I will tweak structure and form for far too long sometimes. Layout ends up getting less of my attention.

✧✦Catherine✦✧9h ago

@GeoffWozniak @ireneista I see layout as part of the form, I guess? I write source code files in much the same way as one would write chapters in a book: somewhat self-contained, and intended to make sense when read top-to-bottom linearly and with roughly one full-displayful of contex. so if rustfmt decides to blow up a function call into 20 lines out of nowhere it very much messes with that, for example

Geoff Wozniak 9h ago

@whitequark @ireneista Well, I do have limits.

In my case I spend my time in Binutils and GCC. Do I love the GNU style? No. But does consistency help? Yes. So I demur. But I will restructure things so the single line curly braces don't take over.