geofflangdaleDoes anyone have the "Big Assembly" or "SuperCoder" benchmark sets? These seem cool (big sets of inputs for superoptimizers) but I don't see any way to download either.
@regehr might know?
John Regehr@geofflangdale sorry no! but these sound interesting so please keep me in the loop
@regehr I can try contacting authors. My guess is the whole thing is a big legally grey license-wise so they didn't want to make a Gigantic Hairball of Redistributed Derived Work (a rather quaint concern these days), but we'll see.
@geofflangdale who are the authors? what paper is this?
@regehr SuperCoder: Assembly Program Superoptimization with Large Language Models is https://arxiv.org/abs/2505.11480
SuperCoder: Assembly Program Superoptimization with Large Language Models
Superoptimization is the task of transforming a program into a faster one while preserving its input-output behavior. In this work, we investigate whether large language models (LLMs) can serve as superoptimizers, generating assembly programs that outperform code already optimized by industry-standard compilers. We construct the first large-scale benchmark for this problem, consisting of 8,072 assembly programs averaging 130 lines, in contrast to prior datasets restricted to 2-15 straight-line, loop-free programs. We evaluate 23 LLMs on this benchmark and find that the strongest baseline, Claude-opus-4, achieves a 51.5% test-passing rate and a 1.43x average speedup over gcc -O3. To further enhance performance, we fine-tune models with reinforcement learning, optimizing a reward function that integrates correctness and performance speedup. Starting from Qwen2.5-Coder-7B-Instruct (61.4% correctness, 1.10x speedup), the fine-tuned model SuperCoder attains 95.0% correctness and 1.46x average speedup, with additional improvement enabled by Best-of-N sampling and iterative refinement. Our results demonstrate, for the first time, that LLMs can be applied as superoptimizers for assembly programs, establishing a foundation for future research in program performance optimization beyond compiler heuristics.
@regehr BigAssembly is mentioned here: https://arxiv.org/abs/2109.13498
Learning to Superoptimize Real-world Programs
Program optimization is the process of modifying software to execute more efficiently. Superoptimizers attempt to find the optimal program by employing significantly more expensive search and constraint solving techniques. Generally, these methods do not scale well to programs in real development scenarios, and as a result, superoptimization has largely been confined to small-scale, domain-specific, and/or synthetic program benchmarks. In this paper, we propose a framework to learn to superoptimize real-world programs by using neural sequence-to-sequence models. We created a dataset consisting of over 25K real-world x86-64 assembly functions mined from open-source projects and propose an approach, Self Imitation Learning for Optimization (SILO) that is easy to implement and outperforms a standard policy gradient learning approach on our dataset. Our method, SILO, superoptimizes 5.9% of our test set when compared with the gcc version 10.3 compiler's aggressive optimization level -O3. We also report that SILO's rate of superoptimization on our test set is over five times that of a standard policy gradient approach and a model pre-trained on compiler optimization demonstration.
@regehr Twitter user Dougallj found some stuff related to SuperCoder. It looks like one could grovel around in it and pull out some asm, fwiw.
