I'm a computer science professor at Princeton. I write about AI hype & harms, tech platforms, algorithmic bias, and the surveillance economy.
I've been studying decentralized social media since the late 2000s, so I'm excited to use and write about Mastodon at the same time.
Check out this symposium on algorithmic amplification that I'm co-organizing: https://knightcolumbia.org/events/optimizing-for-what-algorithmic-amplification-and-society
| Website | https://www.cs.princeton.edu/~arvindn/ |
| Substack: AI Snake Oil | https://aisnakeoil.com/ |
| Book: Fairness and machine learning | https://fairmlbook.org/ |
interesting new paper on the feasibility of using LLM agents for computational reproducibility checks by Narayanan and colleagues @randomwalker
AI agents have the potential to aid users on a variety of consequential tasks, including conducting scientific research. To spur the development of useful agents, we need benchmarks that are challenging, but more crucially, directly correspond to real-world tasks of interest. This paper introduces such a benchmark, designed to measure the accuracy of AI agents in tackling a crucial yet surprisingly challenging aspect of scientific research: computational reproducibility. This task, fundamental to the scientific process, involves reproducing the results of a study using the provided code and data. We introduce CORE-Bench (Computational Reproducibility Agent Benchmark), a benchmark consisting of 270 tasks based on 90 scientific papers across three disciplines (computer science, social science, and medicine). Tasks in CORE-Bench consist of three difficulty levels and include both language-only and vision-language tasks. We provide an evaluation system to measure the accuracy of agents in a fast and parallelizable way, saving days of evaluation time for each run compared to a sequential implementation. We evaluated two baseline agents: the general-purpose AutoGPT and a task-specific agent called CORE-Agent. We tested both variants using two underlying language models: GPT-4o and GPT-4o-mini. The best agent achieved an accuracy of 21% on the hardest task, showing the vast scope for improvement in automating routine scientific tasks. Having agents that can reproduce existing work is a necessary step towards building agents that can conduct novel research and could verify and improve the performance of other research agents. We hope that CORE-Bench can improve the state of reproducibility and spur the development of future research agents.
I'm ecstatic to share that preorders are now open for the AI Snake Oil book! The book will be released on September 24, 2024.
@randomwalker and I have been working on this for the past two years, and we can't wait to share it with the world.
Preorder: https://princeton.press/gpl5al2h
Ted
Ulrike Hahn
Sayash Kapoor
Frederik Borgesius
Knight First Amendment Inst.
Alex Stamos
Mike Masnick ✅