Mastodawn

Ryan Steed (on the job market)Mar 4, 2024

New paper w/ Victor Ojewale, Briana Vecchione, @abebab & Deb Raji!

We interviewed 35 practitioners & analyzed 390 *AI audit tools*, identifying gaps between existing tooling and effective accountability.

https://arxiv.org/abs/2402.17861

Towards AI Accountability Infrastructure: Gaps and Opportunities in AI Audit Tooling

Audits are critical mechanisms for identifying the risks and limitations of deployed artificial intelligence (AI) systems. However, the effective execution of AI audits remains incredibly difficult, and practitioners often need to make use of various tools to support their efforts. Drawing on interviews with 35 AI audit practitioners and a landscape analysis of 435 tools, we compare the current ecosystem of AI audit tooling to practitioner needs. While many tools are designed to help set standards and evaluate AI systems, they often fall short in supporting accountability. We outline challenges practitioners faced in their efforts to use AI audit tools and highlight areas for future tool development beyond evaluation -- from harms discovery to advocacy. We conclude that the available resources do not currently support the full scope of AI audit practitioners' needs and recommend that the field move beyond tools for just evaluation and towards more comprehensive infrastructure for AI accountability.

arXiv.org

Show thread

Ryan Steed (on the job market)Jan 31, 2024

We also find that most published audit work focuses products/models/algorithms — so we echo calls from others to expand audit work to include the entire ecosystem of affected stakeholders.

Show thread

Ryan Steed (on the job market)Jan 31, 2024

I learned a lot from this study — some of the most impactful audit work is done academia (e.g. by journalists or regulators) often using very different methods.

Ryan Steed (on the job market)Jan 31, 2024

Excited to share a new paper with @abebab, Victor Ojewale, Briana Vecchione & Deb Raji

We surveyed 300+ AI audit studies from academia, civil society, govt etc. to understand what work is being done + how it relates to impact & accountability.

https://arxiv.org/abs/2401.14462

AI auditing: The Broken Bus on the Road to AI Accountability

One of the most concrete measures to take towards meaningful AI accountability is to consequentially assess and report the systems' performance and impact. However, the practical nature of the "AI audit" ecosystem is muddled and imprecise, making it difficult to work through various concepts and map out the stakeholders involved in the practice. First, we taxonomize current AI audit practices as completed by regulators, law firms, civil society, journalism, academia, consulting agencies. Next, we assess the impact of audits done by stakeholders within each domain. We find that only a subset of AI audit studies translate to desired accountability outcomes. We thus assess and isolate practices necessary for effective AI audit results, articulating the observed connections between AI audit design, methodology and institutional context on its effectiveness as a meaningful mechanism for accountability.

arXiv.org