Jimenez et al. (Princeton)
datasetactive

SWE-bench: Can Language Models Resolve Real-World GitHub Issues?

Jimenez et al. (Princeton)

View original resource

Princeton benchmark of 2,294 real GitHub issues from 12 Python repositories paired with expert-written test patches. Measures whether agents can read codebases, edit files, and produce patches passing the hidden tests that fixed each issue.

Tags

agentic AIevaluation

At a glance

Published

2023

Jurisdiction

International

Category

Evaluation and benchmarks

Access

Public access

More in Evaluation and benchmarks

tau-bench: A benchmark for tool-agent-user interaction

Sierra Research • 2024

WebArena: A Realistic Web Environment for Building Autonomous Agents

Zhou et al. • 2023

GAIA: A Benchmark for General AI Assistants

Mialon et al. • 2023

Related resources

Practices for governing agentic AI systems: OpenAI's seven safety principles

Governance frameworks • OpenAI

Taxonomy of Failure Mode in Agentic AI Systems

Risk taxonomies • Microsoft

EleutherAI LM Evaluation Harness

Assessment and evaluation • EleutherAI

Build your AI governance program

VerifyWise helps you implement AI governance frameworks, track compliance, and manage risk across your AI systems.

Explore the libraryStart free trial
SWE-bench: Can Language Models Resolve Real-World GitHub Issues? | VerifyWise AI Governance Library