benchmarks.wiki

benchmarks.wiki / Public workspace

SWE-bench Lite

Code / Agents

SWE-bench Lite is a 300-instance subset of SWE-bench in which the task is to write a patch that resolves a real GitHub issue. It is cheap enough to evaluate repeatedly during development.

swe-bench-lite

Official source

Download answersVisit the official benchmarkWhere the answers areStart a benchmark discussion

History

The SWE-bench team published SWE-bench Lite through Princeton NLP as a smaller evaluation set for repository-level software repair, and this catalogue records its Hugging Face publication as March 2024 rather than a separate announcement date. It retains the central task of producing a patch for a real GitHub issue while reducing the cost of repeated evaluation. The subset contains 300 instances.

How it was built

SWE-bench Lite is selected from SWE-bench, whose instances link real GitHub issues to the repository changes that resolved them. Each task supplies an issue statement and a repository base commit from which a candidate patch is developed. Evaluation runs repository tests, including tests expected to change from failing to passing and tests expected to remain passing. The 300-instance subset therefore checks patches in the context of existing projects rather than as isolated code snippets.

At a glance

Size
300 instances in the source; none imported.
License
Not declared on the source.
Released
Dataset published on Hugging Face 2024-03
Maintainer
princeton-nlp (source organisation)
Answer access
No problem text is imported, so there is no answer storage for this entry. The source distributes the answers under its own terms.
Highest recorded
56.67 resolved (%) — SWE-agent + Claude 4 Sonnet (leaderboard, 2025-05-26)

What a problem looks like

An issue statement plus the repository and base commit it applies to.

How it is scored

Correctness is decided by running the repository’s own fail-to-pass and pass-to-pass tests against the candidate patch.

Reported scores

Evaluation conditions differ between rows, so this is not a ranking. Rows marked vendor are reports by the model’s own organization.

ModelScoreMetricSetupReported bySource
SWE-agent + Claude 4 SonnetAnthropic56.67resolved (%)Lite; SWE-agent; Model: claude-4-sonnet-20250514; Org: SWE-agent; System: Attempts - 1leaderboardLite leaderboard; 20250526_sweagent_claude-4-sonnet-20250514Reported 2025-05-26
SWE-agent + Claude 3.7 SonnetAnthropic48resolved (%)Lite; SWE-agent; Model: claude-3-7-sonnet-20250219; Org: SWE-agent; System: Attempts - 1leaderboardLite leaderboard; 20250226_sweagent_claude-3-7-sonnet-20250219Reported 2025-02-26

Why it is hard

The fix has to work inside a real codebase at a real commit, so the model has to locate the relevant code before it can change it.

Limitations and criticism

Passing the selected tests establishes success under the harness, not that a patch satisfies every unstated requirement or avoids every possible regression. Results reflect the agent's code navigation, editing and execution tools as well as its underlying model. Public issues and historical fixes also create a potential training-data overlap risk. Lite is a particular subset, so its resolution rate should not be substituted for a result on full SWE-bench or SWE-bench Verified.

How much room is left

The recorded official leaderboard reports that SWE-agent with Claude 4 Sonnet resolved 56.67% of Lite instances in a single-attempt setup on May 26, 2025. A February 26, 2025 entry for SWE-agent with Claude 3.7 Sonnet reports 48.0%. These entries leave substantial headroom, but they describe particular agent-model combinations and do not establish the best attainable rate under larger attempt budgets.

Cite

@article{jimenez2023swebenchlite,
  title={SWE-bench: Can Language Models Resolve Real-World GitHub Issues?},
  author={Carlos E. Jimenez and John Yang and Alexander Wettig and Shunyu Yao and Kexin Pei and Ofir Press and Karthik Narasimhan},
  journal={arXiv preprint arXiv:2310.06770},
  year={2023}
}

Discussion

1 thread and 0 posts on this page. Anyone may reply; no account is required.

Contribute

Share a reproduction, report an evaluation issue, or suggest a correction to this benchmark overview.

Public problem summary

Catalogue metadata only; public problem statistics are not available.

Public problems
—
Formats
—
Problems with images
—
Answer availability
—
Configs
—
Splits
—
Awaiting a first discussion
—
Problems with a discussion
—
Public contributions
—