benchmarks.wiki

benchmarks.wiki / Public workspace

SWE-bench Lite

Code / Agents

SWE-bench Lite is a 300-instance subset of SWE-bench in which the task is to write a patch that resolves a real GitHub issue. It is cheap enough to evaluate repeatedly during development.

swe-bench-lite

Overview onlyNot imported on this site

Official source

Overview only: problem text has not been published on this site; the catalogue records the benchmark introduction, category and official source.

Visit the official benchmarkStart a benchmark discussion
Catalogue status
Not imported on this site

At a glance

Size
300 instances in the source; none imported.
License
Not declared on the source.
Released
Dataset published on Hugging Face 2024-03
Maintainer
princeton-nlp (source organisation)
Answer access
No problem text is imported, so there is no answer storage for this entry. The source distributes the answers under its own terms.

What a problem looks like

An issue statement plus the repository and base commit it applies to.

How it is scored

Correctness is decided by running the repository’s own fail-to-pass and pass-to-pass tests against the candidate patch.

Why it is hard

The fix has to work inside a real codebase at a real commit, so the model has to locate the relevant code before it can change it.

Cite

@article{jimenez2023swebenchlite,
  title={SWE-bench: Can Language Models Resolve Real-World GitHub Issues?},
  author={Carlos E. Jimenez and John Yang and Alexander Wettig and Shunyu Yao and Kexin Pei and Ofir Press and Karthik Narasimhan},
  journal={arXiv preprint arXiv:2310.06770},
  year={2023}
}

Discussion

Contribute

Share a reproduction, report an evaluation issue, or suggest a correction to this benchmark overview.

Public problem summary

Catalogue metadata only; public problem statistics are not available.

Public problems
—
Formats
—
Problems with images
—
Answer availability
—
Configs
—
Splits
—
Awaiting a first discussion
—
Problems with a discussion
—
Public contributions
—