benchmarks.wiki

benchmarks.wiki / Public workspace

SWE-Bench Pro

Code / Agents

SWE-Bench Pro evaluates agents on long-horizon software engineering tasks in real repositories. The public card describes 731 tasks containing issue descriptions, repository identifiers, and base commits.

swe-bench-pro

Full text200 tasks

Official source

Full text: eligible public problems include text, images and discussion.

Download answersBrowse the problemsAnswersOpen a random oneProblems awaiting a first attempt

History

Scale AI distributes SWE-Bench Pro as an evaluation of agents performing extended software engineering work in real repositories. This catalogue records publication of the Hugging Face dataset in August 2025. Its focus is repository repair over a sequence of actions, rather than isolated function completion.

How it was built

The public dataset card describes 731 tasks with issue descriptions, repository identifiers and base commits. Task records also specify requirements and interfaces, placing each requested repair in a particular repository state. Evaluation uses tests that must change from failing to passing and tests that must continue to pass; matching a reference patch is not the scoring criterion.

At a glance

Size
731 public tasks described in the catalog introduction; this test selection expects 200 task records with a source-row limit of 200.
Released
Dataset published on Hugging Face 2025-08
Maintainer
ScaleAI (Hugging Face organisation)
Answer access
Patches and tests are kept outside prompt fields in restricted storage; evaluation requires a test runner, not just a reference patch.

What a problem looks like

Repository repair tasks; the import accepts problem_statement, requirements, interface, repo, base_commit, repo_language, and dockerhub_tag.

Read an actual problem

How it is scored

Repository patches are evaluated with fail_to_pass and pass_to_pass tests; a patch alone is not a graded result.

Why it is hard

Requires long-horizon software engineering in real repositories.

Limitations and criticism

Test success measures the behaviours exercised by the evaluation suite rather than every consequence of a repository change. Agent scaffolding, available tools, runtime environment and execution budget can affect the result as well as the underlying model. The 731 tasks described in the public card should be identified as that public collection when comparing reports, rather than assumed to represent every evaluation carrying the SWE-Bench Pro name.

Discussion

1 thread and 0 posts on this page. Anyone may reply; no account is required.

Contribute

Share a reproduction, report an evaluation issue, or suggest a correction to this benchmark overview.

Tasks

Whole public benchmark.

Public problems
200
Formats
text (200)
Problems with images
0
Answer availability
Scored by the benchmark’s own harness (200)
Configs
Default (200)
Splits
test (200)
Awaiting a first discussion
200
Problems with a discussion
0
Public contributions
0

Provenance and terms

Pinned source revisions, newest first:

  • 7ab5114912baf22bb098818e604c02fe7ad2c11f · acquired 2026-09-09 · release 4cd4b68249ab
  • 7ab5114912baf22bb098818e604c02fe7ad2c11f · acquired 2026-09-09 · release 16026b211da6
  • 7ab5114912baf22bb098818e604c02fe7ad2c11f · acquired 2026-09-09 · release afc4a41520ba
  • 7ab5114912baf22bb098818e604c02fe7ad2c11f · acquired 2026-09-09 · release 43e81a1c7541
Licence
Not recorded. This catalogue has not verified redistribution terms.
Redistribution terms
Not recorded. This catalogue has not verified redistribution terms.
Publisher requests
Not recorded. This catalogue has not verified redistribution terms.
Terms last checked
Never