benchmarks.wiki

benchmarks.wiki / Public workspace

CodeContests

Code

CodeContests is the competitive-programming dataset built for AlphaCode, drawing problems from Codeforces and similar sites. Each problem ships with both correct and incorrect submissions and a large generated test suite.

code-contests

Official source

Download answersVisit the official benchmarkWhere the answers areStart a benchmark discussion

History

DeepMind built CodeContests for its AlphaCode work in 2022 to study program generation on competitive-programming problems. The motivation was to evaluate whether a system could derive an algorithm from a problem statement and implement it under competition constraints. The catalogue records the source dataset publication on Hugging Face in July 2022.

How it was built

The dataset collects existing competition problems from Codeforces and other programming-contest sources, together with correct and incorrect submissions. Problems include statements, input and output specifications, and execution limits. The source contains roughly 13,500 problems and combines public, private, and generated test cases for execution-based assessment.

At a glance

Size
Roughly 13,500 problems in the source; none imported.
License
cc-by-4.0 (declared on the Hugging Face dataset card)
Released
Dataset published on Hugging Face 2022-07
Maintainer
deepmind (source organisation)
Answer access
No problem text is imported, so there is no answer storage for this entry. The source distributes the answers under its own terms.

What a problem looks like

A full competitive-programming statement with input and output specifications, time and memory limits.

How it is scored

Correctness is decided by running the candidate against the public, private and generated tests.

Why it is hard

Competition problems hide an algorithmic idea behind a story, and a solution that is correct but too slow still fails the time limit.

Limitations and criticism

Contest programs emphasize self-contained algorithmic solutions rather than repository maintenance, requirements negotiation, or interaction with an existing software system. Passing a finite test suite does not prove correctness on every legal input. Candidate counts, selection procedures, execution environments, and time limits can change measured success, so results obtained with different generation and evaluation budgets are not directly interchangeable. Public contest statements and submissions also make training-data overlap possible.

Cite

@article{li2022codecontests,
  title={Competition-Level Code Generation with AlphaCode},
  author={Yujia Li and David Choi and Junyoung Chung and Nate Kushman and Julian Schrittwieser and Rémi Leblond and Tom Eccles and James Keeling and Felix Gimeno and Agustin Dal Lago and Thomas Hubert and Peter Choy and Cyprien de Masson d'Autume and Igor Babuschkin and Xinyun Chen and Po-Sen Huang and Johannes Welbl and Sven Gowal and Alexey Cherepanov and James Molloy and Daniel J. Mankowitz and Esme Sutherland Robson and Pushmeet Kohli and Nando de Freitas and Koray Kavukcuoglu and Oriol Vinyals},
  journal={arXiv preprint arXiv:2203.07814},
  year={2022}
}

Discussion

1 thread and 0 posts on this page. Anyone may reply; no account is required.

Contribute

Share a reproduction, report an evaluation issue, or suggest a correction to this benchmark overview.

Public problem summary

Catalogue metadata only; public problem statistics are not available.

Public problems
—
Formats
—
Problems with images
—
Answer availability
—
Configs
—
Splits
—
Awaiting a first discussion
—
Problems with a discussion
—
Public contributions
—