benchmarks.wiki

benchmarks.wiki / Public workspace

LiveCodeBench

Code

LiveCodeBench continuously collects new competitive-programming problems to evaluate coding capabilities. Its initial release_v1 contains 400 problems, with source platform, contest date, and dataset version tracked explicitly.

livecodebench

Full text100 tasks

Official source

Full text: eligible public problems include text, images and discussion.

Download answersBrowse the problemsAnswersOpen a random oneProblems awaiting a first attempt

History

The LiveCodeBench research group introduced the benchmark in 2024 to evaluate coding capabilities using newly collected competitive-programming problems. Continuing to add problems made it possible to evaluate on material released after a model training cutoff, addressing a weakness of static public coding datasets. The catalogue records publication of the source dataset on Hugging Face in April 2024.

How it was built

LiveCodeBench collects competition questions and records their source platform, contest date, difficulty, and dataset version. Generated programs are checked against public and private test cases using an external execution runner. The initial release_v1 contains 400 problems, while this catalogue selection includes only the first 100 source records from that release.

At a glance

Size
400 problems in initial release_v1 described in the catalog introduction; this release_v1 / test selection expects 100 task records, limited to the first 100 source rows.
License
cc (declared on the Hugging Face dataset card)
Released
Dataset published on Hugging Face 2024-04
Maintainer
livecodebench (Hugging Face organisation)
Answer access
Test cases remain outside the prompt in restricted storage. They support execution-based evaluation and are not reference programs.

What a problem looks like

Competitive-programming questions; the import accepts question_title, question_content, platform, contest_id, contest_date, difficulty, and starter_code.

Read an actual problem

How it is scored

Generated programs are evaluated with public_test_cases and private_test_cases using an external runner.

Why it is hard

Requires solving competitive-programming problems; the dataset continuously collects new problems.

Limitations and criticism

A benchmark name alone does not identify a LiveCodeBench evaluation: release version and contest-date window determine which problems are included. Recent problems reduce the opportunity for training exposure only when their dates are later than the relevant training cutoff; an older public release does not retain that advantage indefinitely. Execution settings and sampling budgets also affect comparisons, and competition code covers only part of practical software engineering.

Discussion

1 thread and 0 posts on this page. Anyone may reply; no account is required.

Contribute

Share a reproduction, report an evaluation issue, or suggest a correction to this benchmark overview.

Tasks

Whole public benchmark.

Public problems
100
Formats
text (100)
Problems with images
0
Answer availability
Scored by the benchmark’s own harness (100)
Configs
release_v1 (100)
Splits
test (100)
Awaiting a first discussion
100
Problems with a discussion
0
Public contributions
0

Provenance and terms

Pinned source revisions, newest first:

  • 0fe84c3912ea0c4d4a78037083943e8f0c4dd505 · acquired 2026-09-09 · release 93a2ce37da59
  • 0fe84c3912ea0c4d4a78037083943e8f0c4dd505 · acquired 2026-09-09 · release f75d3beb07ea
  • 0fe84c3912ea0c4d4a78037083943e8f0c4dd505 · acquired 2026-09-09 · release 2167c113d1ed
  • 0fe84c3912ea0c4d4a78037083943e8f0c4dd505 · acquired 2026-09-09 · release 8a40ca236bd7
Licence
Not recorded. This catalogue has not verified redistribution terms.
Redistribution terms
Not recorded. This catalogue has not verified redistribution terms.
Publisher requests
Not recorded. This catalogue has not verified redistribution terms.
Terms last checked
Never