benchmarks.wiki

benchmarks.wiki / Public workspace

Humanity's Last Code Exam

Code

Humanity's Last Code Exam evaluates problem solving and code generation using ICPC World Finals and IOI problems. The project describes 235 problems; this import selects its public ICPC dataset of 146 problems.

hlce

Full text146 tasks

Official source

Full text: eligible public problems include text, images and discussion.

Download answersBrowse the problemsAnswersOpen a random oneProblems awaiting a first attempt

History

The Humanity's Last Code Exam project assembled a benchmark of ICPC World Finals and International Olympiad in Informatics problems to evaluate demanding algorithmic problem solving and code generation. Its GitHub repository, maintained by the Humanity-s-Last-Code-Exam organisation, was created in May 2025. The project drew on established programming competitions as a source of difficult tasks.

How it was built

The project describes 235 problems drawn from ICPC World Finals and IOI competitions. This catalogue selects 146 public ICPC problem records, whose prompt fields include the title, statement, and platform. Evaluation is defined by the official repository scripts for the two competitions; generated responses and scoring outputs in the selected example data are not themselves independent correctness criteria.

At a glance

Size
235 problems described in the catalog introduction; this selection expects 146 ICPC problem records from ICPC-World-Finals / examples, with a source-row limit of 200.
Released
Repository created on GitHub 2025-05
Maintainer
Humanity-s-Last-Code-Exam (GitHub organisation)
Answer access
Generated responses and scoring outputs in the selected example data remain in restricted storage.

What a problem looks like

Competitive-programming problem statements; the import accepts question_title, question_content, and platform.

Read an actual problem

How it is scored

For the 146 imported problems, the source ships an answer with every problem; this catalogue keeps it in restricted storage and never renders it. Definitive results come from the IOI and ICPC World Finals scripts in the official repository.

Why it is hard

Requires problem solving and code generation on ICPC World Finals and IOI problems.

Limitations and criticism

The imported ICPC subset does not cover the full benchmark or establish performance on the IOI portion. Competitive-programming tasks test algorithm design and implementation under contest conditions, with limited coverage of software maintenance or collaborative development. Because the problems originate in public competitions, exposure to statements or solutions during training is a possibility that requires separate investigation.

Cite

@misc{li2025humanityscodeexamadvanced,
      title={Humanity's Last Code Exam: Can Advanced LLMs Conquer Human's Hardest Code Competition?}, 
      author={Xiangyang Li and Xiaopeng Li and Kuicai Dong and Quanhu Zhang and Rongju Ruan and Xinyi Dai and Xiaoshuang Liu and Shengchun Xu and Yasheng Wang and Ruiming Tang},
      year={2025},
      eprint={2506.12713},
      archivePrefix={arXiv},
      primaryClass={cs.SE},
      url={https://arxiv.org/abs/2506.12713}, 
}

Discussion

1 thread and 0 posts on this page. Anyone may reply; no account is required.

Contribute

Share a reproduction, report an evaluation issue, or suggest a correction to this benchmark overview.

Tasks

Whole public benchmark.

Public problems
146
Formats
text (146)
Problems with images
0
Answer availability
Answer published by the source (146)
Configs
ICPC-World-Finals (146)
Splits
examples (146)
Awaiting a first discussion
146
Problems with a discussion
0
Public contributions
0

Provenance and terms

Pinned source revisions, newest first:

  • 7ff35699eb56a09ba17ea00ed3dbb6e520635832 · acquired 2026-09-09 · release aa405bca77f4
  • 7ff35699eb56a09ba17ea00ed3dbb6e520635832 · acquired 2026-09-09 · release ce11e5bbbba7
  • 7ff35699eb56a09ba17ea00ed3dbb6e520635832 · acquired 2026-09-09 · release 31d1c4183bd0
  • 7ff35699eb56a09ba17ea00ed3dbb6e520635832 · acquired 2026-09-09 · release 20b0aecfd177
Licence
Not recorded. This catalogue has not verified redistribution terms.
Redistribution terms
Not recorded. This catalogue has not verified redistribution terms.
Publisher requests
Not recorded. This catalogue has not verified redistribution terms.
Terms last checked
Never