Benchmark AI / Public workspace

Terminal-Bench 2.1 / Tasks

Choose a problem to read its original content, examples and discussion.

Option counts cover the whole public benchmark.

Clear filters

1–50 / 89 tasks

Default config / tasks
You are provided with a dataset sample from a Bayesian Network in…
Answer availability not recorded
Identifiers
upstream_id
bn-fit-modify
task_id
4c712309-0236-5d55-8089-abe47f7acf52
task_key
tasks--bn~2dfit~2dmodify
Default config / tasks
Create a Python function called `async run_tasks(tasks: list[Callable[[],…
Answer availability not recorded
Identifiers
upstream_id
cancel-async-tasks
task_id
488c3685-afe3-5992-b911-a48e53b83561
task_key
tasks--cancel~2dasync~2dtasks
Default config / tasks
You are given a COBOL program located at /app/src/program.cbl. This program…
Answer availability not recorded
Identifiers
upstream_id
cobol-modernization
task_id
0813824c-df2f-5378-b6a4-3c8d51c7f5f6
task_key
tasks--cobol~2dmodernization
Default config / tasks
You're given an image of a pseudocode snippet at `/app/code.png`. Implement the…
Answer availability not recorded
Identifiers
upstream_id
code-from-image
task_id
7f1428ea-028a-5634-a8ff-f2deb768f6c9
task_key
tasks--code~2dfrom~2dimage
Default config / tasks
Under /tmp/CompCert/, build the CompCert C verified compiler (version 3.13.1)…
Answer availability not recorded
Identifiers
upstream_id
compile-compcert
task_id
7d369dae-c61c-59da-8637-78198de9ad51
task_key
tasks--compile~2dcompcert
Default config / tasks
Configure a git server so that I can run on my computer
Answer availability not recorded
Identifiers
upstream_id
configure-git-webserver
task_id
09116978-d557-52b4-b940-282334443edf
task_key
tasks--configure~2dgit~2dwebserver
Default config / tasks
Find a 1-hour meeting slot for Alice, Bob, and Carol during January 15-19, 2024…
Answer availability not recorded
Identifiers
upstream_id
constraints-scheduling
task_id
4a156ed6-21d9-5a4e-8279-43fc3c12da01
task_key
tasks--constraints~2dscheduling
Default config / tasks
Tell me how many deepseek tokens are there in the science domain of the…
Answer availability not recorded
Identifiers
upstream_id
count-dataset-tokens
task_id
53f78b10-2369-568b-9db1-b17a9bd2297f
task_key
tasks--count~2ddataset~2dtokens
Default config / tasks
You need to create a file called "/app/solution.txt" with the word found in…
Answer availability not recorded
Identifiers
upstream_id
crack-7z-hash
task_id
6fb92124-9d81-54a9-83db-c0b94107864e
task_key
tasks--crack~2d7z~2dhash
Default config / tasks
I have a program at `/app/` that crashes under RELEASE mode but not under DEBUG…
Answer availability not recorded
Identifiers
upstream_id
custom-memory-heap-crash
task_id
4e770a26-e31a-5f77-ad16-fcbc6b3fdf16
task_key
tasks--custom~2dmemory~2dheap~2dcrash
Default config / tasks
Finding a target probability distribution for LLM confidence metrics:
Answer availability not recorded
Identifiers
upstream_id
distribution-search
task_id
6501eddf-0f6b-5dd7-8ef3-66cdbeb37cff
task_key
tasks--distribution~2dsearch
Default config / tasks
The file titled sequences.fasta contains the following sequences:
Answer availability not recorded
Identifiers
upstream_id
dna-assembly
task_id
792bcd98-b75b-517e-aff3-7aecf7f9fe73
task_key
tasks--dna~2dassembly
Default config / tasks
The file titled sequences.fasta contains the sequence for a circular input…
Answer availability not recorded
Identifiers
upstream_id
dna-insert
task_id
07537ad2-709c-566e-98ef-ba837d442df3
task_key
tasks--dna~2dinsert
Default config / tasks
I have provided a file a.out that's a compiled C binary. Write me a program…
Answer availability not recorded
Identifiers
upstream_id
extract-elf
task_id
6cf3296f-4825-5279-8bdd-b37467aefbc5
task_key
tasks--extract~2delf
Default config / tasks
A video of someone playing zork is located at /app/video.mp4. Transcribe the…
Answer availability not recorded
Identifiers
upstream_id
extract-moves-from-video
task_id
058da236-236d-5e16-a241-4dd510eedd1e
task_key
tasks--extract~2dmoves~2dfrom~2dvideo
Default config / tasks
The file /app/feal.py implements a FEAL-like encryption function. Implement a…
Answer availability not recorded
Identifiers
upstream_id
feal-differential-cryptanalysis
task_id
40a9b46b-a14d-5b9a-adf0-706a7181da02
task_key
tasks--feal~2ddifferential~2dcryptanalysis
Default config / tasks
The file /app/feal.c implements a FEAL-like encryption function,
Answer availability not recorded
Identifiers
upstream_id
feal-linear-cryptanalysis
task_id
261a6d7d-bf41-51d3-a125-99c126dc5c58
task_key
tasks--feal~2dlinear~2dcryptanalysis
Default config / tasks
You need to identify and fix the vulnerability in the code according to the…
Answer availability not recorded
Identifiers
upstream_id
fix-code-vulnerability
task_id
527839ba-e39f-56dc-9fbf-ec996fbbbfe2
task_key
tasks--fix~2dcode~2dvulnerability
Default config / tasks
I just made some changes to my personal site and checked out master, but now I…
Answer availability not recorded
Identifiers
upstream_id
fix-git
task_id
2a445a39-dc1c-5ad6-a220-9937d97ff9f3
task_key
tasks--fix~2dgit
Default config / tasks
I have been making some changes to the OCaml garbage collector, improving the…
Answer availability not recorded
Identifiers
upstream_id
fix-ocaml-gc
task_id
7c425884-372b-5cb7-b032-a5353e67ca49
task_key
tasks--fix~2docaml~2dgc
Default config / tasks
A secret was accidentally committed to this repository in /app and then removed…
Answer availability not recorded
Identifiers
upstream_id
git-leak-recovery
task_id
52d4b99a-af08-54e7-9637-56b68b56b41b
task_key
tasks--git~2dleak~2drecovery
Default config / tasks
Set up a Git server that hosts a project over SSH at git@localhost:/git/project.
Answer availability not recorded
Identifiers
upstream_id
git-multibranch
task_id
2669f1d1-d57c-5093-987e-b8a754fddae7
task_key
tasks--git~2dmultibranch
Default config / tasks
You need to build a simple KV store server using grpc that records number values…
Answer availability not recorded
Identifiers
upstream_id
kv-store-grpc
task_id
1b90d696-7c03-5ebe-9b66-f318bacd1cde
task_key
tasks--kv~2dstore~2dgrpc
Default config / tasks
Complete the implementation in find_dominant_eigenvalue_and_eigenvector in…
Answer availability not recorded
Identifiers
upstream_id
largest-eigenval
task_id
00cf045e-7d8a-502b-aba0-1f2355acdd16
task_key
tasks--largest~2deigenval
Default config / tasks
Transform /app/input.csv (1 million rows) to match /app/expected.csv exactly…
Answer availability not recorded
Identifiers
upstream_id
large-scale-text-editing
task_id
286e3c5c-bd7e-5658-b3b9-d23f621615d2
task_key
tasks--large~2dscale~2dtext~2dediting
Default config / tasks
In this task you will implement an **LLM inference batching scheduler…
Answer availability not recorded
Identifiers
upstream_id
llm-inference-batching-scheduler
task_id
04fb7ecc-6eab-5b6b-a1fd-2ede50e2da66
task_key
tasks--llm~2dinference~2dbatching~2dscheduler
Default config / tasks
You are given multiple log files stored in /app/logs. Each log file name follows…
Answer availability not recorded
Identifiers
upstream_id
log-summary-date-ranges
task_id
7646fc7e-b024-5529-a39d-41798af98f58
task_key
tasks--log~2dsummary~2ddate~2dranges
Default config / tasks
Spin up a mailing list server for our reading group: reading-group@local.edu…
Answer availability not recorded
Identifiers
upstream_id
mailman
task_id
444a2b1b-1fdc-572b-a0a7-530d2542f6ff
task_key
tasks--mailman
Default config / tasks
I have provided /app/doomgeneric/, the source code to doom. I've also wrote a…
Answer availability not recorded
Identifiers
upstream_id
make-doom-for-mips
task_id
0d9d7f9a-5686-5623-b992-1598f0345538
task_key
tasks--make~2ddoom~2dfor~2dmips
Default config / tasks
Sample from a hierarchical Bayesian model using R and Stan, and estimate the…
Answer availability not recorded
Identifiers
upstream_id
mcmc-sampling-stan
task_id
2120754e-813e-59b1-a2bc-8a88340e8031
task_key
tasks--mcmc~2dsampling~2dstan
Default config / tasks
mkdir /app/repo, then initialize a git repo at /app/repo.
Answer availability not recorded
Identifiers
upstream_id
merge-diff-arc-agi-task
task_id
139f3032-6174-54f6-85b1-f0029f0e3005
task_key
tasks--merge~2ddiff~2darc~2dagi~2dtask
Default config / tasks
I have a ReLU neural network that you can query by importing `forward.py` and…
Answer availability not recorded
Identifiers
upstream_id
model-extraction-relu-logits
task_id
08a88261-6520-5fd9-a0e6-bf8c4b44cc0a
task_key
tasks--model~2dextraction~2drelu~2dlogits
Default config / tasks
Modernize Legacy Scientific Computing Stack
Answer availability not recorded
Identifiers
upstream_id
modernize-scientific-stack
task_id
03e2dfe1-94cd-5d93-a291-533b654fa2fb
task_key
tasks--modernize~2dscientific~2dstack
Default config / tasks
Given the query "terminal-bench" retrieve the document with the 5th highest…
Answer availability not recorded
Identifiers
upstream_id
mteb-retrieve
task_id
0c59891f-c9fd-52ff-bc3f-cb1cd659fa7f
task_key
tasks--mteb~2dretrieve
Default config / tasks
Your company needs a self-signed TLS certificate for an internal development…
Answer availability not recorded
Identifiers
upstream_id
openssl-selfsigned-cert
task_id
58d6a13b-f10f-52fa-9bd7-cf6adf91b115
task_key
tasks--openssl~2dselfsigned~2dcert
Default config / tasks
You need to perform a digital forensic recovery task. A system administrator has…
Answer availability not recorded
Identifiers
upstream_id
password-recovery
task_id
4016f33d-2d43-56e2-8f5a-03ac3d7306ec
task_key
tasks--password~2drecovery
Default config / tasks
Write me a single file in /app/polyglot/main.py.c which is a polyglot, that is,…
Answer availability not recorded
Identifiers
upstream_id
polyglot-c-py
task_id
3fc10a23-6366-5467-8936-458f1b8c487d
task_key
tasks--polyglot~2dc~2dpy
Default config / tasks
You have a Python baseline in portfolio_baseline.py that calculates portfolio…
Answer availability not recorded
Identifiers
upstream_id
portfolio-optimization
task_id
7fde2b13-a882-5add-94e1-0e4d914508c6
task_key
tasks--portfolio~2doptimization
Default config / tasks
Fix the incomplete proof of addition commutativity in the file plus_comm.v. The…
Answer availability not recorded
Identifiers
upstream_id
prove-plus-comm
task_id
4b4e3579-01b5-56e4-a5e7-023412e1f252
task_key
tasks--prove~2dplus~2dcomm
Default config / tasks
- You are given a PyTorch state dictionary (/app/weights.pt) representing the…
Answer availability not recorded
Identifiers
upstream_id
pytorch-model-recovery
task_id
02f642e5-ceeb-558e-bc94-13ad6458f4c0
task_key
tasks--pytorch~2dmodel~2drecovery
Default config / tasks
You are given the Open English Wordnet (OEWN) database in SQLite format, located…
Answer availability not recorded
Identifiers
upstream_id
query-optimize
task_id
0febdb24-9dbd-5796-a5c9-8e943ce4c407
task_key
tasks--query~2doptimize
Default config / tasks
You are given the output file of a Raman Setup. We used it to measure some…
Answer availability not recorded
Identifiers
upstream_id
raman-fitting
task_id
0611edf5-87e5-55be-a874-8a0975887093
task_key
tasks--raman~2dfitting
Default config / tasks
Write a regex expression that matches dates in the format YYYY-MM-DD appearing…
Answer availability not recorded
Identifiers
upstream_id
regex-log
task_id
45736f85-d6a3-57d9-9169-1b275a05a891
task_key
tasks--regex~2dlog
Default config / tasks
I have annotated histopathology slides with cell masks. The problem is that some…
Answer availability not recorded
Identifiers
upstream_id
sam-cell-seg
task_id
51ee8714-c1df-5309-8d2f-28e363289146
task_key
tasks--sam~2dcell~2dseg
Default config / tasks
Please help sanitize my github repository "dclm" of all API keys.
Answer availability not recorded
Identifiers
upstream_id
sanitize-git-repo
task_id
092eef1c-7363-5c44-8a7b-abcb284aaddb
task_key
tasks--sanitize~2dgit~2drepo
Default config / tasks
Write a file eval.scm that is a metacircular evaluator capable of interpreting…
Answer availability not recorded
Identifiers
upstream_id
schemelike-metacircular-eval
task_id
69066930-b664-58a9-b65a-d495b4dc1eff
task_key
tasks--schemelike~2dmetacircular~2deval
Default config / tasks
You are provided with a Turtle file at `/app/university_graph.ttl` containing…
Answer availability not recorded
Identifiers
upstream_id
sparql-university
task_id
0e7cc46a-54a6-50af-a387-1b7fb3a9fdd4
task_key
tasks--sparql~2duniversity
Default config / tasks
Compile SQLite in /app/sqlite with gcov instrumentation and make it available in…
Answer availability not recorded
Identifiers
upstream_id
sqlite-with-gcov
task_id
2e617dbc-df6d-51a5-b807-5e7f390f448e
task_key
tasks--sqlite~2dwith~2dgcov
Default config / tasks
Please train a fasttext model on the yelp data in the data/ folder.
Answer availability not recorded
Identifiers
upstream_id
train-fasttext
task_id
190dbbfb-9c10-5a54-825b-4acf5febe254
task_key
tasks--train~2dfasttext
Default config / tasks
Can you tune this MuJoCo model file (mjcf) such that it takes 60% of the…
Answer availability not recorded
Identifiers
upstream_id
tune-mjcf
task_id
79c87de9-d486-56de-a0cd-b1e1e10f7bbd
task_key
tasks--tune~2dmjcf