{"kind":"task","effective_mode":"full","benchmark":{"kind":"benchmark","effective_mode":"full","slug":"bbeh","formal_name":"BIG-Bench Extra Hard (BBEH)","introduction":"BIG-Bench Extra Hard evaluates language-model reasoning through challenging task categories. It offers full and mini distributions; this import contains the full distribution: 4,520 questions across all 23 task categories.","introduction_ja":"","introduction_en":"","category":"Category not supplied","task_count":null,"acquisition_status":"Acquisition status not supplied","official_url":"https://github.com/google-deepmind/bbeh","indexing_mode":"index","profile":{"resources":[],"task_format":"","scoring":"","metric":"","size":"","answer_access":"","license":"","citation":"","maintainer":"","released":"","why_hard":"","related":[],"history":"","construction":"","limitations":"","saturation":"","scores":[]}},"task_id":"7a74734a-2be3-580d-bfe0-e2d982211eaa","task_key":"bbeh~5flinguini--full--7a74734a-2be3-580d-bfe0-e2d982211eaa","task_revision_id":"1","upstream_id":"","short_description":"The squares of the numbers 1 to 10 are spelt out in the Ndom language, in…","config":"bbeh_linguini","split":"full","body":"{\"input\":\"The squares of the numbers 1 to 10 are spelt out in the Ndom language, in arbitrary order:\\n\\nnif abo mer an thef abo sas\\nnif thef abo tondor abo mer abo thonith\\nmer an thef abo thonith\\nnif\\nmer abo ithin\\nthonith\\nsas\\nnif thef abo mer abo ithin\\nnif abo tondor abo mer abo thonith\\ntondor abo mer abo sas\\n\\nWrite in numerals: mer an thef abo meregh.\"}","display_format":"text","language":"","answer_status":"published","assets":[],"source_url":"https://github.com/google-deepmind/bbeh","history":"initial import","indexing_mode":"index","subproblems":[],"grids":[]}