{"kind":"task","effective_mode":"full","benchmark":{"kind":"benchmark","effective_mode":"full","slug":"bbeh","formal_name":"BIG-Bench Extra Hard (BBEH)","introduction":"BIG-Bench Extra Hard evaluates language-model reasoning through challenging task categories. It offers full and mini distributions; this import contains the full distribution: 4,520 questions across all 23 task categories.","introduction_ja":"","introduction_en":"","category":"Category not supplied","task_count":null,"acquisition_status":"Acquisition status not supplied","official_url":"https://github.com/google-deepmind/bbeh","indexing_mode":"index","profile":{"resources":[],"task_format":"","scoring":"","metric":"","size":"","answer_access":"","license":"","citation":"","maintainer":"","released":"","why_hard":"","related":[],"history":"","construction":"","limitations":"","saturation":"","scores":[]}},"task_id":"b028414f-939c-5739-ae5b-b82661dc20ca","task_key":"bbeh~5fnycc--full--b028414f-939c-5739-ae5b-b82661dc20ca","task_revision_id":"1","upstream_id":"","short_description":"The following is a description of a funny cartoon for the New Yorker Caption…","config":"bbeh_nycc","split":"full","body":"{\"input\":\"The following is a description of a funny cartoon for the New Yorker Caption Contest\\n\\n  Description: A woman is standing in an office hallway, looking at a group of identical men wearing glasses, white shirts, and ties. The men are all lined up, coming through a doorway.\\n\\n  Which of the following captions is the funniest?\\n(A) I think the copier was left on the Fred setting again.\\n(B) I'm sending up that crop of newly minted MBAs you wanted.\\n(C) The Accountants are multiplying again.\\n(D) I'm pretty sure that Dr. Janson has perfected a highly advanced variant of asexual reproduction.\\n(E) Yes, Sir. Your description was perfect!\\n(F) The sperm bank class action lawsuit clients are here.\\n(G) Did you say look for the bald guy with a small mustache and round glasses?\\n(H) I wanted clowns for the party, not clones.\\n(I) Cancel the shrimp tempura. He says they're all allergic.\\n(J) Ask the repairman to come back, still something wrong with the copier.\\n\"}","display_format":"text","language":"","answer_status":"published","assets":[],"source_url":"https://github.com/google-deepmind/bbeh","history":"initial import","indexing_mode":"index","subproblems":[],"grids":[]}