{"kind":"task","effective_mode":"full","benchmark":{"kind":"benchmark","effective_mode":"full","slug":"bbeh","formal_name":"BIG-Bench Extra Hard (BBEH)","introduction":"BIG-Bench Extra Hard evaluates language-model reasoning through challenging task categories. It offers full and mini distributions; this import contains the full distribution: 4,520 questions across all 23 task categories.","introduction_ja":"","introduction_en":"","category":"Category not supplied","task_count":null,"acquisition_status":"Acquisition status not supplied","official_url":"https://github.com/google-deepmind/bbeh","indexing_mode":"index","profile":{"resources":[],"task_format":"","scoring":"","metric":"","size":"","answer_access":"","license":"","citation":"","maintainer":"","released":"","why_hard":"","related":[],"history":"","construction":"","limitations":"","saturation":"","scores":[]}},"task_id":"70745f5e-dc95-5654-98a4-1dc25d11aee4","task_key":"bbeh~5fnycc--full--70745f5e-dc95-5654-98a4-1dc25d11aee4","task_revision_id":"1","upstream_id":"","short_description":"The following is a description of a funny cartoon for the New Yorker Caption…","config":"bbeh_nycc","split":"full","body":"{\"input\":\"The following is a description of a funny cartoon for the New Yorker Caption Contest\\n\\n  Description: A scientist is standing in front of a large blackboard filled with complex mathematical equations and diagrams. Another person, also in a lab coat, is entering the room holding a cup of coffee.\\n\\n  Which of the following captions is the funniest?\\n(A) I never imagined baby formula could be so complex!\\n(B) Is there a problem in here?\\n(C) I can't find the eraser!\\n(D) How many times have I told you: Multiply THEN Divide.\\n(E) Will you be done with this room by the end of time?\\n(F) Put it together and what have you got?\\n(G) I’ve simplified my original equation.\\n(H) The answer is, of course, a cat.\\n(I) For heavens sake George, let's just split the lunch bill five ways.\\n(J) I need the black chalk\\n\"}","display_format":"text","language":"","answer_status":"published","assets":[],"source_url":"https://github.com/google-deepmind/bbeh","history":"initial import","indexing_mode":"index","subproblems":[],"grids":[]}