{"kind":"task","effective_mode":"full","benchmark":{"kind":"benchmark","effective_mode":"full","slug":"bbeh","formal_name":"BIG-Bench Extra Hard (BBEH)","introduction":"BIG-Bench Extra Hard evaluates language-model reasoning through challenging task categories. It offers full and mini distributions; this import contains the full distribution: 4,520 questions across all 23 task categories.","introduction_ja":"","introduction_en":"","category":"Category not supplied","task_count":null,"acquisition_status":"Acquisition status not supplied","official_url":"https://github.com/google-deepmind/bbeh","indexing_mode":"index","profile":{"resources":[],"task_format":"","scoring":"","metric":"","size":"","answer_access":"","license":"","citation":"","maintainer":"","released":"","why_hard":"","related":[],"history":"","construction":"","limitations":"","saturation":"","scores":[]}},"task_id":"d3409983-4721-5205-b648-68dc8176b34e","task_key":"bbeh~5fnycc--full--d3409983-4721-5205-b648-68dc8176b34e","task_revision_id":"1","upstream_id":"","short_description":"The following is a description of a funny cartoon for the New Yorker Caption…","config":"bbeh_nycc","split":"full","body":"{\"input\":\"The following is a description of a funny cartoon for the New Yorker Caption Contest\\n\\n  Description: Two snowmen are in a boxing ring, wearing boxing gloves and facing each other. One snowman has knocked the head off another snowman, which is lying on the ground.\\n\\n  Which of the following captions is the funniest?\\n(A) My head is off, but I still win on points\\n(B) Just let me melt.\\n(C) I see what you mean.\\n(D) Round two when the kid puts me back on?\\n(E) Oh my God! My butt is huge.\\n(F) Duck and weave,\\\" Coach said. \\\"I'm a snowman,\\\" I said\\n(G) Got any ice?\\n(H) I thought 'knock your block off' was a figure of speech.\\n(I) You were supposed to duck!\\n(J) If it ever snows again, I want a rematch.\\n\"}","display_format":"text","language":"","answer_status":"published","assets":[],"source_url":"https://github.com/google-deepmind/bbeh","history":"initial import","indexing_mode":"index","subproblems":[],"grids":[]}