{"kind":"task","effective_mode":"full","benchmark":{"kind":"benchmark","effective_mode":"full","slug":"bbeh","formal_name":"BIG-Bench Extra Hard (BBEH)","introduction":"BIG-Bench Extra Hard evaluates language-model reasoning through challenging task categories. It offers full and mini distributions; this import contains the full distribution: 4,520 questions across all 23 task categories.","introduction_ja":"","introduction_en":"","category":"Category not supplied","task_count":null,"acquisition_status":"Acquisition status not supplied","official_url":"https://github.com/google-deepmind/bbeh","indexing_mode":"index","profile":{"resources":[],"task_format":"","scoring":"","metric":"","size":"","answer_access":"","license":"","citation":"","maintainer":"","released":"","why_hard":"","related":[],"history":"","construction":"","limitations":"","saturation":"","scores":[]}},"task_id":"0ae27d63-5024-5d5d-b26d-b4a578a0fa19","task_key":"bbeh~5fnycc--full--0ae27d63-5024-5d5d-b26d-b4a578a0fa19","task_revision_id":"1","upstream_id":"","short_description":"The following is a description of a funny cartoon for the New Yorker Caption…","config":"bbeh_nycc","split":"full","body":"{\"input\":\"The following is a description of a funny cartoon for the New Yorker Caption Contest\\n\\n  Description: Two cowboys are standing next to their horses in a desert, watching another cowboy who is performing gymnastics on a pommel horse.\\n\\n  Which of the following captions is the funniest?\\n(A) Don't turn around, I need to change into my leotard for the floor routine.\\n(B) You can take the Cowboy out of the Olympics, but you can’t take the Olympics out of the Cowboy!\\n(C) He's been like this ever since his other horse died.\\n(D) You tell him that's not what we meant when we said we were going for the gold.\\n(E) That’s my Fancy Cousin, he’s from Austin.\\n(F) What do you figger you get, exactly, if you breed a mustang with a pommel horse?\\n(G) He saw the wrong man about a horse.\\n(H) He’s going places, that one.\\n(I) It’s all about the dismount!\\n(J) Yeah,but how’s his ropin’?\\n\"}","display_format":"text","language":"","answer_status":"published","assets":[],"source_url":"https://github.com/google-deepmind/bbeh","history":"initial import","indexing_mode":"index","subproblems":[],"grids":[]}