{"kind":"task","effective_mode":"full","benchmark":{"kind":"benchmark","effective_mode":"full","slug":"bbeh","formal_name":"BIG-Bench Extra Hard (BBEH)","introduction":"BIG-Bench Extra Hard evaluates language-model reasoning through challenging task categories. It offers full and mini distributions; this import contains the full distribution: 4,520 questions across all 23 task categories.","introduction_ja":"","introduction_en":"","category":"Category not supplied","task_count":null,"acquisition_status":"Acquisition status not supplied","official_url":"https://github.com/google-deepmind/bbeh","indexing_mode":"index","profile":{"resources":[],"task_format":"","scoring":"","metric":"","size":"","answer_access":"","license":"","citation":"","maintainer":"","released":"","why_hard":"","related":[],"history":"","construction":"","limitations":"","saturation":"","scores":[]}},"task_id":"8dbcc5ee-ef00-509f-a4bb-91d3ab967a7a","task_key":"bbeh~5fnycc--full--8dbcc5ee-ef00-509f-a4bb-91d3ab967a7a","task_revision_id":"1","upstream_id":"","short_description":"The following is a description of a funny cartoon for the New Yorker Caption…","config":"bbeh_nycc","split":"full","body":"{\"input\":\"The following is a description of a funny cartoon for the New Yorker Caption Contest\\n\\n  Description: Two cavemen are walking hand in hand, waving at two other cavemen who are looking out from a tall stone tower.\\n\\n  Which of the following captions is the funniest?\\n(A) Welcome to our new come.  Take the rope to the fifth floor.\\n(B) I can’t understand what they’re Babelling about.\\n(C) Come on up! We're in 4A\\n(D) Love your matching outfits.\\n(E) Come on up before high tide!\\n(F) Did you invent stairs yet?\\n(G) Inventing fire has been good to you.\\n(H) It’s a mammoth viewing platform\\n(I) They’re from prehistoric money\\n(J) They've come a long way since the cave years.\\n\"}","display_format":"text","language":"","answer_status":"published","assets":[],"source_url":"https://github.com/google-deepmind/bbeh","history":"initial import","indexing_mode":"index","subproblems":[],"grids":[]}