{"kind":"task","effective_mode":"full","benchmark":{"kind":"benchmark","effective_mode":"full","slug":"bbeh","formal_name":"BIG-Bench Extra Hard (BBEH)","introduction":"BIG-Bench Extra Hard evaluates language-model reasoning through challenging task categories. It offers full and mini distributions; this import contains the full distribution: 4,520 questions across all 23 task categories.","introduction_ja":"","introduction_en":"","category":"Category not supplied","task_count":null,"acquisition_status":"Acquisition status not supplied","official_url":"https://github.com/google-deepmind/bbeh","indexing_mode":"index","profile":{"resources":[],"task_format":"","scoring":"","metric":"","size":"","answer_access":"","license":"","citation":"","maintainer":"","released":"","why_hard":"","related":[],"history":"","construction":"","limitations":"","saturation":"","scores":[]}},"task_id":"388b7fab-763e-5c01-a60f-d97eb3fada9b","task_key":"bbeh~5fnycc--full--388b7fab-763e-5c01-a60f-d97eb3fada9b","task_revision_id":"1","upstream_id":"","short_description":"The following is a description of a funny cartoon for the New Yorker Caption…","config":"bbeh_nycc","split":"full","body":"{\"input\":\"The following is a description of a funny cartoon for the New Yorker Caption Contest\\n\\n  Description: A person wearing an apron and a cap is standing in a doorway holding a stack of pancakes. A woman is walking by and looking at the person with the pancakes.\\n\\n  Which of the following captions is the funniest?\\n(A) Psst... last  vegan tiramisu 'til Chelsea.\\n(B) Sure, they're all natural ingredients.  Everything off the back of the truck is natural.\\n(C) Care for a decadent indulgence, ma'am?\\n(D) Just keep walking lady !  Nothing to see here but a creepy dude and his sweet delights .\\n(E) It may be a pile of shit, Gladys, but there's a cherry on top.\\n(F) In case he doesn't get the pardon, I've baked a file in the cake.\\n(G) Parents, teachers, ex-wives all yelling at me \\\"Artie, you take the cake!\\\" Finally I did.\\n(H) Oh god, Jerry, enough, the divorce is final.\\n(I) Psst! Unhealthy food, twenty bucks.\\n(J) Psssst lady. 500 empty calories? $7.00.\\n\"}","display_format":"text","language":"","answer_status":"published","assets":[],"source_url":"https://github.com/google-deepmind/bbeh","history":"initial import","indexing_mode":"index","subproblems":[],"grids":[]}