{"kind":"task","effective_mode":"full","benchmark":{"kind":"benchmark","effective_mode":"full","slug":"bbeh","formal_name":"BIG-Bench Extra Hard (BBEH)","introduction":"BIG-Bench Extra Hard evaluates language-model reasoning through challenging task categories. It offers full and mini distributions; this import contains the full distribution: 4,520 questions across all 23 task categories.","introduction_ja":"","introduction_en":"","category":"Category not supplied","task_count":null,"acquisition_status":"Acquisition status not supplied","official_url":"https://github.com/google-deepmind/bbeh","indexing_mode":"index","profile":{"resources":[],"task_format":"","scoring":"","metric":"","size":"","answer_access":"","license":"","citation":"","maintainer":"","released":"","why_hard":"","related":[],"history":"","construction":"","limitations":"","saturation":"","scores":[]}},"task_id":"ebcf78ee-f05c-552d-913d-7bed1953c0e3","task_key":"bbeh~5fnycc--full--ebcf78ee-f05c-552d-913d-7bed1953c0e3","task_revision_id":"1","upstream_id":"","short_description":"The following is a description of a funny cartoon for the New Yorker Caption…","config":"bbeh_nycc","split":"full","body":"{\"input\":\"The following is a description of a funny cartoon for the New Yorker Caption Contest\\n\\n  Description: Two dogs are standing in front of a giant doghouse. One dog is inside the entrance, and the other is just outside.\\n\\n  Which of the following captions is the funniest?\\n(A) It's scientifically proven to promote higher thought.\\n(B) Our daughter married a greyhound.\\n(C) Customized for when my owner has to visit me...\\n(D) I call it Mar-a-Dawg-o.\\n(E) Sometimes my master needs to join me.\\n(F) I'm dating that basketball playing dog.\\n(G) I thought, why not go big AND go home.\\n(H) Look at what I'm able to afford just by leaving California!\\n(I) I got a great deal from the Great Dane.\\n(J) It's my monument to Dog.\\n\"}","display_format":"text","language":"","answer_status":"published","assets":[],"source_url":"https://github.com/google-deepmind/bbeh","history":"initial import","indexing_mode":"index","subproblems":[],"grids":[]}