{"kind":"task","effective_mode":"full","benchmark":{"kind":"benchmark","effective_mode":"full","slug":"bbeh","formal_name":"BIG-Bench Extra Hard (BBEH)","introduction":"BIG-Bench Extra Hard evaluates language-model reasoning through challenging task categories. It offers full and mini distributions; this import contains the full distribution: 4,520 questions across all 23 task categories.","introduction_ja":"","introduction_en":"","category":"Category not supplied","task_count":null,"acquisition_status":"Acquisition status not supplied","official_url":"https://github.com/google-deepmind/bbeh","indexing_mode":"index","profile":{"resources":[],"task_format":"","scoring":"","metric":"","size":"","answer_access":"","license":"","citation":"","maintainer":"","released":"","why_hard":"","related":[],"history":"","construction":"","limitations":"","saturation":"","scores":[]}},"task_id":"d1131294-188c-5862-9c12-062f51355dcc","task_key":"bbeh~5fnycc--full--d1131294-188c-5862-9c12-062f51355dcc","task_revision_id":"1","upstream_id":"","short_description":"The following is a description of a funny cartoon for the New Yorker Caption…","config":"bbeh_nycc","split":"full","body":"{\"input\":\"The following is a description of a funny cartoon for the New Yorker Caption Contest\\n\\n  Description: Two snakes are sitting by the side of a winding road, looking at a road sign that indicates a curvy path ahead.\\n\\n  Which of the following captions is the funniest?\\n(A) See. Our species is advancing. We now have our own road crossing sign.\\n(B) You adopted a highway?\\n(C) This section of road is dedicated to my Uncle Sid. He thought he was a racer.\\n(D) Body like a back road.\\n(E) Caution, that's a highway to heaven.\\n(F) Poor guy! I told him to be careful that you don't end  as a road sign\\n(G) That’s beware of snakes crossing the road sign.\\n(H) Interesting, but not an anatomically accurate depiction\\n(I) Hot damn! She's got more curves than Gina Boa Brigdita!\\n(J) Is that your cousin, or what?\\n\"}","display_format":"text","language":"","answer_status":"published","assets":[],"source_url":"https://github.com/google-deepmind/bbeh","history":"initial import","indexing_mode":"index","subproblems":[],"grids":[]}