{"kind":"task","effective_mode":"full","benchmark":{"kind":"benchmark","effective_mode":"full","slug":"bbeh","formal_name":"BIG-Bench Extra Hard (BBEH)","introduction":"BIG-Bench Extra Hard evaluates language-model reasoning through challenging task categories. It offers full and mini distributions; this import contains the full distribution: 4,520 questions across all 23 task categories.","introduction_ja":"","introduction_en":"","category":"Category not supplied","task_count":null,"acquisition_status":"Acquisition status not supplied","official_url":"https://github.com/google-deepmind/bbeh","indexing_mode":"index","profile":{"resources":[],"task_format":"","scoring":"","metric":"","size":"","answer_access":"","license":"","citation":"","maintainer":"","released":"","why_hard":"","related":[],"history":"","construction":"","limitations":"","saturation":"","scores":[]}},"task_id":"050a8d11-ce26-527a-9c31-7b2a95ac392f","task_key":"bbeh~5fnycc--full--050a8d11-ce26-527a-9c31-7b2a95ac392f","task_revision_id":"1","upstream_id":"","short_description":"The following is a description of a funny cartoon for the New Yorker Caption…","config":"bbeh_nycc","split":"full","body":"{\"input\":\"The following is a description of a funny cartoon for the New Yorker Caption Contest\\n\\n  Description: A man dressed in old-fashioned clothing is shopping in a grocery store. Next to him, a large whale is also pushing a shopping cart.\\n\\n  Which of the following captions is the funniest?\\n(A) Let's just let bygones be bygones.\\n(B) Krill? Check out the fish counter.\\n(C) I chased you down for 700 pages and now I run into you in the cereal aisle?\\n(D) It's easier to buy it than catch it yourself.\\n(E) How about chicken legs instead?\\n(F) My curiosity starts with how in the heck did you get through the front door.\\n(G) Well my leg wasn’t certified organic either.\\n(H) Remember; we're looking for sales not seals.\\n(I) Are you following me or is this just an unlucky fluke?\\n(J) Rather than revenge, an enlightened Ahab teaches the benefits of vegetarianism to his biggest foe.\\n\"}","display_format":"text","language":"","answer_status":"published","assets":[],"source_url":"https://github.com/google-deepmind/bbeh","history":"initial import","indexing_mode":"index","subproblems":[],"grids":[]}