{"kind":"task","effective_mode":"full","benchmark":{"kind":"benchmark","effective_mode":"full","slug":"bbeh","formal_name":"BIG-Bench Extra Hard (BBEH)","introduction":"BIG-Bench Extra Hard evaluates language-model reasoning through challenging task categories. It offers full and mini distributions; this import contains the full distribution: 4,520 questions across all 23 task categories.","introduction_ja":"","introduction_en":"","category":"Category not supplied","task_count":null,"acquisition_status":"Acquisition status not supplied","official_url":"https://github.com/google-deepmind/bbeh","indexing_mode":"index","profile":{"resources":[],"task_format":"","scoring":"","metric":"","size":"","answer_access":"","license":"","citation":"","maintainer":"","released":"","why_hard":"","related":[],"history":"","construction":"","limitations":"","saturation":"","scores":[]}},"task_id":"2e128f97-d3fb-5a92-b5ba-64a53c698a94","task_key":"bbeh~5fnycc--full--2e128f97-d3fb-5a92-b5ba-64a53c698a94","task_revision_id":"1","upstream_id":"","short_description":"The following is a description of a funny cartoon for the New Yorker Caption…","config":"bbeh_nycc","split":"full","body":"{\"input\":\"The following is a description of a funny cartoon for the New Yorker Caption Contest\\n\\n  Description: A couple is sitting at a table on a balcony, holding menus. A waiter is standing inside, looking at them with concern.\\n\\n  Which of the following captions is the funniest?\\n(A) Yes this is the smoking section.\\n(B) We told you up front that those who dine al fresco with us once never dine elsewhere.\\n(C) Tonight's special is wild caught pigeon.\\n(D) The importance of making a reservation cannot be overstated.\\n(E) Perhaps the lady would be more comfortable inside, Mr. Kent?\\n(F) Most of our customers were blown away by today's specials.\\n(G) We have three specials tonight\\nFlying fish\\nRoosted pigeon\\nLedge of Lamb\\n(H) You requested the \\\"No Tipping\\\" section!\\n(I) Will you be expecting anyone else?\\n(J) Our special this evening is the Harold Lloyd.\\n\"}","display_format":"text","language":"","answer_status":"published","assets":[],"source_url":"https://github.com/google-deepmind/bbeh","history":"initial import","indexing_mode":"index","subproblems":[],"grids":[]}