{"kind":"task","effective_mode":"full","benchmark":{"kind":"benchmark","effective_mode":"full","slug":"bbeh","formal_name":"BIG-Bench Extra Hard (BBEH)","introduction":"BIG-Bench Extra Hard evaluates language-model reasoning through challenging task categories. It offers full and mini distributions; this import contains the full distribution: 4,520 questions across all 23 task categories.","introduction_ja":"","introduction_en":"","category":"Category not supplied","task_count":null,"acquisition_status":"Acquisition status not supplied","official_url":"https://github.com/google-deepmind/bbeh","indexing_mode":"index","profile":{"resources":[],"task_format":"","scoring":"","metric":"","size":"","answer_access":"","license":"","citation":"","maintainer":"","released":"","why_hard":"","related":[],"history":"","construction":"","limitations":"","saturation":"","scores":[]}},"task_id":"ba3acefd-d281-5f6c-9a5d-ceda278ff16e","task_key":"bbeh~5fnycc--full--ba3acefd-d281-5f6c-9a5d-ceda278ff16e","task_revision_id":"1","upstream_id":"","short_description":"The following is a description of a funny cartoon for the New Yorker Caption…","config":"bbeh_nycc","split":"full","body":"{\"input\":\"The following is a description of a funny cartoon for the New Yorker Caption Contest\\n\\n  Description: A waiter is serving two people at a table. Instead of food, he has three birds perched on his arm.\\n\\n  Which of the following captions is the funniest?\\n(A) They taste like Wall Street, chicken…\\n(B) Our early bird specials.\\n(C) They say “cheep”, but we’re actually very pricy.\\n(D) which one would you like?\\n(E) Three on the arm is better than one in the hand.\\n(F) A pair of Peeps I said\\n(G) Their names are Manny, Moe, and Jack.\\n(H) Squab is the latest in fine dining.\\n(I) The one on your right but don't throw it in the pot alive, please.\\n(J) Oh, this again! He requested for \\\"the bryds\\\" not \\\"the birds\\\". Why did we even bother to learn human language?\\n\"}","display_format":"text","language":"","answer_status":"published","assets":[],"source_url":"https://github.com/google-deepmind/bbeh","history":"initial import","indexing_mode":"index","subproblems":[],"grids":[]}