{"kind":"task","effective_mode":"full","benchmark":{"kind":"benchmark","effective_mode":"full","slug":"bbeh","formal_name":"BIG-Bench Extra Hard (BBEH)","introduction":"BIG-Bench Extra Hard evaluates language-model reasoning through challenging task categories. It offers full and mini distributions; this import contains the full distribution: 4,520 questions across all 23 task categories.","introduction_ja":"","introduction_en":"","category":"Category not supplied","task_count":null,"acquisition_status":"Acquisition status not supplied","official_url":"https://github.com/google-deepmind/bbeh","indexing_mode":"index","profile":{"resources":[],"task_format":"","scoring":"","metric":"","size":"","answer_access":"","license":"","citation":"","maintainer":"","released":"","why_hard":"","related":[],"history":"","construction":"","limitations":"","saturation":"","scores":[]}},"task_id":"b18ed072-5019-5261-8e49-6d905bba64d3","task_key":"bbeh~5fnycc--full--b18ed072-5019-5261-8e49-6d905bba64d3","task_revision_id":"1","upstream_id":"","short_description":"The following is a description of a funny cartoon for the New Yorker Caption…","config":"bbeh_nycc","split":"full","body":"{\"input\":\"The following is a description of a funny cartoon for the New Yorker Caption Contest\\n\\n  Description: A man wearing a crown and a royal robe is walking down the street with a woman. Other pedestrians are walking by, seemingly unfazed by his attire.\\n\\n  Which of the following captions is the funniest?\\n(A) Promise you’ll take it off when we get there.\\n(B) Okay, I'll call you Harry, formerly known as Prince.\\n(C) Don't worry....I'll find SOMETHING to be King of.\\n(D) Charles, your mother is going to be so mad.\\n(E) Now I know how Harry and Meghan feel.\\n(F) I bet you forgot to feed the crocodiles again\\n(G) At least they let you keep the suit.\\n(H) Whatever happened to the money we wired to your cousin in Nigeria?\\n(I) The thing I like the least about it is the elections.\\n(J) No, I’m saying it’s the wrong outfit for ‘Hamilton’.\\n\"}","display_format":"text","language":"","answer_status":"published","assets":[],"source_url":"https://github.com/google-deepmind/bbeh","history":"initial import","indexing_mode":"index","subproblems":[],"grids":[]}