{"kind":"task","effective_mode":"full","benchmark":{"kind":"benchmark","effective_mode":"full","slug":"bbeh","formal_name":"BIG-Bench Extra Hard (BBEH)","introduction":"BIG-Bench Extra Hard evaluates language-model reasoning through challenging task categories. It offers full and mini distributions; this import contains the full distribution: 4,520 questions across all 23 task categories.","introduction_ja":"","introduction_en":"","category":"Category not supplied","task_count":null,"acquisition_status":"Acquisition status not supplied","official_url":"https://github.com/google-deepmind/bbeh","indexing_mode":"index","profile":{"resources":[],"task_format":"","scoring":"","metric":"","size":"","answer_access":"","license":"","citation":"","maintainer":"","released":"","why_hard":"","related":[],"history":"","construction":"","limitations":"","saturation":"","scores":[]}},"task_id":"441e3bd1-c394-5ec0-91da-d0207e05dc35","task_key":"bbeh~5fnycc--full--441e3bd1-c394-5ec0-91da-d0207e05dc35","task_revision_id":"1","upstream_id":"","short_description":"The following is a description of a funny cartoon for the New Yorker Caption…","config":"bbeh_nycc","split":"full","body":"{\"input\":\"The following is a description of a funny cartoon for the New Yorker Caption Contest\\n\\n  Description: A person with a backpack is climbing up a cliff and encounters a grand piano with a pianist sitting at it. The pianist looks disheveled and tired.\\n\\n  Which of the following captions is the funniest?\\n(A) You need an idea? You should build a stairway to here.\\n(B) Not Al, the pianist. Alpinist.\\n(C) ... oh, and your feng shui is all off too.\\n(D) Oh grand one, what is the key to life?\\n(E) The remaining members of my team request that you stop composing in B flat.\\n(F) The Craigslist ad said it's free if I pick it up myself...?\\n(G) I have a request.\\n(H) Aw, com'on Sam, play it!\\n(I) Hey Dad, Mom wants ta know if you'll be home for dinner??\\n(J) Sorry. I don't take requests.\\n\"}","display_format":"text","language":"","answer_status":"published","assets":[],"source_url":"https://github.com/google-deepmind/bbeh","history":"initial import","indexing_mode":"index","subproblems":[],"grids":[]}