{"kind":"task","effective_mode":"full","benchmark":{"kind":"benchmark","effective_mode":"full","slug":"bbeh","formal_name":"BIG-Bench Extra Hard (BBEH)","introduction":"BIG-Bench Extra Hard evaluates language-model reasoning through challenging task categories. It offers full and mini distributions; this import contains the full distribution: 4,520 questions across all 23 task categories.","introduction_ja":"","introduction_en":"","category":"Category not supplied","task_count":null,"acquisition_status":"Acquisition status not supplied","official_url":"https://github.com/google-deepmind/bbeh","indexing_mode":"index","profile":{"resources":[],"task_format":"","scoring":"","metric":"","size":"","answer_access":"","license":"","citation":"","maintainer":"","released":"","why_hard":"","related":[],"history":"","construction":"","limitations":"","saturation":"","scores":[]}},"task_id":"badd425e-82ce-55fb-8d38-78de9c60ca51","task_key":"bbeh~5fnycc--full--badd425e-82ce-55fb-8d38-78de9c60ca51","task_revision_id":"1","upstream_id":"","short_description":"The following is a description of a funny cartoon for the New Yorker Caption…","config":"bbeh_nycc","split":"full","body":"{\"input\":\"The following is a description of a funny cartoon for the New Yorker Caption Contest\\n\\n  Description: Two ears of corn with legs are standing on a road. They are popping popcorn as they walk.\\n\\n  Which of the following captions is the funniest?\\n(A) Is that your corn popping or are you happy to see me?\\n(B) It turns out it is the heat, not the humidity.\\n(C) Great. Now the Uber’s gonna charge us for luggage.\\n(D) So I said \\\"I'm all ears\\\" and she just rolls her eyes and says that my jokes are too corny.  Seriously?\\n(E) Real nice, Frank. Real nice.\\n(F) I was going to wait for a more romantic moment, but I’m ready to pop the question.\\n(G) You have a solution? I'm all ears.\\n(H) If this continues, we'll be husks of our former selves.\\n(I) I told you we shouldn't have moved to Phoenix.\\n(J) Now we know how the glaciers feel.\\n\"}","display_format":"text","language":"","answer_status":"published","assets":[],"source_url":"https://github.com/google-deepmind/bbeh","history":"initial import","indexing_mode":"index","subproblems":[],"grids":[]}