{"kind":"task","effective_mode":"full","benchmark":{"kind":"benchmark","effective_mode":"full","slug":"bbeh","formal_name":"BIG-Bench Extra Hard (BBEH)","introduction":"BIG-Bench Extra Hard evaluates language-model reasoning through challenging task categories. It offers full and mini distributions; this import contains the full distribution: 4,520 questions across all 23 task categories.","introduction_ja":"","introduction_en":"","category":"Category not supplied","task_count":null,"acquisition_status":"Acquisition status not supplied","official_url":"https://github.com/google-deepmind/bbeh","indexing_mode":"index","profile":{"resources":[],"task_format":"","scoring":"","metric":"","size":"","answer_access":"","license":"","citation":"","maintainer":"","released":"","why_hard":"","related":[],"history":"","construction":"","limitations":"","saturation":"","scores":[]}},"task_id":"7c74633d-b7d5-5f50-86bf-212442902052","task_key":"bbeh~5fnycc--full--7c74633d-b7d5-5f50-86bf-212442902052","task_revision_id":"1","upstream_id":"","short_description":"The following is a description of a funny cartoon for the New Yorker Caption…","config":"bbeh_nycc","split":"full","body":"{\"input\":\"The following is a description of a funny cartoon for the New Yorker Caption Contest\\n\\n  Description: A doctor is examining a patient who is dressed as a king, complete with a crown and a fur-lined robe. The patient is sitting on an examination table, and the doctor is reading from a clipboard.\\n\\n  Which of the following captions is the funniest?\\n(A) I can't turn it back from gold, but you can use it to pay your deductible.\\n(B) Well, King Midas, you really shouldn’t have touched that area.\\n(C) Could you please disrobe?\\n(D) Looking at your family history, you're at an increased risk for premature beheading.\\n(E) Thy penchant for meade hath rendered thee crapulent.\\n(F) It's perfectly normal to wake up in the morning feeling feudal.\\n(G) I'm afraid you'll have to switch to peasant-sized candy bars.\\n(H) You should't have glued it on.\\n(I) Unfortunately, it has to do with your family jewels.\\n(J) Hum, your tests results actually do show you have blue blood.\\n\"}","display_format":"text","language":"","answer_status":"published","assets":[],"source_url":"https://github.com/google-deepmind/bbeh","history":"initial import","indexing_mode":"index","subproblems":[],"grids":[]}