{"kind":"task","effective_mode":"full","benchmark":{"kind":"benchmark","effective_mode":"full","slug":"bbeh","formal_name":"BIG-Bench Extra Hard (BBEH)","introduction":"BIG-Bench Extra Hard evaluates language-model reasoning through challenging task categories. It offers full and mini distributions; this import contains the full distribution: 4,520 questions across all 23 task categories.","introduction_ja":"","introduction_en":"","category":"Category not supplied","task_count":null,"acquisition_status":"Acquisition status not supplied","official_url":"https://github.com/google-deepmind/bbeh","indexing_mode":"index","profile":{"resources":[],"task_format":"","scoring":"","metric":"","size":"","answer_access":"","license":"","citation":"","maintainer":"","released":"","why_hard":"","related":[],"history":"","construction":"","limitations":"","saturation":"","scores":[]}},"task_id":"c6750c32-70ae-5b1c-ab98-396400b70118","task_key":"bbeh~5fnycc--full--c6750c32-70ae-5b1c-ab98-396400b70118","task_revision_id":"1","upstream_id":"","short_description":"The following is a description of a funny cartoon for the New Yorker Caption…","config":"bbeh_nycc","split":"full","body":"{\"input\":\"The following is a description of a funny cartoon for the New Yorker Caption Contest\\n\\n  Description: A king is sitting on his throne while a servant reads a scroll to him. Above the king's head, a large sword is hanging by a thread.\\n\\n  Which of the following captions is the funniest?\\n(A) Sire, your royal life insurance policy covers everything except: Death by giant sword while sitting on the throne.\\n(B) The cease and desist letter is from some guy named Damocles.\\n(C) And the king lived happily severed after.\\n(D) It appears that has been taken, Sire. What about Mack the Dagger?\\n(E) King Occam, I think the Razor might be a better option\\n(F) It's the new guillotine 2.0, sir.\\n(G) Your majesty, the queen beseeches you for a trial separation!\\n(H) A gift from your brother, your Majesty. He sends his regards.\\n(I) It says the queen knows about your mistress. So you've got to split.\\n(J) Not just mightier, but considerably more safe as well!\\n\"}","display_format":"text","language":"","answer_status":"published","assets":[],"source_url":"https://github.com/google-deepmind/bbeh","history":"initial import","indexing_mode":"index","subproblems":[],"grids":[]}