{"kind":"task","effective_mode":"full","benchmark":{"kind":"benchmark","effective_mode":"full","slug":"bbeh","formal_name":"BIG-Bench Extra Hard (BBEH)","introduction":"BIG-Bench Extra Hard evaluates language-model reasoning through challenging task categories. It offers full and mini distributions; this import contains the full distribution: 4,520 questions across all 23 task categories.","introduction_ja":"","introduction_en":"","category":"Category not supplied","task_count":null,"acquisition_status":"Acquisition status not supplied","official_url":"https://github.com/google-deepmind/bbeh","indexing_mode":"index","profile":{"resources":[],"task_format":"","scoring":"","metric":"","size":"","answer_access":"","license":"","citation":"","maintainer":"","released":"","why_hard":"","related":[],"history":"","construction":"","limitations":"","saturation":"","scores":[]}},"task_id":"e37fbc0d-57a0-5b57-898d-631bb0bede4c","task_key":"bbeh~5fnycc--full--e37fbc0d-57a0-5b57-898d-631bb0bede4c","task_revision_id":"1","upstream_id":"","short_description":"The following is a description of a funny cartoon for the New Yorker Caption…","config":"bbeh_nycc","split":"full","body":"{\"input\":\"The following is a description of a funny cartoon for the New Yorker Caption Contest\\n\\n  Description: Two antelopes are standing on a stage, one of them holding a piece of paper and looking nervous. They are addressing an audience of lions who are seated and looking at them.\\n\\n  Which of the following captions is the funniest?\\n(A) Just read! Don't just stand there like a you-know-what in the headlights!\\n(B) No no, they are licking their lips with anticipation…!\\n(C) Idiot.  No wonder you were able to get us the gig.  This is not one of The Lions Clubs I had in mind.\\n(D) No! You go first.\\n(E) All you need to do is introduce the Zebra Choir.  And be quick about it.\\n(F) Relax, it's an after dinner speech.\\n(G) Grin and bear it.\\n(H) Don't worry, the last comedian killed.\\n(I) Just remember, taking the free public speaking class at the Lions Club was all your idea.\\\"\\n(J) Just picture that they're fully clothed.\\n\"}","display_format":"text","language":"","answer_status":"published","assets":[],"source_url":"https://github.com/google-deepmind/bbeh","history":"initial import","indexing_mode":"index","subproblems":[],"grids":[]}