{"kind":"task","effective_mode":"full","benchmark":{"kind":"benchmark","effective_mode":"full","slug":"bbeh","formal_name":"BIG-Bench Extra Hard (BBEH)","introduction":"BIG-Bench Extra Hard evaluates language-model reasoning through challenging task categories. It offers full and mini distributions; this import contains the full distribution: 4,520 questions across all 23 task categories.","introduction_ja":"","introduction_en":"","category":"Category not supplied","task_count":null,"acquisition_status":"Acquisition status not supplied","official_url":"https://github.com/google-deepmind/bbeh","indexing_mode":"index","profile":{"resources":[],"task_format":"","scoring":"","metric":"","size":"","answer_access":"","license":"","citation":"","maintainer":"","released":"","why_hard":"","related":[],"history":"","construction":"","limitations":"","saturation":"","scores":[]}},"task_id":"d17b2152-7811-57a2-9c66-e033336d19f0","task_key":"bbeh~5fnycc--full--d17b2152-7811-57a2-9c66-e033336d19f0","task_revision_id":"1","upstream_id":"","short_description":"The following is a description of a funny cartoon for the New Yorker Caption…","config":"bbeh_nycc","split":"full","body":"{\"input\":\"The following is a description of a funny cartoon for the New Yorker Caption Contest\\n\\n  Description: A person is looking out of their window at a neighboring building. In the neighboring building, the Grim Reaper is sitting by a window playing an electric guitar.\\n\\n  Which of the following captions is the funniest?\\n(A) Turn it down in there!\\n(B) Your music sucks...and my mother's just fine, thank you!\\n(C) Turn it down! it's loud enough to raise the dead!\\n(D) Stop killing me with that death metal!\\\"\\n(E) Do you take last requests?\\n(F) Is the end near?\\n(G) Better trade that axe in for a scythe!\\\"\\n(H) Don't make me come down there- Mr.  Death Metal\\n(I) Wait! That’s my apartment!\\n(J) I never could stand The Grateful Dead.\\n\"}","display_format":"text","language":"","answer_status":"published","assets":[],"source_url":"https://github.com/google-deepmind/bbeh","history":"initial import","indexing_mode":"index","subproblems":[],"grids":[]}