{"kind":"task","effective_mode":"full","benchmark":{"kind":"benchmark","effective_mode":"full","slug":"bbeh","formal_name":"BIG-Bench Extra Hard (BBEH)","introduction":"BIG-Bench Extra Hard evaluates language-model reasoning through challenging task categories. It offers full and mini distributions; this import contains the full distribution: 4,520 questions across all 23 task categories.","introduction_ja":"","introduction_en":"","category":"Category not supplied","task_count":null,"acquisition_status":"Acquisition status not supplied","official_url":"https://github.com/google-deepmind/bbeh","indexing_mode":"index","profile":{"resources":[],"task_format":"","scoring":"","metric":"","size":"","answer_access":"","license":"","citation":"","maintainer":"","released":"","why_hard":"","related":[],"history":"","construction":"","limitations":"","saturation":"","scores":[]}},"task_id":"7ef7ada6-c585-5b08-bd99-385959311a24","task_key":"bbeh~5fnycc--full--7ef7ada6-c585-5b08-bd99-385959311a24","task_revision_id":"1","upstream_id":"","short_description":"The following is a description of a funny cartoon for the New Yorker Caption…","config":"bbeh_nycc","split":"full","body":"{\"input\":\"The following is a description of a funny cartoon for the New Yorker Caption Contest\\n\\n  Description: A bearded man holding a staff is standing in front of a stone structure with an open door. Two large fish are lying on the steps leading to the door.\\n\\n  Which of the following captions is the funniest?\\n(A) Hey, I think we're gonna walk.\\n(B) Sir Noah, do we go in the ark or under it?\\\"\\n(C) Yeay...dry land\\n(D) You interrupted our spawning season for this!?\\n(E) Holy Moses it's about time!\\n(F) We’re here to feed the bears.\\n(G) You took us out of school for this?\\n(H) You're forgetting the loaves.\\n(I) Did you ever consider we may have enjoyed the flood?\\n(J) Well God told me he was sending the flood as a gift, so I think we’re kind of at an impasse here.\\n\"}","display_format":"text","language":"","answer_status":"published","assets":[],"source_url":"https://github.com/google-deepmind/bbeh","history":"initial import","indexing_mode":"index","subproblems":[],"grids":[]}