{"kind":"task","effective_mode":"full","benchmark":{"kind":"benchmark","effective_mode":"full","slug":"bbeh","formal_name":"BIG-Bench Extra Hard (BBEH)","introduction":"BIG-Bench Extra Hard evaluates language-model reasoning through challenging task categories. It offers full and mini distributions; this import contains the full distribution: 4,520 questions across all 23 task categories.","introduction_ja":"","introduction_en":"","category":"Category not supplied","task_count":null,"acquisition_status":"Acquisition status not supplied","official_url":"https://github.com/google-deepmind/bbeh","indexing_mode":"index","profile":{"resources":[],"task_format":"","scoring":"","metric":"","size":"","answer_access":"","license":"","citation":"","maintainer":"","released":"","why_hard":"","related":[],"history":"","construction":"","limitations":"","saturation":"","scores":[]}},"task_id":"522a904e-0b89-500a-ac55-b9f59ef269bf","task_key":"bbeh~5fnycc--full--522a904e-0b89-500a-ac55-b9f59ef269bf","task_revision_id":"1","upstream_id":"","short_description":"The following is a description of a funny cartoon for the New Yorker Caption…","config":"bbeh_nycc","split":"full","body":"{\"input\":\"The following is a description of a funny cartoon for the New Yorker Caption Contest\\n\\n  Description: A bull is standing behind the counter in a china shop, carefully examining a piece of china. Two customers are watching the bull with surprise.\\n\\n  Which of the following captions is the funniest?\\n(A) Between breakage and sales he comes out about even.\\n(B) No one can sell durability quite like Bill\\n(C) He's addressing unconscious bias.\\n(D) I guess all those ballet lessons really paid off.\\n(E) It's 2021. What could possibly go wrong?\\n(F) Well, he's actually a steer, so technically...\\n(G) First metaphor I ever hired.\\n(H) His Community service.\\n(I) They always charge too much\\n(J) Things have been much calmer since we took away his Twitter.\\n\"}","display_format":"text","language":"","answer_status":"published","assets":[],"source_url":"https://github.com/google-deepmind/bbeh","history":"initial import","indexing_mode":"index","subproblems":[],"grids":[]}