{"kind":"task","effective_mode":"full","benchmark":{"kind":"benchmark","effective_mode":"full","slug":"bbeh","formal_name":"BIG-Bench Extra Hard (BBEH)","introduction":"BIG-Bench Extra Hard evaluates language-model reasoning through challenging task categories. It offers full and mini distributions; this import contains the full distribution: 4,520 questions across all 23 task categories.","introduction_ja":"","introduction_en":"","category":"Category not supplied","task_count":null,"acquisition_status":"Acquisition status not supplied","official_url":"https://github.com/google-deepmind/bbeh","indexing_mode":"index","profile":{"resources":[],"task_format":"","scoring":"","metric":"","size":"","answer_access":"","license":"","citation":"","maintainer":"","released":"","why_hard":"","related":[],"history":"","construction":"","limitations":"","saturation":"","scores":[]}},"task_id":"58d8b9d1-57b4-587e-86f4-7a979057734f","task_key":"bbeh~5fnycc--full--58d8b9d1-57b4-587e-86f4-7a979057734f","task_revision_id":"1","upstream_id":"","short_description":"The following is a description of a funny cartoon for the New Yorker Caption…","config":"bbeh_nycc","split":"full","body":"{\"input\":\"The following is a description of a funny cartoon for the New Yorker Caption Contest\\n\\n  Description: A man in a bathrobe is standing in a doorway, looking surprised. A woman is standing in the room, and there is a large vacuum cleaner in the middle of the room. The scene appears to be set at night.\\n\\n  Which of the following captions is the funniest?\\n(A) On behalf of loud snorers, this is cultural appropriation.\\n(B) Its very late, Mrs. Hoover.\\n(C) Eureka, I’ve found it.\\n(D) I'm asking for my dog.\\n(E) It's much smaller than I imagined.\\\"\\n(F) Just wondering if you have seen Barky, our Great Dane?\\n(G) Yeah, pretty good thanks. Yourself?\\n(H) I'd be okay with it, but my Roomba won't come out from under the bed.\\n(I) I SAID, PLEASE TURN IT OFF! MY CATS ARE STUCK TO THE CEILING!\\n(J) The building does not allow particle accelerators.\\n\"}","display_format":"text","language":"","answer_status":"published","assets":[],"source_url":"https://github.com/google-deepmind/bbeh","history":"initial import","indexing_mode":"index","subproblems":[],"grids":[]}