{"kind":"task","effective_mode":"full","benchmark":{"kind":"benchmark","effective_mode":"full","slug":"bbeh","formal_name":"BIG-Bench Extra Hard (BBEH)","introduction":"BIG-Bench Extra Hard evaluates language-model reasoning through challenging task categories. It offers full and mini distributions; this import contains the full distribution: 4,520 questions across all 23 task categories.","introduction_ja":"","introduction_en":"","category":"Category not supplied","task_count":null,"acquisition_status":"Acquisition status not supplied","official_url":"https://github.com/google-deepmind/bbeh","indexing_mode":"index","profile":{"resources":[],"task_format":"","scoring":"","metric":"","size":"","answer_access":"","license":"","citation":"","maintainer":"","released":"","why_hard":"","related":[],"history":"","construction":"","limitations":"","saturation":"","scores":[]}},"task_id":"25f6dcd1-b542-51f7-b5ea-55d68e1e0d92","task_key":"bbeh~5fnycc--full--25f6dcd1-b542-51f7-b5ea-55d68e1e0d92","task_revision_id":"1","upstream_id":"","short_description":"The following is a description of a funny cartoon for the New Yorker Caption…","config":"bbeh_nycc","split":"full","body":"{\"input\":\"The following is a description of a funny cartoon for the New Yorker Caption Contest\\n\\n  Description: Two dinosaurs are having a meeting in an office. One dinosaur is sitting behind a desk, and the other is sitting in a chair in front of the desk. Outside the window, a meteor is falling from the sky.\\n\\n  Which of the following captions is the funniest?\\n(A) we're all going to have to adjust to a new normal\\n(B) Tell me about a time you identified a problem that others didn't see coming.\\n(C) I’m glad you caught that error, Jerry. It could have been the end of the world.\\n(D) All this crazy panic over Comet 19!\\n(E) Anyway, happy new year. I have a good feeling about 2020.\\n(F) Things have gone pretty well since they started draining the swamp!\\n(G) I can’t help but notice that you seem a little distracted\\n(H) Just sign here, and it will be over in a flash.\\n(I) That'll never happen.\\\"\\n(J) Thanks John, oh, and one last thing.\\n\"}","display_format":"text","language":"","answer_status":"published","assets":[],"source_url":"https://github.com/google-deepmind/bbeh","history":"initial import","indexing_mode":"index","subproblems":[],"grids":[]}