{"kind":"task","effective_mode":"full","benchmark":{"kind":"benchmark","effective_mode":"full","slug":"bbeh","formal_name":"BIG-Bench Extra Hard (BBEH)","introduction":"BIG-Bench Extra Hard evaluates language-model reasoning through challenging task categories. It offers full and mini distributions; this import contains the full distribution: 4,520 questions across all 23 task categories.","introduction_ja":"","introduction_en":"","category":"Category not supplied","task_count":null,"acquisition_status":"Acquisition status not supplied","official_url":"https://github.com/google-deepmind/bbeh","indexing_mode":"index","profile":{"resources":[],"task_format":"","scoring":"","metric":"","size":"","answer_access":"","license":"","citation":"","maintainer":"","released":"","why_hard":"","related":[],"history":"","construction":"","limitations":"","saturation":"","scores":[]}},"task_id":"c27cdff6-4ac8-52d6-bac8-4ca01ab7eb6c","task_key":"bbeh~5fnycc--full--c27cdff6-4ac8-52d6-bac8-4ca01ab7eb6c","task_revision_id":"1","upstream_id":"","short_description":"The following is a description of a funny cartoon for the New Yorker Caption…","config":"bbeh_nycc","split":"full","body":"{\"input\":\"The following is a description of a funny cartoon for the New Yorker Caption Contest\\n\\n  Description: A peanut character dressed in a top hat and monocle is standing in a room, facing a large elephant that takes up most of the space.\\n\\n  Which of the following captions is the funniest?\\n(A) Consider carefully. If you have an undiagnosed peanut allergy and you eat me then it’s lights out for you too.\\n(B) Either you’re an elephant or I am just plain nuts!\\n(C) I'm just their mascot.\\n(D) What do you mean, 'we can do this the smooth way or the crunchy way'...?!\\n(E) You don't remember me?\\n(F) So I guess we can address the other \\\"elephant in the room\\\"...why a peanut is wearing a  monacle.\\n(G) So we meet at last, Elephant In the Room…\\n(H) Can I offer you some grass?\\n(I) The Rorschach test revealed Mr. Peanut's deepest fear\\n(J) “I think I over did it on the micro dosing.”\\n\"}","display_format":"text","language":"","answer_status":"published","assets":[],"source_url":"https://github.com/google-deepmind/bbeh","history":"initial import","indexing_mode":"index","subproblems":[],"grids":[]}