{"kind":"task","effective_mode":"full","benchmark":{"kind":"benchmark","effective_mode":"full","slug":"bbeh","formal_name":"BIG-Bench Extra Hard (BBEH)","introduction":"BIG-Bench Extra Hard evaluates language-model reasoning through challenging task categories. It offers full and mini distributions; this import contains the full distribution: 4,520 questions across all 23 task categories.","introduction_ja":"","introduction_en":"","category":"Category not supplied","task_count":null,"acquisition_status":"Acquisition status not supplied","official_url":"https://github.com/google-deepmind/bbeh","indexing_mode":"index","profile":{"resources":[],"task_format":"","scoring":"","metric":"","size":"","answer_access":"","license":"","citation":"","maintainer":"","released":"","why_hard":"","related":[],"history":"","construction":"","limitations":"","saturation":"","scores":[]}},"task_id":"a694f30e-3b7e-572e-a390-0d499f70e977","task_key":"bbeh~5fnycc--full--a694f30e-3b7e-572e-a390-0d499f70e977","task_revision_id":"1","upstream_id":"","short_description":"The following is a description of a funny cartoon for the New Yorker Caption…","config":"bbeh_nycc","split":"full","body":"{\"input\":\"The following is a description of a funny cartoon for the New Yorker Caption Contest\\n\\n  Description: A giant piano has crashed into a house, causing significant damage. Two people are standing on the sidewalk, looking at the destruction in disbelief.\\n\\n  Which of the following captions is the funniest?\\n(A) Ding dong. See, the pitch is dead.\\n(B) Now there was an encore that brought the house down.\\n(C) You signed up for Amazon's drone service too?\\n(D) I'll buy it for a dollar.\\n(E) Bezos needs to work on those drone deliveries.\\n(F) Amazon drone delivery,” you said. \\n“It’s so convenient,” you said.\\n(G) We may need to get it tuned....\\n(H) It could be worse. It could be raining.\\n(I) Are those the new house keys you ordered?\\n(J) At least you didn’t leave the oven on.\\n\"}","display_format":"text","language":"","answer_status":"published","assets":[],"source_url":"https://github.com/google-deepmind/bbeh","history":"initial import","indexing_mode":"index","subproblems":[],"grids":[]}