{"kind":"task","effective_mode":"full","benchmark":{"kind":"benchmark","effective_mode":"full","slug":"bbeh","formal_name":"BIG-Bench Extra Hard (BBEH)","introduction":"BIG-Bench Extra Hard evaluates language-model reasoning through challenging task categories. It offers full and mini distributions; this import contains the full distribution: 4,520 questions across all 23 task categories.","introduction_ja":"","introduction_en":"","category":"Category not supplied","task_count":null,"acquisition_status":"Acquisition status not supplied","official_url":"https://github.com/google-deepmind/bbeh","indexing_mode":"index","profile":{"resources":[],"task_format":"","scoring":"","metric":"","size":"","answer_access":"","license":"","citation":"","maintainer":"","released":"","why_hard":"","related":[],"history":"","construction":"","limitations":"","saturation":"","scores":[]}},"task_id":"ccdd2273-c5b3-5f2d-9526-403ddb7bec91","task_key":"bbeh~5fnycc--full--ccdd2273-c5b3-5f2d-9526-403ddb7bec91","task_revision_id":"1","upstream_id":"","short_description":"The following is a description of a funny cartoon for the New Yorker Caption…","config":"bbeh_nycc","split":"full","body":"{\"input\":\"The following is a description of a funny cartoon for the New Yorker Caption Contest\\n\\n  Description: Three people are standing in front of the gates of heaven, looking up at a skier who is flying through the air. A bearded man, possibly St. Peter, is standing at a podium with a book, observing the scene.\\n\\n  Which of the following captions is the funniest?\\n(A) Sorry.  You have the wrong address.  These are the gnarly gates.\\n(B) Yes, there are many ways to enter heaven. However, that is not one of them.\\n(C) You’d think he’d wear a helmet this time\\n(D) What do you think people reading the New Yorker are going to say.\\n(E) They say the skiing is to die for.\\n(F) It's a nice try...but there's no snow where he's headed.\\n(G) Oh, It's just a returning snow angel.\\n(H) You won’t need lessons for anything, trust me!\\n(I) Free all day pass in heaven!\\n(J) Hey look! It’s King Slalomon!\\n\"}","display_format":"text","language":"","answer_status":"published","assets":[],"source_url":"https://github.com/google-deepmind/bbeh","history":"initial import","indexing_mode":"index","subproblems":[],"grids":[]}