{"kind":"task","effective_mode":"full","benchmark":{"kind":"benchmark","effective_mode":"full","slug":"bbeh","formal_name":"BIG-Bench Extra Hard (BBEH)","introduction":"BIG-Bench Extra Hard evaluates language-model reasoning through challenging task categories. It offers full and mini distributions; this import contains the full distribution: 4,520 questions across all 23 task categories.","introduction_ja":"","introduction_en":"","category":"Category not supplied","task_count":null,"acquisition_status":"Acquisition status not supplied","official_url":"https://github.com/google-deepmind/bbeh","indexing_mode":"index","profile":{"resources":[],"task_format":"","scoring":"","metric":"","size":"","answer_access":"","license":"","citation":"","maintainer":"","released":"","why_hard":"","related":[],"history":"","construction":"","limitations":"","saturation":"","scores":[]}},"task_id":"aab3b9bb-e067-5c7b-b912-b5b3b645ffc8","task_key":"bbeh~5fnycc--full--aab3b9bb-e067-5c7b-b912-b5b3b645ffc8","task_revision_id":"1","upstream_id":"","short_description":"The following is a description of a funny cartoon for the New Yorker Caption…","config":"bbeh_nycc","split":"full","body":"{\"input\":\"The following is a description of a funny cartoon for the New Yorker Caption Contest\\n\\n  Description: Two mice are looking at an instruction manual while sitting in front of a pile of flat-pack furniture pieces and screws.\\n\\n  Which of the following captions is the funniest?\\n(A) See, I told you. It says right here, “Cheese not included.”\\n(B) I think we can build a better one.\\n(C) Cutbacks\\n(D) Fine. See if YOU can do any better without thumbs.\\n(E) This would be easier with opposable thumbs.\\n(F) Let's do the maze at IKEA, you said... It'll be FUN, you said...\\n(G) I think this whole animal testing thing has gone a bit far.\\n(H) I'm sure the website said cheese was included.\\n(I) Now that we've taken it apart it should be easier to find the food.\\n(J) The cat will never find us once we get this thing built.\\n\"}","display_format":"text","language":"","answer_status":"published","assets":[],"source_url":"https://github.com/google-deepmind/bbeh","history":"initial import","indexing_mode":"index","subproblems":[],"grids":[]}