{"kind":"task","effective_mode":"full","benchmark":{"kind":"benchmark","effective_mode":"full","slug":"bbeh","formal_name":"BIG-Bench Extra Hard (BBEH)","introduction":"BIG-Bench Extra Hard evaluates language-model reasoning through challenging task categories. It offers full and mini distributions; this import contains the full distribution: 4,520 questions across all 23 task categories.","introduction_ja":"","introduction_en":"","category":"Category not supplied","task_count":null,"acquisition_status":"Acquisition status not supplied","official_url":"https://github.com/google-deepmind/bbeh","indexing_mode":"index","profile":{"resources":[],"task_format":"","scoring":"","metric":"","size":"","answer_access":"","license":"","citation":"","maintainer":"","released":"","why_hard":"","related":[],"history":"","construction":"","limitations":"","saturation":"","scores":[]}},"task_id":"7a2f74da-bb62-5260-91f1-0f03d2310478","task_key":"bbeh~5fnycc--full--7a2f74da-bb62-5260-91f1-0f03d2310478","task_revision_id":"1","upstream_id":"","short_description":"The following is a description of a funny cartoon for the New Yorker Caption…","config":"bbeh_nycc","split":"full","body":"{\"input\":\"The following is a description of a funny cartoon for the New Yorker Caption Contest\\n\\n  Description: A group of people are standing together, each looking at their phones. Four of them have emoji heads: a smiley face with sunglasses, a flame, a purple devil, and a peach. The fifth person, in the middle, has a normal human head and looks confused or concerned.\\n\\n  Which of the following captions is the funniest?\\n(A) The millennial urge to write \\\"lol\\\" is stronger guys, sorry.\\n(B) So I guess this makes me the eggplant?\\n(C) OMG! It's true! You are what you tweet!\\n(D) Why do I always have to be the poop emoji!\\n(E) Somehow, I'm not in sync with you all.\\n(F) I told you I’m not getting the update until the bugs are worked out.\\n(G) Um what does the human face emoji mean?\\n(H) Well…ANDROIDS HAVE BETTER CAMERA QUALITY!\\n(I) Has Mensa been taken over by A.I.?\\n(J) But Mama said to use my words.\\n\"}","display_format":"text","language":"","answer_status":"published","assets":[],"source_url":"https://github.com/google-deepmind/bbeh","history":"initial import","indexing_mode":"index","subproblems":[],"grids":[]}