{"kind":"task","effective_mode":"full","benchmark":{"kind":"benchmark","effective_mode":"full","slug":"bbeh","formal_name":"BIG-Bench Extra Hard (BBEH)","introduction":"BIG-Bench Extra Hard evaluates language-model reasoning through challenging task categories. It offers full and mini distributions; this import contains the full distribution: 4,520 questions across all 23 task categories.","introduction_ja":"","introduction_en":"","category":"Category not supplied","task_count":null,"acquisition_status":"Acquisition status not supplied","official_url":"https://github.com/google-deepmind/bbeh","indexing_mode":"index","profile":{"resources":[],"task_format":"","scoring":"","metric":"","size":"","answer_access":"","license":"","citation":"","maintainer":"","released":"","why_hard":"","related":[],"history":"","construction":"","limitations":"","saturation":"","scores":[]}},"task_id":"73ca742c-411a-584b-a413-c0688de6eaa3","task_key":"bbeh~5fnycc--full--73ca742c-411a-584b-a413-c0688de6eaa3","task_revision_id":"1","upstream_id":"","short_description":"The following is a description of a funny cartoon for the New Yorker Caption…","config":"bbeh_nycc","split":"full","body":"{\"input\":\"The following is a description of a funny cartoon for the New Yorker Caption Contest\\n\\n  Description: Two people are standing next to a large hot dog-shaped vehicle at a gas station. One person appears to be showing something on a tablet to the other person.\\n\\n  Which of the following captions is the funniest?\\n(A) You’re low on oil but high in sodium.”\\n(B) Don’t think the old dog can cut the mustard any more.\\n(C) Take it to Nathan’s.\\nAsk for the Deli Style.\\n(D) I wish, but I need Grey Poupon.  We're from New York.\\n(E) You’re low on mustard. You use Dijon or regular?\\n(F) Looks like you're overdue for a mustard change\\n(G) How long have you been driving with the Check Condiment light on?\\n(H) You are down two quarts of Dijon. If you don’t fill it up, it will get wurst.\\n(I) More than an oil change.  I would recommend vegan.\\n(J) I see the problem now--this car was made in Chicago, you're not supposed to use ketchup.\\n\"}","display_format":"text","language":"","answer_status":"published","assets":[],"source_url":"https://github.com/google-deepmind/bbeh","history":"initial import","indexing_mode":"index","subproblems":[],"grids":[]}