{"kind":"task","effective_mode":"full","benchmark":{"kind":"benchmark","effective_mode":"full","slug":"bbeh","formal_name":"BIG-Bench Extra Hard (BBEH)","introduction":"BIG-Bench Extra Hard evaluates language-model reasoning through challenging task categories. It offers full and mini distributions; this import contains the full distribution: 4,520 questions across all 23 task categories.","introduction_ja":"","introduction_en":"","category":"Category not supplied","task_count":null,"acquisition_status":"Acquisition status not supplied","official_url":"https://github.com/google-deepmind/bbeh","indexing_mode":"index","profile":{"resources":[],"task_format":"","scoring":"","metric":"","size":"","answer_access":"","license":"","citation":"","maintainer":"","released":"","why_hard":"","related":[],"history":"","construction":"","limitations":"","saturation":"","scores":[]}},"task_id":"e3946680-a8de-5f3b-8d41-2725d35ab87c","task_key":"bbeh~5fnycc--full--e3946680-a8de-5f3b-8d41-2725d35ab87c","task_revision_id":"1","upstream_id":"","short_description":"The following is a description of a funny cartoon for the New Yorker Caption…","config":"bbeh_nycc","split":"full","body":"{\"input\":\"The following is a description of a funny cartoon for the New Yorker Caption Contest\\n\\n  Description: Anthropomorphic vegetables, including broccoli, asparagus, and a tomato, are sitting at a table in a restaurant, eating meals. In the background, a human couple is also dining and looking at the vegetables.\\n\\n  Which of the following captions is the funniest?\\n(A) Somebody should tell the kitchen that their vegetable have gone bad.\\n(B) It could get really messy if we have to Heimlich the tomato.\\n(C) You are what what you eat eats.\\n(D) I'm beginning to have my doubts about 'farm to table' dining.\\n(E) I liked it better when they consumed dead animals underground through roots.\\n(F) I didn’t know this was a farm-to-table restaurant.\\n(G) Well, I do see that it can be a tough choice between veganism and cannibalism.\\n(H) Well, would you rather they ate dirt and manure?\\n(I) Omg, is this a veggie burger? Oh, no, Uncle Joe!\\n(J) My spleen burger is delicious, how are your American thighs?\\n\"}","display_format":"text","language":"","answer_status":"published","assets":[],"source_url":"https://github.com/google-deepmind/bbeh","history":"initial import","indexing_mode":"index","subproblems":[],"grids":[]}