{"kind":"task","effective_mode":"full","benchmark":{"kind":"benchmark","effective_mode":"full","slug":"bbeh","formal_name":"BIG-Bench Extra Hard (BBEH)","introduction":"BIG-Bench Extra Hard evaluates language-model reasoning through challenging task categories. It offers full and mini distributions; this import contains the full distribution: 4,520 questions across all 23 task categories.","introduction_ja":"","introduction_en":"","category":"Category not supplied","task_count":null,"acquisition_status":"Acquisition status not supplied","official_url":"https://github.com/google-deepmind/bbeh","indexing_mode":"index","profile":{"resources":[],"task_format":"","scoring":"","metric":"","size":"","answer_access":"","license":"","citation":"","maintainer":"","released":"","why_hard":"","related":[],"history":"","construction":"","limitations":"","saturation":"","scores":[]}},"task_id":"2ba0f041-fc2c-58f8-a590-bc650c587be7","task_key":"bbeh~5fnycc--full--2ba0f041-fc2c-58f8-a590-bc650c587be7","task_revision_id":"1","upstream_id":"","short_description":"The following is a description of a funny cartoon for the New Yorker Caption…","config":"bbeh_nycc","split":"full","body":"{\"input\":\"The following is a description of a funny cartoon for the New Yorker Caption Contest\\n\\n  Description: A giant snail is in the middle of a city, towering over buildings and causing destruction. A police officer is looking at the snail through binoculars while a scientist stands next to him.\\n\\n  Which of the following captions is the funniest?\\n(A) Every disaster movie starts with scientists being ignored\\n(B) I’m afraid we can’t do anything. Technically, it’s working from home.\\n(C) We need to find cover in the next hour or two.\\n(D) Give it another week, and you won't need the binoculars.\\n(E) We traced the slime trail back to Ted Cruz's house.\\n(F) On a positive note, the slime trail melted the snow.\\n(G) Six weeks of destruction all the way from Times Square to the Empire State Building.\\n(H) Giuliani promised that the lost election ballots would arrive by snail mail...\\n(I) Sir, we had better get moving or we'll be dead in a week!\\n(J) The French chefs are on their way.\\n\"}","display_format":"text","language":"","answer_status":"published","assets":[],"source_url":"https://github.com/google-deepmind/bbeh","history":"initial import","indexing_mode":"index","subproblems":[],"grids":[]}