{"kind":"task","effective_mode":"full","benchmark":{"kind":"benchmark","effective_mode":"full","slug":"bbeh","formal_name":"BIG-Bench Extra Hard (BBEH)","introduction":"BIG-Bench Extra Hard evaluates language-model reasoning through challenging task categories. It offers full and mini distributions; this import contains the full distribution: 4,520 questions across all 23 task categories.","introduction_ja":"","introduction_en":"","category":"Category not supplied","task_count":null,"acquisition_status":"Acquisition status not supplied","official_url":"https://github.com/google-deepmind/bbeh","indexing_mode":"index","profile":{"resources":[],"task_format":"","scoring":"","metric":"","size":"","answer_access":"","license":"","citation":"","maintainer":"","released":"","why_hard":"","related":[],"history":"","construction":"","limitations":"","saturation":"","scores":[]}},"task_id":"668422c2-8a4d-56ac-8485-29a48d493492","task_key":"bbeh~5fnycc--full--668422c2-8a4d-56ac-8485-29a48d493492","task_revision_id":"1","upstream_id":"","short_description":"The following is a description of a funny cartoon for the New Yorker Caption…","config":"bbeh_nycc","split":"full","body":"{\"input\":\"The following is a description of a funny cartoon for the New Yorker Caption Contest\\n\\n  Description: Two scientists are observing a group of mice navigating through a maze that resembles an art gallery. The mice are looking at the paintings on the walls.\\n\\n  Which of the following captions is the funniest?\\n(A) Let's go with the brie.\\n(B) First we tried giving typewriters to the monkeys.\\n(C) They're not taking selfies, so it won't apply to people.\\n(D) It's uncanny how much better they are at identifying forgeries than our experts.\\n(E) It's the preview to the opening of the new maze.\\n(F) Nothing has sold yet, but we’ve gotten a few nibbles.\\n(G) The art is a placebo.\\n(H) Art is its own reward.\\n(I) It's like Members Night at the Met!\\n(J) They keep licking the Impressionists - it must mean something.\\n\"}","display_format":"text","language":"","answer_status":"published","assets":[],"source_url":"https://github.com/google-deepmind/bbeh","history":"initial import","indexing_mode":"index","subproblems":[],"grids":[]}