{"kind":"task","effective_mode":"full","benchmark":{"kind":"benchmark","effective_mode":"full","slug":"bbeh","formal_name":"BIG-Bench Extra Hard (BBEH)","introduction":"BIG-Bench Extra Hard evaluates language-model reasoning through challenging task categories. It offers full and mini distributions; this import contains the full distribution: 4,520 questions across all 23 task categories.","introduction_ja":"","introduction_en":"","category":"Category not supplied","task_count":null,"acquisition_status":"Acquisition status not supplied","official_url":"https://github.com/google-deepmind/bbeh","indexing_mode":"index","profile":{"resources":[],"task_format":"","scoring":"","metric":"","size":"","answer_access":"","license":"","citation":"","maintainer":"","released":"","why_hard":"","related":[],"history":"","construction":"","limitations":"","saturation":"","scores":[]}},"task_id":"4d792338-57a9-53e5-9f74-3dde1a181d28","task_key":"bbeh~5fnycc--full--4d792338-57a9-53e5-9f74-3dde1a181d28","task_revision_id":"1","upstream_id":"","short_description":"The following is a description of a funny cartoon for the New Yorker Caption…","config":"bbeh_nycc","split":"full","body":"{\"input\":\"The following is a description of a funny cartoon for the New Yorker Caption Contest\\n\\n  Description: Two scientists are observing a rat navigating through a miniature corn maze on a table. The male scientist is holding a clipboard and taking notes while the female scientist watches. There are cages with other animals in the background.\\n\\n  Which of the following captions is the funniest?\\n(A) To support the ban on animal cruelty, we use mechanical rats\\n(B) Next, we're going to see if it can survive a night in a tiny haunted cabin.\\n(C) The grant required we use sustainable materials.\\n(D) It's a new type of maze and a new species of maize.\\n(E) This is Donald. He is going to get lost and then declare victory.\\n(F) Whatever the result, at we have mini-corn.\\n(G) So who’s idea was it to use field mice?\\n(H) We've gone over to recyclables for the walls.\\n(I) Next is apple-bobbing and pumpkin-chucking.\\n(J) We'll just make up the results\\\"\\n\"}","display_format":"text","language":"","answer_status":"published","assets":[],"source_url":"https://github.com/google-deepmind/bbeh","history":"initial import","indexing_mode":"index","subproblems":[],"grids":[]}