{"kind":"task","effective_mode":"full","benchmark":{"kind":"benchmark","effective_mode":"full","slug":"bbeh","formal_name":"BIG-Bench Extra Hard (BBEH)","introduction":"BIG-Bench Extra Hard evaluates language-model reasoning through challenging task categories. It offers full and mini distributions; this import contains the full distribution: 4,520 questions across all 23 task categories.","introduction_ja":"","introduction_en":"","category":"Category not supplied","task_count":null,"acquisition_status":"Acquisition status not supplied","official_url":"https://github.com/google-deepmind/bbeh","indexing_mode":"index","profile":{"resources":[],"task_format":"","scoring":"","metric":"","size":"","answer_access":"","license":"","citation":"","maintainer":"","released":"","why_hard":"","related":[],"history":"","construction":"","limitations":"","saturation":"","scores":[]}},"task_id":"c01b536a-c3ef-5fc4-af63-cd39b2fd1156","task_key":"bbeh~5fnycc--full--c01b536a-c3ef-5fc4-af63-cd39b2fd1156","task_revision_id":"1","upstream_id":"","short_description":"The following is a description of a funny cartoon for the New Yorker Caption…","config":"bbeh_nycc","split":"full","body":"{\"input\":\"The following is a description of a funny cartoon for the New Yorker Caption Contest\\n\\n  Description: A group of vultures is sitting in a lecture hall, each in front of a laptop. Two humans are standing at the front of the hall, observing the scene.\\n\\n  Which of the following captions is the funniest?\\n(A) They're close to the solution of why chickens cross the road.\\n(B) Due to budget cuts, we can now only afford to pay respondents in breadcrumbs.\\n(C) If it looks like a duck test, it's a duck test.\\n(D) When I snap my fingers, you will go home, eat breadcrumbs and crap all over the park.\\n(E) TRUMP'S ONLINE APPROVAL POLLING\\n(F) The feathers don't begin to fly until the bird seed icon appears.\\n(G) Yeah, touch-typing - I get by just fine with hunt-and-peck.\\n(H) They look confused, is the Wi-Fi working?\\n(I) Please tell me they're proficient in Excel.\\n(J) So much for diversity\\n\"}","display_format":"text","language":"","answer_status":"published","assets":[],"source_url":"https://github.com/google-deepmind/bbeh","history":"initial import","indexing_mode":"index","subproblems":[],"grids":[]}