{"kind":"task","effective_mode":"full","benchmark":{"kind":"benchmark","effective_mode":"full","slug":"bbeh","formal_name":"BIG-Bench Extra Hard (BBEH)","introduction":"BIG-Bench Extra Hard evaluates language-model reasoning through challenging task categories. It offers full and mini distributions; this import contains the full distribution: 4,520 questions across all 23 task categories.","introduction_ja":"","introduction_en":"","category":"Category not supplied","task_count":null,"acquisition_status":"Acquisition status not supplied","official_url":"https://github.com/google-deepmind/bbeh","indexing_mode":"index","profile":{"resources":[],"task_format":"","scoring":"","metric":"","size":"","answer_access":"","license":"","citation":"","maintainer":"","released":"","why_hard":"","related":[],"history":"","construction":"","limitations":"","saturation":"","scores":[]}},"task_id":"ea14394b-3f48-58ff-a890-a48bef595634","task_key":"bbeh~5fnycc--full--ea14394b-3f48-58ff-a890-a48bef595634","task_revision_id":"1","upstream_id":"","short_description":"The following is a description of a funny cartoon for the New Yorker Caption…","config":"bbeh_nycc","split":"full","body":"{\"input\":\"The following is a description of a funny cartoon for the New Yorker Caption Contest\\n\\n  Description: Two reptiles, one resembling a turtle and the other a snake, are facing each other in a jungle setting. The turtle has a snake-like tongue extended towards the snake.\\n\\n  Which of the following captions is the funniest?\\n(A) Oh,you found something gnu.\\n(B) Given the climate change flooding predictions, I went with the retractable sail option\\n(C) When I invited you over for dinner, I didn't expect you to eat the guest of honor.\\n(D) This too shall pass.\\n(E) Hey, did you eat the last water buffalo?\\n(F) Don't complain to me if you're up all night with indigestion\\n(G) I think you misheard me, dude, I said \\\"DON'T have a cow.\\n(H) You could have invited me to lunch.\\n(I) You are so full of it, Jimothy.\\n(J) What do you mean you didn't know it was my lunch? I wrote my name right on that boar.\\n\"}","display_format":"text","language":"","answer_status":"published","assets":[],"source_url":"https://github.com/google-deepmind/bbeh","history":"initial import","indexing_mode":"index","subproblems":[],"grids":[]}