{"kind":"task","effective_mode":"full","benchmark":{"kind":"benchmark","effective_mode":"full","slug":"bbeh","formal_name":"BIG-Bench Extra Hard (BBEH)","introduction":"BIG-Bench Extra Hard evaluates language-model reasoning through challenging task categories. It offers full and mini distributions; this import contains the full distribution: 4,520 questions across all 23 task categories.","introduction_ja":"","introduction_en":"","category":"Category not supplied","task_count":null,"acquisition_status":"Acquisition status not supplied","official_url":"https://github.com/google-deepmind/bbeh","indexing_mode":"index","profile":{"resources":[],"task_format":"","scoring":"","metric":"","size":"","answer_access":"","license":"","citation":"","maintainer":"","released":"","why_hard":"","related":[],"history":"","construction":"","limitations":"","saturation":"","scores":[]}},"task_id":"f749e222-b632-5507-a55e-fd45f4b25127","task_key":"bbeh~5fnycc--full--f749e222-b632-5507-a55e-fd45f4b25127","task_revision_id":"1","upstream_id":"","short_description":"The following is a description of a funny cartoon for the New Yorker Caption…","config":"bbeh_nycc","split":"full","body":"{\"input\":\"The following is a description of a funny cartoon for the New Yorker Caption Contest\\n\\n  Description: A man is standing in his yard with a rake, looking at a UFO that is beaming up a pile of leaves.\\n\\n  Which of the following captions is the funniest?\\n(A) Luckily Trump can’t close the border to all aliens\\n(B) You weren’t kidding Urkh, down here it does grow on trees.\\n(C) This must be their currency. I heard him say they were raking it in.\\n(D) Jack could not accept the apparent evidence that leaf removal was the alien's ultimate objective\\n(E) It costs extra because it's 'Earth Weed.' Trust me, rich Martian kids always fall for it every time.\\n(F) Leaf him alone, Scotty\\n(G) Next, I'll show you how the Tractor Beam 3000 mulches.\\n(H) I liked it better when we did anal probes.\\n(I) It's menial work, but when you're undocumented, jobs are hard to come by.\\n(J) After the year they’ve had, it’s the least we can do.\\n\"}","display_format":"text","language":"","answer_status":"published","assets":[],"source_url":"https://github.com/google-deepmind/bbeh","history":"initial import","indexing_mode":"index","subproblems":[],"grids":[]}