{"kind":"task","effective_mode":"full","benchmark":{"kind":"benchmark","effective_mode":"full","slug":"bbeh","formal_name":"BIG-Bench Extra Hard (BBEH)","introduction":"BIG-Bench Extra Hard evaluates language-model reasoning through challenging task categories. It offers full and mini distributions; this import contains the full distribution: 4,520 questions across all 23 task categories.","introduction_ja":"","introduction_en":"","category":"Category not supplied","task_count":null,"acquisition_status":"Acquisition status not supplied","official_url":"https://github.com/google-deepmind/bbeh","indexing_mode":"index","profile":{"resources":[],"task_format":"","scoring":"","metric":"","size":"","answer_access":"","license":"","citation":"","maintainer":"","released":"","why_hard":"","related":[],"history":"","construction":"","limitations":"","saturation":"","scores":[]}},"task_id":"ee80a487-1e1a-5770-866f-f37a3b967e7d","task_key":"bbeh~5fnycc--full--ee80a487-1e1a-5770-866f-f37a3b967e7d","task_revision_id":"1","upstream_id":"","short_description":"The following is a description of a funny cartoon for the New Yorker Caption…","config":"bbeh_nycc","split":"full","body":"{\"input\":\"The following is a description of a funny cartoon for the New Yorker Caption Contest\\n\\n  Description: A magician is performing a trick on a balcony, pulling a rabbit out of a hat. People in the surrounding building windows are watching the performance, while a woman sitting at a table nearby is looking at her phone.\\n\\n  Which of the following captions is the funniest?\\n(A) The rabbit’s Go Fund Me site is now up to five heads of lettuce and thirteen carrots!\\n(B) Sawing me in half better not be his next trick.\\n(C) No new likes, but your views are rising.\\n(D) Now it says to saw the rabbit in half ... wait that can't be right.\\n(E) Thank God she didn't order the veal!\\n(F) They’re not impressed. They say Mr. Renault two balconies below us is juggling three penguins and a brick of Gorgonzola.\\n(G) Yo have 25 ooh's, 32 ahh's and 47 How the hell did he do that's\\n(H) Imitating the Fed.\\n(I) The real magic trick is how he got her to say yes to a rooftop dinner.\\n(J) Nope.  They're all super disappointed.  It autocorrected where I said you'd pull the rabbit from.\\n\"}","display_format":"text","language":"","answer_status":"published","assets":[],"source_url":"https://github.com/google-deepmind/bbeh","history":"initial import","indexing_mode":"index","subproblems":[],"grids":[]}