{"kind":"task","effective_mode":"full","benchmark":{"kind":"benchmark","effective_mode":"full","slug":"bbeh","formal_name":"BIG-Bench Extra Hard (BBEH)","introduction":"BIG-Bench Extra Hard evaluates language-model reasoning through challenging task categories. It offers full and mini distributions; this import contains the full distribution: 4,520 questions across all 23 task categories.","introduction_ja":"","introduction_en":"","category":"Category not supplied","task_count":null,"acquisition_status":"Acquisition status not supplied","official_url":"https://github.com/google-deepmind/bbeh","indexing_mode":"index","profile":{"resources":[],"task_format":"","scoring":"","metric":"","size":"","answer_access":"","license":"","citation":"","maintainer":"","released":"","why_hard":"","related":[],"history":"","construction":"","limitations":"","saturation":"","scores":[]}},"task_id":"9e4f1008-f2e5-5129-8565-c7a64e5cbd63","task_key":"bbeh~5fnycc--full--9e4f1008-f2e5-5129-8565-c7a64e5cbd63","task_revision_id":"1","upstream_id":"","short_description":"The following is a description of a funny cartoon for the New Yorker Caption…","config":"bbeh_nycc","split":"full","body":"{\"input\":\"The following is a description of a funny cartoon for the New Yorker Caption Contest\\n\\n  Description: A man is standing outside an elevator, looking at another man who is inside the elevator with an elephant. The elevator doors are open, and the elephant is taking up most of the space inside the elevator.\\n\\n  Which of the following captions is the funniest?\\n(A) Sorry! Gotta deliver these to an awkward conversation on the fifth floor.\\n(B) Yes we are your upstairs neighbors- how did you know?\\n(C) I would get the next one. Someone has farted in this one.\\n(D) At least it's no longer in the room . . .\\n(E) Turns out, the weight limits in these things are more of a suggestion.\\n(F) Turns out, some Amazon Prime Day deals should not be made after a large glass of wine.\\n(G) Hey, Chuck, how’s it goin?\\n(H) Is that what you’re wearing tonight?\\n(I) Well, I was reading Babar to the kids, and....\\n(J) Yeah, I know - but he thinks there are lions on the stairs.\\n\"}","display_format":"text","language":"","answer_status":"published","assets":[],"source_url":"https://github.com/google-deepmind/bbeh","history":"initial import","indexing_mode":"index","subproblems":[],"grids":[]}