{"kind":"task","effective_mode":"full","benchmark":{"kind":"benchmark","effective_mode":"full","slug":"bbeh","formal_name":"BIG-Bench Extra Hard (BBEH)","introduction":"BIG-Bench Extra Hard evaluates language-model reasoning through challenging task categories. It offers full and mini distributions; this import contains the full distribution: 4,520 questions across all 23 task categories.","introduction_ja":"","introduction_en":"","category":"Category not supplied","task_count":null,"acquisition_status":"Acquisition status not supplied","official_url":"https://github.com/google-deepmind/bbeh","indexing_mode":"index","profile":{"resources":[],"task_format":"","scoring":"","metric":"","size":"","answer_access":"","license":"","citation":"","maintainer":"","released":"","why_hard":"","related":[],"history":"","construction":"","limitations":"","saturation":"","scores":[]}},"task_id":"0638b8d6-238a-5485-8c30-d7d8fbbff889","task_key":"bbeh~5fnycc--full--0638b8d6-238a-5485-8c30-d7d8fbbff889","task_revision_id":"1","upstream_id":"","short_description":"The following is a description of a funny cartoon for the New Yorker Caption…","config":"bbeh_nycc","split":"full","body":"{\"input\":\"The following is a description of a funny cartoon for the New Yorker Caption Contest\\n\\n  Description: A man and a woman are sitting in water under a beach umbrella. They are wearing sunglasses and hats, and the man is shirtless. The water is up to their chests, and they appear to be sitting on chairs.\\n\\n  Which of the following captions is the funniest?\\n(A) Still want to vote Republican, Ed?\\n(B) I know you wanted to stay until hell or high water, but....\\n(C) Well, Owen, the sand castle is gone.\\n(D) I voted for Gore.\\n(E) It's OK.  Trump said it was fake tide.\\n(F) I believe in global warming, I just didn't expect it to come to the Cape.\\n(G) I just feel like we're drifting apart.\\n(H) So... this is how you got crabs?\\n(I) I swear he said the Ark was on its way...\\n(J) Well, at least the kids are quiet.\\n\"}","display_format":"text","language":"","answer_status":"published","assets":[],"source_url":"https://github.com/google-deepmind/bbeh","history":"initial import","indexing_mode":"index","subproblems":[],"grids":[]}