{"kind":"task","effective_mode":"full","benchmark":{"kind":"benchmark","effective_mode":"full","slug":"bbeh","formal_name":"BIG-Bench Extra Hard (BBEH)","introduction":"BIG-Bench Extra Hard evaluates language-model reasoning through challenging task categories. It offers full and mini distributions; this import contains the full distribution: 4,520 questions across all 23 task categories.","introduction_ja":"","introduction_en":"","category":"Category not supplied","task_count":null,"acquisition_status":"Acquisition status not supplied","official_url":"https://github.com/google-deepmind/bbeh","indexing_mode":"index","profile":{"resources":[],"task_format":"","scoring":"","metric":"","size":"","answer_access":"","license":"","citation":"","maintainer":"","released":"","why_hard":"","related":[],"history":"","construction":"","limitations":"","saturation":"","scores":[]}},"task_id":"0f1d7805-0e9c-5d22-b76d-695c7cd0fd81","task_key":"bbeh~5fnycc--full--0f1d7805-0e9c-5d22-b76d-695c7cd0fd81","task_revision_id":"1","upstream_id":"","short_description":"The following is a description of a funny cartoon for the New Yorker Caption…","config":"bbeh_nycc","split":"full","body":"{\"input\":\"The following is a description of a funny cartoon for the New Yorker Caption Contest\\n\\n  Description: A man in a trench coat and hat is standing in an alley, showing another person a selection of kittens hidden inside his coat. The person looks surprised or intrigued.\\n\\n  Which of the following captions is the funniest?\\n(A) Spayed or neutered and all their shots. Trust me.\\n(B) They'll guard you from creepy men wearing trench coats in dark alleys.\\n(C) Ok, dogs are expensive, need walking, and want you to love them to death. Pick a cat.\\n(D) C'mon, man, they're so cute!  You KNOW you want one.  And the first one's free!\\n(E) And of course, you’ll be wanting the extended warranty.\\n(F) When I said “wait till you see these puppies,” what did you think I meant?\\n(G) Your follower numbers will go through the roof with these bad boys!\\n(H) These are genuine Rolex watch dogs.\\n(I) Sorry, I must have grabbed my wife’s coat by mistake.\\n(J) No animals were harmed in making this fur lining.\\n\"}","display_format":"text","language":"","answer_status":"published","assets":[],"source_url":"https://github.com/google-deepmind/bbeh","history":"initial import","indexing_mode":"index","subproblems":[],"grids":[]}