{"kind":"task","effective_mode":"full","benchmark":{"kind":"benchmark","effective_mode":"full","slug":"bbeh","formal_name":"BIG-Bench Extra Hard (BBEH)","introduction":"BIG-Bench Extra Hard evaluates language-model reasoning through challenging task categories. It offers full and mini distributions; this import contains the full distribution: 4,520 questions across all 23 task categories.","introduction_ja":"","introduction_en":"","category":"Category not supplied","task_count":null,"acquisition_status":"Acquisition status not supplied","official_url":"https://github.com/google-deepmind/bbeh","indexing_mode":"index","profile":{"resources":[],"task_format":"","scoring":"","metric":"","size":"","answer_access":"","license":"","citation":"","maintainer":"","released":"","why_hard":"","related":[],"history":"","construction":"","limitations":"","saturation":"","scores":[]}},"task_id":"c1ac40e3-3146-5082-98cf-9b054d69f056","task_key":"bbeh~5fnycc--full--c1ac40e3-3146-5082-98cf-9b054d69f056","task_revision_id":"1","upstream_id":"","short_description":"The following is a description of a funny cartoon for the New Yorker Caption…","config":"bbeh_nycc","split":"full","body":"{\"input\":\"The following is a description of a funny cartoon for the New Yorker Caption Contest\\n\\n  Description: People are flying kites in a park. One of the kites is shaped like a dog.\\n\\n  Which of the following captions is the funniest?\\n(A) He is retrieving his fribee.\\n(B) Forget the plastic baggie, you're gonna need a net.\\n(C) Just so you know, this is an off-leash park.\\n(D) What? It was on his bucket list!\\n(E) Sir, I'm going to have to ask you to please curb your kite.\\n(F) Granted, she's got a tail and she's flying just like the others - but one drop of pee or poo and you're out of here.\\n(G) 'I don't care what he had for breakfast. That's not what we meant by learning to harness wind power.'\\n(H) She really should've done her business before she got up there.\\n(I) I SEE YOUR DOG HAS SOME FLATULENCE\\n(J) Skye terrier or Airedale?\\n\"}","display_format":"text","language":"","answer_status":"published","assets":[],"source_url":"https://github.com/google-deepmind/bbeh","history":"initial import","indexing_mode":"index","subproblems":[],"grids":[]}