{"kind":"task","effective_mode":"full","benchmark":{"kind":"benchmark","effective_mode":"full","slug":"bbeh","formal_name":"BIG-Bench Extra Hard (BBEH)","introduction":"BIG-Bench Extra Hard evaluates language-model reasoning through challenging task categories. It offers full and mini distributions; this import contains the full distribution: 4,520 questions across all 23 task categories.","introduction_ja":"","introduction_en":"","category":"Category not supplied","task_count":null,"acquisition_status":"Acquisition status not supplied","official_url":"https://github.com/google-deepmind/bbeh","indexing_mode":"index","profile":{"resources":[],"task_format":"","scoring":"","metric":"","size":"","answer_access":"","license":"","citation":"","maintainer":"","released":"","why_hard":"","related":[],"history":"","construction":"","limitations":"","saturation":"","scores":[]}},"task_id":"38ee02f4-3046-5725-9c3e-009bac385c2b","task_key":"bbeh~5fnycc--full--38ee02f4-3046-5725-9c3e-009bac385c2b","task_revision_id":"1","upstream_id":"","short_description":"The following is a description of a funny cartoon for the New Yorker Caption…","config":"bbeh_nycc","split":"full","body":"{\"input\":\"The following is a description of a funny cartoon for the New Yorker Caption Contest\\n\\n  Description: A man is sitting on a couch reading a book. A large dinosaur-like creature is peering at him from very close, almost filling the room.\\n\\n  Which of the following captions is the funniest?\\n(A) Yes, Mother?\\n(B) Just let me finish three more pages and I'll take you for a walk.\\n(C) I know the end is near, but I insist on starting from the beginning.\\n(D) No, I cannot take you back to the dentist! You ate him!\\n(E) Brontobibliophilesaurus\\n(F) Honey, do you remember when Ryan’s iguana got loose last summer?\\n(G) I just fed you.\\n(H) This is Beowulf. Grendel is still on the shelf.\\n(I) And after he eats the evil prince, the brave monster and the beautiful princess live happily ever after.\\n(J) Search continues for contaminated GM pet food.\\n\"}","display_format":"text","language":"","answer_status":"published","assets":[],"source_url":"https://github.com/google-deepmind/bbeh","history":"initial import","indexing_mode":"index","subproblems":[],"grids":[]}