{"kind":"task","effective_mode":"full","benchmark":{"kind":"benchmark","effective_mode":"full","slug":"bbeh","formal_name":"BIG-Bench Extra Hard (BBEH)","introduction":"BIG-Bench Extra Hard evaluates language-model reasoning through challenging task categories. It offers full and mini distributions; this import contains the full distribution: 4,520 questions across all 23 task categories.","introduction_ja":"","introduction_en":"","category":"Category not supplied","task_count":null,"acquisition_status":"Acquisition status not supplied","official_url":"https://github.com/google-deepmind/bbeh","indexing_mode":"index","profile":{"resources":[],"task_format":"","scoring":"","metric":"","size":"","answer_access":"","license":"","citation":"","maintainer":"","released":"","why_hard":"","related":[],"history":"","construction":"","limitations":"","saturation":"","scores":[]}},"task_id":"43f3e685-afa3-5459-8220-9feb2ffb4ba3","task_key":"bbeh~5fnycc--full--43f3e685-afa3-5459-8220-9feb2ffb4ba3","task_revision_id":"1","upstream_id":"","short_description":"The following is a description of a funny cartoon for the New Yorker Caption…","config":"bbeh_nycc","split":"full","body":"{\"input\":\"The following is a description of a funny cartoon for the New Yorker Caption Contest\\n\\n  Description: A couple is sitting in their living room, one person is using a laptop while the other is knitting. Outside the window, a large group of snowmen is gathered, looking inside.\\n\\n  Which of the following captions is the funniest?\\n(A) I'm crowd-sourcing the New Year's Eve party invitations.\\n(B) I guess we're snowed in.\\n(C) I'll get the flamethrower over-nighted.\\n(D) Do I really have to explain to you why you can't let them in to warm up?\\n(E) Tell them Elsa and Anna have moved and they're trespassing.\\n(F) Honey, did you see this ridiculous news article about Zombie Snow-Men terrorizing the area... silly... must be fake news.\\n(G) They want us to reduce our carbon footprint.\\n(H) Honey I found it. A group of snowmen is called a squall.\\n(I) Honey I think I see Waldo.\\n(J) Abominable aren't they?\\n\"}","display_format":"text","language":"","answer_status":"published","assets":[],"source_url":"https://github.com/google-deepmind/bbeh","history":"initial import","indexing_mode":"index","subproblems":[],"grids":[]}