{"kind":"task","effective_mode":"full","benchmark":{"kind":"benchmark","effective_mode":"full","slug":"bbeh","formal_name":"BIG-Bench Extra Hard (BBEH)","introduction":"BIG-Bench Extra Hard evaluates language-model reasoning through challenging task categories. It offers full and mini distributions; this import contains the full distribution: 4,520 questions across all 23 task categories.","introduction_ja":"","introduction_en":"","category":"Category not supplied","task_count":null,"acquisition_status":"Acquisition status not supplied","official_url":"https://github.com/google-deepmind/bbeh","indexing_mode":"index","profile":{"resources":[],"task_format":"","scoring":"","metric":"","size":"","answer_access":"","license":"","citation":"","maintainer":"","released":"","why_hard":"","related":[],"history":"","construction":"","limitations":"","saturation":"","scores":[]}},"task_id":"afa6ec57-dbea-5b53-8a41-59c1d44ae447","task_key":"bbeh~5fnycc--full--afa6ec57-dbea-5b53-8a41-59c1d44ae447","task_revision_id":"1","upstream_id":"","short_description":"The following is a description of a funny cartoon for the New Yorker Caption…","config":"bbeh_nycc","split":"full","body":"{\"input\":\"The following is a description of a funny cartoon for the New Yorker Caption Contest\\n\\n  Description: A boy is standing in a kitchen holding a magic lamp. A genie has appeared from the lamp and is floating in front of the boy.\\n\\n  Which of the following captions is the funniest?\\n(A) Well, you're the one who expects me to just pop up whenever you need me...\\n(B) Welcome to the internet of things.\\n(C) I moved. The rent is cheaper and it comes with free heat...\\n(D) 'What do you mean, \\\"where's the woman?\\\" Listen kid, I'm the Genii of Kitchen-Appliances... When you said, \\\"I want her, hot, black and in the palm of my hand.\\\", I assumed you meant Mrs. Coffee, over here'\\n(E) you have three wishes, as long as they're light, medium and dark..\\n(F) Sorry, I just do toast and bagels. The cappuccino genie is the next door down.\\n(G) It's your science fair project; figure it out.\\n(H) Even I can't make it cook evenly.\\n(I) I cover for the lamp genie on weekends.\\n(J) My union says I can't work both appliances.\\n\"}","display_format":"text","language":"","answer_status":"published","assets":[],"source_url":"https://github.com/google-deepmind/bbeh","history":"initial import","indexing_mode":"index","subproblems":[],"grids":[]}