{"kind":"task","effective_mode":"full","benchmark":{"kind":"benchmark","effective_mode":"full","slug":"bbeh","formal_name":"BIG-Bench Extra Hard (BBEH)","introduction":"BIG-Bench Extra Hard evaluates language-model reasoning through challenging task categories. It offers full and mini distributions; this import contains the full distribution: 4,520 questions across all 23 task categories.","introduction_ja":"","introduction_en":"","category":"Category not supplied","task_count":null,"acquisition_status":"Acquisition status not supplied","official_url":"https://github.com/google-deepmind/bbeh","indexing_mode":"index","profile":{"resources":[],"task_format":"","scoring":"","metric":"","size":"","answer_access":"","license":"","citation":"","maintainer":"","released":"","why_hard":"","related":[],"history":"","construction":"","limitations":"","saturation":"","scores":[]}},"task_id":"406d46b8-ff21-5a60-aaad-08fd20b59012","task_key":"bbeh~5fnycc--full--406d46b8-ff21-5a60-aaad-08fd20b59012","task_revision_id":"1","upstream_id":"","short_description":"The following is a description of a funny cartoon for the New Yorker Caption…","config":"bbeh_nycc","split":"full","body":"{\"input\":\"The following is a description of a funny cartoon for the New Yorker Caption Contest\\n\\n  Description: A woman is lying in bed reading a book. A man is standing next to the bed with a drink in his hand, speaking into a microphone.\\n\\n  Which of the following captions is the funniest?\\n(A) I’ll be doing all next week on the couch.\\n(B) I would thank you for coming, but...\\n(C) Let me know if you've heard this one.\\n(D) Thank you. I'm your husband and I'll be here every night this week\\n(E) Honey, do I really have to introduce the next act?\\n(F) Well  ladies did you hear the one about the couple who had sex once a year?\\n(G) Great to be here at Bob and Bab's bedroom tonight! Hey, guys!  Anyone ever get married to your soulmate, only to find out that she suddenly finds you repulsive? Show of hands!\\n(H) I didn't complain when YOU started working from home.\\n(I) I've had a lot of requests, but I'm going to sing anyway.\\n(J) I’ll be here all week.\\n\"}","display_format":"text","language":"","answer_status":"published","assets":[],"source_url":"https://github.com/google-deepmind/bbeh","history":"initial import","indexing_mode":"index","subproblems":[],"grids":[]}