{"kind":"task","effective_mode":"full","benchmark":{"kind":"benchmark","effective_mode":"full","slug":"bbeh","formal_name":"BIG-Bench Extra Hard (BBEH)","introduction":"BIG-Bench Extra Hard evaluates language-model reasoning through challenging task categories. It offers full and mini distributions; this import contains the full distribution: 4,520 questions across all 23 task categories.","introduction_ja":"","introduction_en":"","category":"Category not supplied","task_count":null,"acquisition_status":"Acquisition status not supplied","official_url":"https://github.com/google-deepmind/bbeh","indexing_mode":"index","profile":{"resources":[],"task_format":"","scoring":"","metric":"","size":"","answer_access":"","license":"","citation":"","maintainer":"","released":"","why_hard":"","related":[],"history":"","construction":"","limitations":"","saturation":"","scores":[]}},"task_id":"482647ae-2e69-55b1-a9fc-9ca9813422d0","task_key":"bbeh~5fnycc--full--482647ae-2e69-55b1-a9fc-9ca9813422d0","task_revision_id":"1","upstream_id":"","short_description":"The following is a description of a funny cartoon for the New Yorker Caption…","config":"bbeh_nycc","split":"full","body":"{\"input\":\"The following is a description of a funny cartoon for the New Yorker Caption Contest\\n\\n  Description: A doctor is standing in an examination room, smiling and talking to a wine bottle with a face and arms.\\n\\n  Which of the following captions is the funniest?\\n(A) Okay now, I’m just going to need you to breathe.\\n(B) So, your date of birth is 1945?\\n(C) We need to talk about your blood pressure\\n(D) I hear you are feeling stopped up.\\n(E) I promise.  You will feel better after a few quick turns.\\n(F) You're at risk of being caught in a downward spiral.  Trust me.\\n(G) You'll feel awful. But I'll feel amazing.\\n(H) Wait,  I already had my colonoscopy.\\n(I) First the good news...you've aged well.\\n(J) Look, you're 50 years old, it really is time to get screened.\\n\"}","display_format":"text","language":"","answer_status":"published","assets":[],"source_url":"https://github.com/google-deepmind/bbeh","history":"initial import","indexing_mode":"index","subproblems":[],"grids":[]}