{"kind":"task","effective_mode":"full","benchmark":{"kind":"benchmark","effective_mode":"full","slug":"bbeh","formal_name":"BIG-Bench Extra Hard (BBEH)","introduction":"BIG-Bench Extra Hard evaluates language-model reasoning through challenging task categories. It offers full and mini distributions; this import contains the full distribution: 4,520 questions across all 23 task categories.","introduction_ja":"","introduction_en":"","category":"Category not supplied","task_count":null,"acquisition_status":"Acquisition status not supplied","official_url":"https://github.com/google-deepmind/bbeh","indexing_mode":"index","profile":{"resources":[],"task_format":"","scoring":"","metric":"","size":"","answer_access":"","license":"","citation":"","maintainer":"","released":"","why_hard":"","related":[],"history":"","construction":"","limitations":"","saturation":"","scores":[]}},"task_id":"b4206819-061d-580a-bde5-e3ebd11edd1e","task_key":"bbeh~5fnycc--full--b4206819-061d-580a-bde5-e3ebd11edd1e","task_revision_id":"1","upstream_id":"","short_description":"The following is a description of a funny cartoon for the New Yorker Caption…","config":"bbeh_nycc","split":"full","body":"{\"input\":\"The following is a description of a funny cartoon for the New Yorker Caption Contest\\n\\n  Description: A man dressed in medieval attire is holding a crossbow and aiming at a child who has an apple on their head. The child is standing against a tree with a castle and mountain in the background.\\n\\n  Which of the following captions is the funniest?\\n(A) Let’s try the Diet Coke and Mentos challenge instead.\\n(B) Why did you tell mom that I am insured now?\\n(C) Daddy, if you claim you nailed it, I won't tell.\\n(D) No, seriously, add the whip cream first.\\n(E) Ok, Dad. You win. I'll pay for the new lederhosen out of my allowance.\\n(F) I'd suggest we use a PUMPKIN for this......\\n(G) Dad, please tell me this isn't some kind of Abraham Isaac thing.\\n(H) Couldn’t we go fishing instead?\\n(I) What we won't do just to get an overture!\\n(J) Either you need to get closer or I'm going to get a bigger apple.\\n\"}","display_format":"text","language":"","answer_status":"published","assets":[],"source_url":"https://github.com/google-deepmind/bbeh","history":"initial import","indexing_mode":"index","subproblems":[],"grids":[]}