{"kind":"task","effective_mode":"full","benchmark":{"kind":"benchmark","effective_mode":"full","slug":"bbeh","formal_name":"BIG-Bench Extra Hard (BBEH)","introduction":"BIG-Bench Extra Hard evaluates language-model reasoning through challenging task categories. It offers full and mini distributions; this import contains the full distribution: 4,520 questions across all 23 task categories.","introduction_ja":"","introduction_en":"","category":"Category not supplied","task_count":null,"acquisition_status":"Acquisition status not supplied","official_url":"https://github.com/google-deepmind/bbeh","indexing_mode":"index","profile":{"resources":[],"task_format":"","scoring":"","metric":"","size":"","answer_access":"","license":"","citation":"","maintainer":"","released":"","why_hard":"","related":[],"history":"","construction":"","limitations":"","saturation":"","scores":[]}},"task_id":"d3dfe494-86a6-5d5b-aa01-f91df862b5e8","task_key":"bbeh~5fnycc--full--d3dfe494-86a6-5d5b-aa01-f91df862b5e8","task_revision_id":"1","upstream_id":"","short_description":"The following is a description of a funny cartoon for the New Yorker Caption…","config":"bbeh_nycc","split":"full","body":"{\"input\":\"The following is a description of a funny cartoon for the New Yorker Caption Contest\\n\\n  Description: A person dressed as the Grim Reaper is standing in a doorway, holding a scythe. A woman is standing in front of them, looking unimpressed.\\n\\n  Which of the following captions is the funniest?\\n(A) Yeah well, you don't look like your Tinder profile picture, either.\\n(B) I think you're taking your job a bit too casually.\\n(C) I could understand the mix up. I can't tell you how frustrating it is to have your identity stolen.\\n(D) Hey, it's casual doomsday.\\n(E) He's out back.\\n(F) He's in the back.\\n(G) He is hiding in the bathroom.\\n(H) That hoodie could get you shot.\\n(I) Unfortunately, the man of the house is not in.\\n(J) This wasn't the sort of edgy role play I had in mind, Harold.\\n\"}","display_format":"text","language":"","answer_status":"published","assets":[],"source_url":"https://github.com/google-deepmind/bbeh","history":"initial import","indexing_mode":"index","subproblems":[],"grids":[]}