{"kind":"task","effective_mode":"full","benchmark":{"kind":"benchmark","effective_mode":"full","slug":"bbeh","formal_name":"BIG-Bench Extra Hard (BBEH)","introduction":"BIG-Bench Extra Hard evaluates language-model reasoning through challenging task categories. It offers full and mini distributions; this import contains the full distribution: 4,520 questions across all 23 task categories.","introduction_ja":"","introduction_en":"","category":"Category not supplied","task_count":null,"acquisition_status":"Acquisition status not supplied","official_url":"https://github.com/google-deepmind/bbeh","indexing_mode":"index","profile":{"resources":[],"task_format":"","scoring":"","metric":"","size":"","answer_access":"","license":"","citation":"","maintainer":"","released":"","why_hard":"","related":[],"history":"","construction":"","limitations":"","saturation":"","scores":[]}},"task_id":"7afcbdc1-034a-5a1e-b8b2-3f5960c5fd95","task_key":"bbeh~5fnycc--full--7afcbdc1-034a-5a1e-b8b2-3f5960c5fd95","task_revision_id":"1","upstream_id":"","short_description":"The following is a description of a funny cartoon for the New Yorker Caption…","config":"bbeh_nycc","split":"full","body":"{\"input\":\"The following is a description of a funny cartoon for the New Yorker Caption Contest\\n\\n  Description: The Earth, personified with arms and a face, is sitting at a bar with a drink in front of it. A man is standing next to the Earth, looking at it.\\n\\n  Which of the following captions is the funniest?\\n(A) Even when I'm sober, it feels like I'm spinning.\\n(B) Kids! They never listen!\\n(C) You don’t know the half of it.\\n(D) There was room for ice.\\n(E) One small drink for me and a large one for the end  of mankind!\\n(F) And then I dreamed they were crawling all over me and buzzing around me and...it was just awful.\\n(G) I feel like I’m spinning.\\n(H) Do you know how hard it is to find signs of intelligent life around here?\\n(I) It’s mostly water weight.\\n(J) Call me old fashioned but I prefer it with ice.\\n\"}","display_format":"text","language":"","answer_status":"published","assets":[],"source_url":"https://github.com/google-deepmind/bbeh","history":"initial import","indexing_mode":"index","subproblems":[],"grids":[]}