{"kind":"task","effective_mode":"full","benchmark":{"kind":"benchmark","effective_mode":"full","slug":"bbeh","formal_name":"BIG-Bench Extra Hard (BBEH)","introduction":"BIG-Bench Extra Hard evaluates language-model reasoning through challenging task categories. It offers full and mini distributions; this import contains the full distribution: 4,520 questions across all 23 task categories.","introduction_ja":"","introduction_en":"","category":"Category not supplied","task_count":null,"acquisition_status":"Acquisition status not supplied","official_url":"https://github.com/google-deepmind/bbeh","indexing_mode":"index","profile":{"resources":[],"task_format":"","scoring":"","metric":"","size":"","answer_access":"","license":"","citation":"","maintainer":"","released":"","why_hard":"","related":[],"history":"","construction":"","limitations":"","saturation":"","scores":[]}},"task_id":"7dd3ad45-1637-5a95-a28e-15bfe7d33542","task_key":"bbeh~5fnycc--full--7dd3ad45-1637-5a95-a28e-15bfe7d33542","task_revision_id":"1","upstream_id":"","short_description":"The following is a description of a funny cartoon for the New Yorker Caption…","config":"bbeh_nycc","split":"full","body":"{\"input\":\"The following is a description of a funny cartoon for the New Yorker Caption Contest\\n\\n  Description: A woman is standing in a laundry room with a cat. A man is lying on an ironing board, and his head is being ironed.\\n\\n  Which of the following captions is the funniest?\\n(A) Who taught you to iron your own clothes?\\n(B) Not enough starch Ironman.\\n(C) When I said things would iron themselves out, this is not what I meant.\\n(D) It looks like you ran out of steam.\\n(E) I think I know why you're always feeling pressed for time.\\n(F) The ONE time he offered to help\\n(G) You call this training for the ironman?\\n(H) I told you to use the steamer!\\n(I) No, what I said was you need to learn to express yourself.\\n(J) Tough day at the office, dear?\\n\"}","display_format":"text","language":"","answer_status":"published","assets":[],"source_url":"https://github.com/google-deepmind/bbeh","history":"initial import","indexing_mode":"index","subproblems":[],"grids":[]}