{"kind":"task","effective_mode":"full","benchmark":{"kind":"benchmark","effective_mode":"full","slug":"bbeh","formal_name":"BIG-Bench Extra Hard (BBEH)","introduction":"BIG-Bench Extra Hard evaluates language-model reasoning through challenging task categories. It offers full and mini distributions; this import contains the full distribution: 4,520 questions across all 23 task categories.","introduction_ja":"","introduction_en":"","category":"Category not supplied","task_count":null,"acquisition_status":"Acquisition status not supplied","official_url":"https://github.com/google-deepmind/bbeh","indexing_mode":"index","profile":{"resources":[],"task_format":"","scoring":"","metric":"","size":"","answer_access":"","license":"","citation":"","maintainer":"","released":"","why_hard":"","related":[],"history":"","construction":"","limitations":"","saturation":"","scores":[]}},"task_id":"a6035888-80d9-52e0-bb1f-4f83d5445d2b","task_key":"bbeh~5fnycc--full--a6035888-80d9-52e0-bb1f-4f83d5445d2b","task_revision_id":"1","upstream_id":"","short_description":"The following is a description of a funny cartoon for the New Yorker Caption…","config":"bbeh_nycc","split":"full","body":"{\"input\":\"The following is a description of a funny cartoon for the New Yorker Caption Contest\\n\\n  Description: An alien is sitting on a chair next to a child's bed, reading a book to the child. The child looks surprised and a bit scared.\\n\\n  Which of the following captions is the funniest?\\n(A) Zargle blobben blorbongorgle\\n(B) Actually, you're the result of a close encounter of the first kind.\\n(C) And the humans were never heard from again.\\n(D) And they were abducted happily ever after.\\n(E) I think we both know how this one ends, don't we, Melania?\\n(F) Do you want the alien ending or the sappy earthling ending?\\n(G) Well, you won't find good babysitters just around the corner!\\n(H) once upon a time there lived humans\\n(I) Goodnight Moon” is fine, but “War of the Worlds” is better\\n(J) Again, no I'm not looking under the bed.\\n\"}","display_format":"text","language":"","answer_status":"published","assets":[],"source_url":"https://github.com/google-deepmind/bbeh","history":"initial import","indexing_mode":"index","subproblems":[],"grids":[]}