{"kind":"task","effective_mode":"full","benchmark":{"kind":"benchmark","effective_mode":"full","slug":"bbeh","formal_name":"BIG-Bench Extra Hard (BBEH)","introduction":"BIG-Bench Extra Hard evaluates language-model reasoning through challenging task categories. It offers full and mini distributions; this import contains the full distribution: 4,520 questions across all 23 task categories.","introduction_ja":"","introduction_en":"","category":"Category not supplied","task_count":null,"acquisition_status":"Acquisition status not supplied","official_url":"https://github.com/google-deepmind/bbeh","indexing_mode":"index","profile":{"resources":[],"task_format":"","scoring":"","metric":"","size":"","answer_access":"","license":"","citation":"","maintainer":"","released":"","why_hard":"","related":[],"history":"","construction":"","limitations":"","saturation":"","scores":[]}},"task_id":"2d3a3f4d-b65e-58e8-8541-9a068c535b49","task_key":"bbeh~5fnycc--full--2d3a3f4d-b65e-58e8-8541-9a068c535b49","task_revision_id":"1","upstream_id":"","short_description":"The following is a description of a funny cartoon for the New Yorker Caption…","config":"bbeh_nycc","split":"full","body":"{\"input\":\"The following is a description of a funny cartoon for the New Yorker Caption Contest\\n\\n  Description: A man with large butterfly wings is hovering in the living room while another person sits on the couch looking at him in surprise.\\n\\n  Which of the following captions is the funniest?\\n(A) Ok...you're carrying this social butterfly thing a little too far.\\n(B) Go back in your coroom.\\n(C) Please stop drinking Red Bull\\n(D) Stop hovering over me.\\n(E) Gregor! Shoes off!\\n(F) Someone’s feeling social today!\\n(G) Is it me or my sweater you're eyeing so hungrily .\\n(H) I call this look 'the butterfly affect.'\\n(I) When you said you were going to \\\"reinvent yourself\\\",I imagined something less \\\"flighty\\\".\\n(J) Do you have to come in here every time I turn on the light?\\n\"}","display_format":"text","language":"","answer_status":"published","assets":[],"source_url":"https://github.com/google-deepmind/bbeh","history":"initial import","indexing_mode":"index","subproblems":[],"grids":[]}