{"kind":"task","effective_mode":"full","benchmark":{"kind":"benchmark","effective_mode":"full","slug":"bbeh","formal_name":"BIG-Bench Extra Hard (BBEH)","introduction":"BIG-Bench Extra Hard evaluates language-model reasoning through challenging task categories. It offers full and mini distributions; this import contains the full distribution: 4,520 questions across all 23 task categories.","introduction_ja":"","introduction_en":"","category":"Category not supplied","task_count":null,"acquisition_status":"Acquisition status not supplied","official_url":"https://github.com/google-deepmind/bbeh","indexing_mode":"index","profile":{"resources":[],"task_format":"","scoring":"","metric":"","size":"","answer_access":"","license":"","citation":"","maintainer":"","released":"","why_hard":"","related":[],"history":"","construction":"","limitations":"","saturation":"","scores":[]}},"task_id":"86e8960b-aa25-519b-a9c5-439c2f114f2e","task_key":"bbeh~5fnycc--full--86e8960b-aa25-519b-a9c5-439c2f114f2e","task_revision_id":"1","upstream_id":"","short_description":"The following is a description of a funny cartoon for the New Yorker Caption…","config":"bbeh_nycc","split":"full","body":"{\"input\":\"The following is a description of a funny cartoon for the New Yorker Caption Contest\\n\\n  Description: A man is climbing a giant beanstalk that reaches up into the clouds. In the clouds, there is a business meeting taking place with people seated around a table and one person presenting a graph.\\n\\n  Which of the following captions is the funniest?\\n(A) Jack meets the corporate giant.\\n(B) When businesses moved everything into the cloud,Jack realised his mother had been right about the beans\\n(C) I'm sorry Jack but this meeting is about growing our \\\"stock\\\".\\n(D) Giant? Next floor.\\n(E) Excuse me, but this is a private Board Meeting in the cloud.\\n(F) Hey, everybody. I want you to meet the guy who made it all possible, our chief bean counter, Jack.\\n(G) I know he sees us, but let's sit tight and see if he can get himself down.\\n(H) We’re cloud analysts. The tech giants are further up.\\n(I) I reserved this conference room weeks ago.\\n(J) Uh hi... Is this the bean counters meeting?\\n\"}","display_format":"text","language":"","answer_status":"published","assets":[],"source_url":"https://github.com/google-deepmind/bbeh","history":"initial import","indexing_mode":"index","subproblems":[],"grids":[]}