{"kind":"task","effective_mode":"full","benchmark":{"kind":"benchmark","effective_mode":"full","slug":"bbeh","formal_name":"BIG-Bench Extra Hard (BBEH)","introduction":"BIG-Bench Extra Hard evaluates language-model reasoning through challenging task categories. It offers full and mini distributions; this import contains the full distribution: 4,520 questions across all 23 task categories.","introduction_ja":"","introduction_en":"","category":"Category not supplied","task_count":null,"acquisition_status":"Acquisition status not supplied","official_url":"https://github.com/google-deepmind/bbeh","indexing_mode":"index","profile":{"resources":[],"task_format":"","scoring":"","metric":"","size":"","answer_access":"","license":"","citation":"","maintainer":"","released":"","why_hard":"","related":[],"history":"","construction":"","limitations":"","saturation":"","scores":[]}},"task_id":"7a89c435-9443-580b-ae23-69ecbc5e9da7","task_key":"bbeh~5fnycc--full--7a89c435-9443-580b-ae23-69ecbc5e9da7","task_revision_id":"1","upstream_id":"","short_description":"The following is a description of a funny cartoon for the New Yorker Caption…","config":"bbeh_nycc","split":"full","body":"{\"input\":\"The following is a description of a funny cartoon for the New Yorker Caption Contest\\n\\n  Description: Two hikers are climbing a mountain with walking sticks, while a businessman in a suit and tie, holding a briefcase and talking on a cellphone, follows them.\\n\\n  Which of the following captions is the funniest?\\n(A) Sorry boss, these pollsters are unlikely to predict anything.\\n(B) I'm what you may call an extremely remote worker.\\n(C) Because it would be false advertising if I didn't \\\"go the extra mile\\\" for them.\\n(D) It's healthier than chasing ambulances\\n(E) Didn't you see the sign? Beware of Personal Injury lawyers.\\n(F) I guess \\\"trail lawyer\\\" wasn't a typo after all.\\n(G) He insists on working remotely.\\n(H) You got my email away message so you called?\\n\\nYeah, I’m on vacation, hiking right now actually. \\n\\nNo it’s fine, I can take care of it.\\n(I) All they want is landslide insurance\\n(J) It's beautiful up here. Untouched. Pristine. Construction starts in a week.\\n\"}","display_format":"text","language":"","answer_status":"published","assets":[],"source_url":"https://github.com/google-deepmind/bbeh","history":"initial import","indexing_mode":"index","subproblems":[],"grids":[]}