{"kind":"task","effective_mode":"full","benchmark":{"kind":"benchmark","effective_mode":"full","slug":"bbeh","formal_name":"BIG-Bench Extra Hard (BBEH)","introduction":"BIG-Bench Extra Hard evaluates language-model reasoning through challenging task categories. It offers full and mini distributions; this import contains the full distribution: 4,520 questions across all 23 task categories.","introduction_ja":"","introduction_en":"","category":"Category not supplied","task_count":null,"acquisition_status":"Acquisition status not supplied","official_url":"https://github.com/google-deepmind/bbeh","indexing_mode":"index","profile":{"resources":[],"task_format":"","scoring":"","metric":"","size":"","answer_access":"","license":"","citation":"","maintainer":"","released":"","why_hard":"","related":[],"history":"","construction":"","limitations":"","saturation":"","scores":[]}},"task_id":"8517c629-2652-5de0-9198-9517b8d79380","task_key":"bbeh~5fnycc--full--8517c629-2652-5de0-9198-9517b8d79380","task_revision_id":"1","upstream_id":"","short_description":"The following is a description of a funny cartoon for the New Yorker Caption…","config":"bbeh_nycc","split":"full","body":"{\"input\":\"The following is a description of a funny cartoon for the New Yorker Caption Contest\\n\\n  Description: A man is standing in a bookstore, and the cashier is behind the counter with a large alligator lying on top of the books.\\n\\n  Which of the following captions is the funniest?\\n(A) Try saying 'see ya later'. Trust me, it's funny.\\n(B) Don’t worry, she’s already eaten!\\n(C) It’s alligator hunting season. He figures this is the last place they’d look.\\n(D) Don't ask me why, he just has this thing about New Fiction. But you're welcome to try.\\n(E) He’s here to provide the tears of regret for the Trump reinstatement section.\\n(F) We take 'publish or perish' rather seriously.\\n(G) His name's Readers Digest.\\n(H) Fake news is on the other side of the store, this is all crock.\\n(I) The labor market is so tight, all the candidates we see are candidates from when Trump drained the swamp.\\n(J) Bibliophile crocodile! 🐊\\n\"}","display_format":"text","language":"","answer_status":"published","assets":[],"source_url":"https://github.com/google-deepmind/bbeh","history":"initial import","indexing_mode":"index","subproblems":[],"grids":[]}