{"kind":"task","effective_mode":"full","benchmark":{"kind":"benchmark","effective_mode":"full","slug":"bbeh","formal_name":"BIG-Bench Extra Hard (BBEH)","introduction":"BIG-Bench Extra Hard evaluates language-model reasoning through challenging task categories. It offers full and mini distributions; this import contains the full distribution: 4,520 questions across all 23 task categories.","introduction_ja":"","introduction_en":"","category":"Category not supplied","task_count":null,"acquisition_status":"Acquisition status not supplied","official_url":"https://github.com/google-deepmind/bbeh","indexing_mode":"index","profile":{"resources":[],"task_format":"","scoring":"","metric":"","size":"","answer_access":"","license":"","citation":"","maintainer":"","released":"","why_hard":"","related":[],"history":"","construction":"","limitations":"","saturation":"","scores":[]}},"task_id":"be27ecd5-3b60-528d-8073-5b9a943533a8","task_key":"bbeh~5fsarc~5ftriples--full--be27ecd5-3b60-528d-8073-5b9a943533a8","task_revision_id":"1","upstream_id":"","short_description":"Here are three (post, reply) pairs from Reddit. Your task is to decide whether…","config":"bbeh_sarc_triples","split":"full","body":"{\"input\":\"Here are three (post, reply) pairs from Reddit. Your task is to decide whether each reply is sarcastic. Specifically, label each pair with a \\\"0\\\" or \\\"1\\\", wherea \\\"1\\\" indicates that the reply is sarcastic, and a \\\"0\\\" indicates that the reply does not contain sarcasm, and provide your final answer as a comma-separated set of labels (e.g., \\\"1,0,0\\\" or \\\"0,0,0\\\").\\n\\nPOST 1: Dunno, seems like if you don't want to go to prison, then it's as simple as not being poor, a minority, or in the wrong place at the wrong time.\\nREPLY 1: Also, a lot of states have private companies that you have to pay so that they will take your mugshot off their website\\n\\nPOST 2: That's not Pahlavi's flag, there's no crown on it, that's THE PEOPLE'S FLAG... inclusive of all Iranians without regard to religion!\\nREPLY 2: well it can't get more \\\"persian\\\" than a wannabe persian raising a wannabe flag after a wannabe \\\"sport event\\\"...\\n\\nPOST 3: Haha too funny for this time in the morning, an open beta is FREE they wont charge anything but if you still dont feel assured just take your bank details off your account lol\\nREPLY 3: Yeah EA will see that you have played the Beta so they'll automatically withdraw 60$ from your bank account.\\n\\n\"}","display_format":"text","language":"","answer_status":"published","assets":[],"source_url":"https://github.com/google-deepmind/bbeh","history":"initial import","indexing_mode":"index","subproblems":[],"grids":[]}