{"kind":"task","effective_mode":"full","benchmark":{"kind":"benchmark","effective_mode":"full","slug":"bbeh","formal_name":"BIG-Bench Extra Hard (BBEH)","introduction":"BIG-Bench Extra Hard evaluates language-model reasoning through challenging task categories. It offers full and mini distributions; this import contains the full distribution: 4,520 questions across all 23 task categories.","introduction_ja":"","introduction_en":"","category":"Category not supplied","task_count":null,"acquisition_status":"Acquisition status not supplied","official_url":"https://github.com/google-deepmind/bbeh","indexing_mode":"index","profile":{"resources":[],"task_format":"","scoring":"","metric":"","size":"","answer_access":"","license":"","citation":"","maintainer":"","released":"","why_hard":"","related":[],"history":"","construction":"","limitations":"","saturation":"","scores":[]}},"task_id":"8be9cc63-7592-5578-b3cd-c478978fc153","task_key":"bbeh~5fsarc~5ftriples--full--8be9cc63-7592-5578-b3cd-c478978fc153","task_revision_id":"1","upstream_id":"","short_description":"Here are three (post, reply) pairs from Reddit. Your task is to decide whether…","config":"bbeh_sarc_triples","split":"full","body":"{\"input\":\"Here are three (post, reply) pairs from Reddit. Your task is to decide whether each reply is sarcastic. Specifically, label each pair with a \\\"0\\\" or \\\"1\\\", wherea \\\"1\\\" indicates that the reply is sarcastic, and a \\\"0\\\" indicates that the reply does not contain sarcasm, and provide your final answer as a comma-separated set of labels (e.g., \\\"1,0,0\\\" or \\\"0,0,0\\\").\\n\\nPOST 1: Martin Shkreli paid $2 million for Wu-Tang Clan's new album and didn't even listen to the whole thing first\\nREPLY 1: Next headline will announce he buys babies for his Sunday roasts, but doesn't even save the leftovers.\\n\\nPOST 2: Remember boys: when a woman tells you to stop touching her clothes and smelling her hair because you're making her deeply scared of coming to work, she's actually just playing hard to get, and you just need to shake up your tactics a bit to make her putty in your arms.\\nREPLY 2: Ivanka was just a prop used by the Trump campaign to make white women more comfortable voting for a pussy grabber.\\n\\nPOST 3: Of course, it's different when the shooter is muslim, because they haven't assimilated yet, and, therefore, are not one of us.\\nREPLY 3: \\\"circlejerk\\\" A lot of of mass shooters want their name to go out in glory and that makes more potential murderers want to masscre so their name goes out in glory Its not a circlejerk\\n\\n\"}","display_format":"text","language":"","answer_status":"published","assets":[],"source_url":"https://github.com/google-deepmind/bbeh","history":"initial import","indexing_mode":"index","subproblems":[],"grids":[]}