{"kind":"task","effective_mode":"full","benchmark":{"kind":"benchmark","effective_mode":"full","slug":"bbeh","formal_name":"BIG-Bench Extra Hard (BBEH)","introduction":"BIG-Bench Extra Hard evaluates language-model reasoning through challenging task categories. It offers full and mini distributions; this import contains the full distribution: 4,520 questions across all 23 task categories.","introduction_ja":"","introduction_en":"","category":"Category not supplied","task_count":null,"acquisition_status":"Acquisition status not supplied","official_url":"https://github.com/google-deepmind/bbeh","indexing_mode":"index","profile":{"resources":[],"task_format":"","scoring":"","metric":"","size":"","answer_access":"","license":"","citation":"","maintainer":"","released":"","why_hard":"","related":[],"history":"","construction":"","limitations":"","saturation":"","scores":[]}},"task_id":"717d03b7-bd97-5993-b81c-34c541136d12","task_key":"bbeh~5fsarc~5ftriples--full--717d03b7-bd97-5993-b81c-34c541136d12","task_revision_id":"1","upstream_id":"","short_description":"Here are three (post, reply) pairs from Reddit. Your task is to decide whether…","config":"bbeh_sarc_triples","split":"full","body":"{\"input\":\"Here are three (post, reply) pairs from Reddit. Your task is to decide whether each reply is sarcastic. Specifically, label each pair with a \\\"0\\\" or \\\"1\\\", wherea \\\"1\\\" indicates that the reply is sarcastic, and a \\\"0\\\" indicates that the reply does not contain sarcasm, and provide your final answer as a comma-separated set of labels (e.g., \\\"1,0,0\\\" or \\\"0,0,0\\\").\\n\\nPOST 1: I'm sure Durst enjoyed the series, considering he was an introverted teenager with serious confidence issues and a longing for acceptance from his father.\\nREPLY 1: I haven't gone \\\"wat\\\" like this since I saw the page image for Cat Girl at TV Tropes (which is about a dozen kinds of wrong, but at least it isn't TEH REI).\\n\\nPOST 2: OH WOOOOOOOOOOOWWWWWWWW NICE TOP QUALITY JOURNALISM YOU GOTS THERE VICE, IT'S TOTALLY NOT SHIT, WWWWWOOOOOOOOOOOWWWWWWWWW.\\nREPLY 2: I can't help thinking that perhaps you're approaching the bottom of the barrel if you're reduced to outrage that not enough women smoke weed because there aren't enough female potheads to be role models for them.\\n\\nPOST 3: Oh look the same 3 guys we see now all acting the same way HAHA SO GOOD but they only got better so now they all suck.\\nREPLY 3: Based on the WWE Network microphone, I can only assume this is either the Extreme Rules or Payback pre-show.\\n\\n\"}","display_format":"text","language":"","answer_status":"published","assets":[],"source_url":"https://github.com/google-deepmind/bbeh","history":"initial import","indexing_mode":"index","subproblems":[],"grids":[]}