{"kind":"task","effective_mode":"full","benchmark":{"kind":"benchmark","effective_mode":"full","slug":"bbeh","formal_name":"BIG-Bench Extra Hard (BBEH)","introduction":"BIG-Bench Extra Hard evaluates language-model reasoning through challenging task categories. It offers full and mini distributions; this import contains the full distribution: 4,520 questions across all 23 task categories.","introduction_ja":"","introduction_en":"","category":"Category not supplied","task_count":null,"acquisition_status":"Acquisition status not supplied","official_url":"https://github.com/google-deepmind/bbeh","indexing_mode":"index","profile":{"resources":[],"task_format":"","scoring":"","metric":"","size":"","answer_access":"","license":"","citation":"","maintainer":"","released":"","why_hard":"","related":[],"history":"","construction":"","limitations":"","saturation":"","scores":[]}},"task_id":"66543cfc-6644-5b71-b451-b8b3539e9aa9","task_key":"bbeh~5fsarc~5ftriples--full--66543cfc-6644-5b71-b451-b8b3539e9aa9","task_revision_id":"1","upstream_id":"","short_description":"Here are three (post, reply) pairs from Reddit. Your task is to decide whether…","config":"bbeh_sarc_triples","split":"full","body":"{\"input\":\"Here are three (post, reply) pairs from Reddit. Your task is to decide whether each reply is sarcastic. Specifically, label each pair with a \\\"0\\\" or \\\"1\\\", wherea \\\"1\\\" indicates that the reply is sarcastic, and a \\\"0\\\" indicates that the reply does not contain sarcasm, and provide your final answer as a comma-separated set of labels (e.g., \\\"1,0,0\\\" or \\\"0,0,0\\\").\\n\\nPOST 1: Create Your Own Kekkei Genkai I thought it'd be fun to have a thread where people can create their own clan/blood-line trait, just to see how creative the sub can really get lol Here is what it should include: -Clan Name: (could be your name or a made up name) -Clane Village: (what village the clan is affiliated with, could be made up) -Clan Symbol: (the symbol of the clan, like the Uchiha fan) -Kekkei Genkai: (what their special ability of the clan is, or something they excel at like binding spells with the Uzumaki) Here is mine: Clan Name: Tsuri Clan Village: Village Hidden In Frost Clan Symbol: A Snow Flake (simple design not too intricate) Clan Ability: Ice/Wood Release: An ability similar to wood release but not as strong, the user can create a forest of ice in the same manor as the 1st and control its shape by changing the ice from water and back)\\nREPLY 1: in real life im really in clan , my nation is all based on a clan system , im Somali, country is fcked to the ground because all these clans are always fighting each other , civil war shit hole\\n\\nPOST 2: Tony Romo is about to become the ONLY QB in NFL history to have 6 games in a season w/ a 135.0+ QB rating.\\nREPLY 2: For anyone wondering, Week 4 vs NO (137.4) Week 7 vs NYG (135.7) Week 10 vs JAX (138.8) Week 12 vs NYG (143.4 wow) Week 14 vs CHI (138.0) And now today.\\n\\nPOST 3: A doctor risks being deregistered because he allegedly refused a referral for an Indian couple who wanted to abort a healthy unborn baby girl at 19 weeks, simply because they wanted a boy.\\nREPLY 3: Had an abortion because it wasn't the right sex, my god, they simply shouldn't be allowed to have kids!\\n\\n\"}","display_format":"text","language":"","answer_status":"published","assets":[],"source_url":"https://github.com/google-deepmind/bbeh","history":"initial import","indexing_mode":"index","subproblems":[],"grids":[]}