{"kind":"task","effective_mode":"full","benchmark":{"kind":"benchmark","effective_mode":"full","slug":"bbeh","formal_name":"BIG-Bench Extra Hard (BBEH)","introduction":"BIG-Bench Extra Hard evaluates language-model reasoning through challenging task categories. It offers full and mini distributions; this import contains the full distribution: 4,520 questions across all 23 task categories.","introduction_ja":"","introduction_en":"","category":"Category not supplied","task_count":null,"acquisition_status":"Acquisition status not supplied","official_url":"https://github.com/google-deepmind/bbeh","indexing_mode":"index","profile":{"resources":[],"task_format":"","scoring":"","metric":"","size":"","answer_access":"","license":"","citation":"","maintainer":"","released":"","why_hard":"","related":[],"history":"","construction":"","limitations":"","saturation":"","scores":[]}},"task_id":"7069e3e7-ebf4-55f2-97bc-664a6a08a5fa","task_key":"bbeh~5fsarc~5ftriples--full--7069e3e7-ebf4-55f2-97bc-664a6a08a5fa","task_revision_id":"1","upstream_id":"","short_description":"Here are three (post, reply) pairs from Reddit. Your task is to decide whether…","config":"bbeh_sarc_triples","split":"full","body":"{\"input\":\"Here are three (post, reply) pairs from Reddit. Your task is to decide whether each reply is sarcastic. Specifically, label each pair with a \\\"0\\\" or \\\"1\\\", wherea \\\"1\\\" indicates that the reply is sarcastic, and a \\\"0\\\" indicates that the reply does not contain sarcasm, and provide your final answer as a comma-separated set of labels (e.g., \\\"1,0,0\\\" or \\\"0,0,0\\\").\\n\\nPOST 1: He did a lot of good things at Oregon I never understood why people thought it wouldn't translate to the NFL , I would like to see him gain a bit of weight though\\nREPLY 1: But he'll never be a good pocket passer coming from a spread offense that had all his receivers wide open.\\n\\nPOST 2: Plot twist, \\\"Elon Musk Creates Sinkhole at UC Berkley, 131 protestors dead, further injuries unknown\\\" \\\"Sorry guys I work for Trump daddy now\\\" Elon Musk\\nREPLY 2: If this thing eventually delivers high speed transit across LA it will be one of the greatest things in history\\n\\nPOST 3: The Federal government needs to stop passing these expensive regulations which put an exorbitant burden on these honest corporations which will always have the best interest of their employees at stake.\\nREPLY 3: Just have the rookies pay for them as part of their hazing instead of making them pay for $50,000 dinners.\\n\\n\"}","display_format":"text","language":"","answer_status":"published","assets":[],"source_url":"https://github.com/google-deepmind/bbeh","history":"initial import","indexing_mode":"index","subproblems":[],"grids":[]}