{"kind":"task","effective_mode":"full","benchmark":{"kind":"benchmark","effective_mode":"full","slug":"bbeh","formal_name":"BIG-Bench Extra Hard (BBEH)","introduction":"BIG-Bench Extra Hard evaluates language-model reasoning through challenging task categories. It offers full and mini distributions; this import contains the full distribution: 4,520 questions across all 23 task categories.","introduction_ja":"","introduction_en":"","category":"Category not supplied","task_count":null,"acquisition_status":"Acquisition status not supplied","official_url":"https://github.com/google-deepmind/bbeh","indexing_mode":"index","profile":{"resources":[],"task_format":"","scoring":"","metric":"","size":"","answer_access":"","license":"","citation":"","maintainer":"","released":"","why_hard":"","related":[],"history":"","construction":"","limitations":"","saturation":"","scores":[]}},"task_id":"d157e6fe-ff5c-5825-9104-3186c58de8c6","task_key":"bbeh~5fsarc~5ftriples--full--d157e6fe-ff5c-5825-9104-3186c58de8c6","task_revision_id":"1","upstream_id":"","short_description":"Here are three (post, reply) pairs from Reddit. Your task is to decide whether…","config":"bbeh_sarc_triples","split":"full","body":"{\"input\":\"Here are three (post, reply) pairs from Reddit. Your task is to decide whether each reply is sarcastic. Specifically, label each pair with a \\\"0\\\" or \\\"1\\\", wherea \\\"1\\\" indicates that the reply is sarcastic, and a \\\"0\\\" indicates that the reply does not contain sarcasm, and provide your final answer as a comma-separated set of labels (e.g., \\\"1,0,0\\\" or \\\"0,0,0\\\").\\n\\nPOST 1: Expect the release of the \\\"De Lima Does Dallas\\\" in the next few days c/o the current Justice Secretary.\\nREPLY 1: De Lima \\\"stop letting people murder each other in the streets\\\" Duterte \\\"stop sleeping with your driver skank lol\\\" Are crazy pills not considered a drug anymore?\\n\\nPOST 2: I'm A Bernie Sanders Supporter Who Won't Back Hillary Clinton & Calling Me Names Won't Change My Mind\\nREPLY 2: I've gone from being mostly indifferent to Hillary's inevitability, to actively rooting for her downfall, purely due to the smug shittiness of her most vocal supporters.\\n\\nPOST 3: yeah, I'm 10lbs underweight so That said, I've not made any SFG AAs, simply because I don't really know any.\\nREPLY 3: You don't seem to understand the women's only purpose in life is to be skinny and attractive, and if they're not skinny, they aren't fulfilling the only thing we care about.\\n\\n\"}","display_format":"text","language":"","answer_status":"published","assets":[],"source_url":"https://github.com/google-deepmind/bbeh","history":"initial import","indexing_mode":"index","subproblems":[],"grids":[]}