{"kind":"task","effective_mode":"full","benchmark":{"kind":"benchmark","effective_mode":"full","slug":"bbeh","formal_name":"BIG-Bench Extra Hard (BBEH)","introduction":"BIG-Bench Extra Hard evaluates language-model reasoning through challenging task categories. It offers full and mini distributions; this import contains the full distribution: 4,520 questions across all 23 task categories.","introduction_ja":"","introduction_en":"","category":"Category not supplied","task_count":null,"acquisition_status":"Acquisition status not supplied","official_url":"https://github.com/google-deepmind/bbeh","indexing_mode":"index","profile":{"resources":[],"task_format":"","scoring":"","metric":"","size":"","answer_access":"","license":"","citation":"","maintainer":"","released":"","why_hard":"","related":[],"history":"","construction":"","limitations":"","saturation":"","scores":[]}},"task_id":"f93a5d6e-ca9b-5ded-80d5-7fdcdc2b8940","task_key":"bbeh~5fsarc~5ftriples--full--f93a5d6e-ca9b-5ded-80d5-7fdcdc2b8940","task_revision_id":"1","upstream_id":"","short_description":"Here are three (post, reply) pairs from Reddit. Your task is to decide whether…","config":"bbeh_sarc_triples","split":"full","body":"{\"input\":\"Here are three (post, reply) pairs from Reddit. Your task is to decide whether each reply is sarcastic. Specifically, label each pair with a \\\"0\\\" or \\\"1\\\", wherea \\\"1\\\" indicates that the reply is sarcastic, and a \\\"0\\\" indicates that the reply does not contain sarcasm, and provide your final answer as a comma-separated set of labels (e.g., \\\"1,0,0\\\" or \\\"0,0,0\\\").\\n\\nPOST 1: Lol but the cure for cancer is being hidden by the government so we can spend more money to buy into this scheme.\\nREPLY 1: Yeah, that's not how knowledge works, that's ridiculous, knowledge is what is we already known, if we don't know \\\"the cure for cancer\\\" then it's because we haven't discovered, not because we don't have the password.\\n\\nPOST 2: TIL when The state of Virginia sent a bronze statue of George Washington to be displayed outside The National Gallery in London, they also sent a ton of earth to respect Washington's vow that he would never set foot on English soil.\\nREPLY 2: I would tell the Commonwealth of Virginia to eat a dick, we're going to weld a monocle and teacup to it.\\n\\nPOST 3: Yes I loved the complete lack of depth perception which made avoiding the raid-wiping mechanics incredibly easy.\\nREPLY 3: Not only visually but lore wise also, it's a huge point in azeroth history, we, mortals, defeated a envoy of the titans that was set to destroy the world, it's like slaping a angel in the face and say, it's our world now, say fuck off to your boss\\n\\n\"}","display_format":"text","language":"","answer_status":"published","assets":[],"source_url":"https://github.com/google-deepmind/bbeh","history":"initial import","indexing_mode":"index","subproblems":[],"grids":[]}