{"kind":"task","effective_mode":"full","benchmark":{"kind":"benchmark","effective_mode":"full","slug":"bbeh","formal_name":"BIG-Bench Extra Hard (BBEH)","introduction":"BIG-Bench Extra Hard evaluates language-model reasoning through challenging task categories. It offers full and mini distributions; this import contains the full distribution: 4,520 questions across all 23 task categories.","introduction_ja":"","introduction_en":"","category":"Category not supplied","task_count":null,"acquisition_status":"Acquisition status not supplied","official_url":"https://github.com/google-deepmind/bbeh","indexing_mode":"index","profile":{"resources":[],"task_format":"","scoring":"","metric":"","size":"","answer_access":"","license":"","citation":"","maintainer":"","released":"","why_hard":"","related":[],"history":"","construction":"","limitations":"","saturation":"","scores":[]}},"task_id":"d5109aaa-705b-5c2b-bbb7-8607bcedfbf6","task_key":"bbeh~5fsarc~5ftriples--full--d5109aaa-705b-5c2b-bbb7-8607bcedfbf6","task_revision_id":"1","upstream_id":"","short_description":"Here are three (post, reply) pairs from Reddit. Your task is to decide whether…","config":"bbeh_sarc_triples","split":"full","body":"{\"input\":\"Here are three (post, reply) pairs from Reddit. Your task is to decide whether each reply is sarcastic. Specifically, label each pair with a \\\"0\\\" or \\\"1\\\", wherea \\\"1\\\" indicates that the reply is sarcastic, and a \\\"0\\\" indicates that the reply does not contain sarcasm, and provide your final answer as a comma-separated set of labels (e.g., \\\"1,0,0\\\" or \\\"0,0,0\\\").\\n\\nPOST 1: I don't know where the suggestion that vaccines cause autism comes from, my parents had me vaccinated and I turned out trains.\\nREPLY 1: HE'S JUST PLAYING 5D CHEST GUYS HE KNEW HE WAS GOING TO RUN FOR PRESIDENT SO HE ONLY MADE HIMSELF LOOK DUMB FOR THE MEDIA ATTENTION FOR HIS PRESIDENCY, 5D chest guys and you feel for it\\n\\nPOST 2: Dylann Roof, the white man accused of killing nine African-Americans in a Charleston church has been declared competent to stand trial\\nREPLY 2: 'The WHITE MAN who killed BLACK PEOPLE\\\" Why not just say \\\"A man who killed 9 people during bible study at a local church\\\" These details are much more significant\\n\\nPOST 3: Japanese stereotypes of European countries, based on Japanese Google autocomplete suggestions | By @nick_kapur on twitter\\nREPLY 3: Practically every Japanese person I've met has told me that British good sucks and yet fails to name a single dish aside from fish and chips.\\n\\n\"}","display_format":"text","language":"","answer_status":"published","assets":[],"source_url":"https://github.com/google-deepmind/bbeh","history":"initial import","indexing_mode":"index","subproblems":[],"grids":[]}