{"kind":"task","effective_mode":"full","benchmark":{"kind":"benchmark","effective_mode":"full","slug":"bbeh","formal_name":"BIG-Bench Extra Hard (BBEH)","introduction":"BIG-Bench Extra Hard evaluates language-model reasoning through challenging task categories. It offers full and mini distributions; this import contains the full distribution: 4,520 questions across all 23 task categories.","introduction_ja":"","introduction_en":"","category":"Category not supplied","task_count":null,"acquisition_status":"Acquisition status not supplied","official_url":"https://github.com/google-deepmind/bbeh","indexing_mode":"index","profile":{"resources":[],"task_format":"","scoring":"","metric":"","size":"","answer_access":"","license":"","citation":"","maintainer":"","released":"","why_hard":"","related":[],"history":"","construction":"","limitations":"","saturation":"","scores":[]}},"task_id":"0289011f-50f3-5118-93c4-61cbff6f4641","task_key":"bbeh~5fsarc~5ftriples--full--0289011f-50f3-5118-93c4-61cbff6f4641","task_revision_id":"1","upstream_id":"","short_description":"Here are three (post, reply) pairs from Reddit. Your task is to decide whether…","config":"bbeh_sarc_triples","split":"full","body":"{\"input\":\"Here are three (post, reply) pairs from Reddit. Your task is to decide whether each reply is sarcastic. Specifically, label each pair with a \\\"0\\\" or \\\"1\\\", wherea \\\"1\\\" indicates that the reply is sarcastic, and a \\\"0\\\" indicates that the reply does not contain sarcasm, and provide your final answer as a comma-separated set of labels (e.g., \\\"1,0,0\\\" or \\\"0,0,0\\\").\\n\\nPOST 1: At least we know who the empty chair at President Obama's last State of the Union address will represent.\\nREPLY 1: Remind Chicago police how bad they are and tell them how to do third job cuz it's probably their fault again.\\n\\nPOST 2: The important thing is that it is up to the men to determine what's harmless to women and what's actually harassment, womens' own experiences can't be trusted to make that distinction.\\nREPLY 2: Yeah because being called certain words by strangers always has the same context and meaning as from someone you love and trust.\\n\\nPOST 3: It is either complete dishonesty or ignorance for FRAN TOWNSEND to make this argument.... laughable.\\nREPLY 3: A civil society can only exist when government officials can operate without accountability and in total secrecy, and when the press is free to protect government's interests and curtail public debate.\\n\\n\"}","display_format":"text","language":"","answer_status":"published","assets":[],"source_url":"https://github.com/google-deepmind/bbeh","history":"initial import","indexing_mode":"index","subproblems":[],"grids":[]}