{"kind":"task","effective_mode":"full","benchmark":{"kind":"benchmark","effective_mode":"full","slug":"bbeh","formal_name":"BIG-Bench Extra Hard (BBEH)","introduction":"BIG-Bench Extra Hard evaluates language-model reasoning through challenging task categories. It offers full and mini distributions; this import contains the full distribution: 4,520 questions across all 23 task categories.","introduction_ja":"","introduction_en":"","category":"Category not supplied","task_count":null,"acquisition_status":"Acquisition status not supplied","official_url":"https://github.com/google-deepmind/bbeh","indexing_mode":"index","profile":{"resources":[],"task_format":"","scoring":"","metric":"","size":"","answer_access":"","license":"","citation":"","maintainer":"","released":"","why_hard":"","related":[],"history":"","construction":"","limitations":"","saturation":"","scores":[]}},"task_id":"14c0613b-2c23-5601-9f24-40cbbbf87729","task_key":"bbeh~5fweb~5fof~5flies--full--14c0613b-2c23-5601-9f24-40cbbbf87729","task_revision_id":"1","upstream_id":"","short_description":"In this question, assume each person either always tells the truth or always…","config":"bbeh_web_of_lies","split":"full","body":"{\"input\":\"In this question, assume each person either always tells the truth or always lies. Mustafa says Odile lies. Ozzie says McKenzie lies. Matteo says Olivia lies. Mariah says Minerva lies. Odessa says Oona tells the truth. Orhan says Octavia lies. Christie says Fletcher tells the truth. Owyn says Maxwell tells the truth. Dallas says exactly one of Sherrie, Orhan and Octavia tell the truth. Teressa says exactly two of Ryan, Otis and Disha tells the truth. Otis says Disha lies. Andree says Christie lies. McKenzie says Ozzie lies. Orla says Miller tells the truth. Othello says Olivia tells the truth. Kristian says only one of Kandi, Othello and Matteo tells the truth. Fletcher says Teressa lies. Mallory says Mason tells the truth. Odile says Mustafa lies. Elanor says Sal tells the truth. Miller says Orla tells the truth. Kandi says exactly one of Elanor, Odile and Mustafa tell the truth. Amberly tells the truth. Inga says either all three of Delbert, Orla and Miller tell the truth or only one of them. Ryan says Dallas tells the truth. Sal says either all three of Inga, Odessa and Oona tell the truth or only one of them. Alejandro says exactly two of Shaunda, Minerva and Mariah tells the truth. Mason says Mallory tells the truth. Ka says exactly two of Willian, McKenzie and Ozzie tells the truth. Maxwell says Owyn tells the truth. Oona says Odessa tells the truth. Minerva says Mariah lies. Shaunda says either exactly two of Kristian, Owyn and Maxwell tell the truth or none of them. Disha says Otis lies. Sherrie says Amberly tells the truth. Delbert says either all three of Andree, Mason and Mallory lie, or two of them tell the truth. Willian says Alejandro tells the truth. Octavia says Orhan lies. Do Willian, Kristian, and Ka tell the truth? Your answer should be a list of three words separated by commas. Each of them must be either yes (if the person tells the truth), no (if the person lies), or unknown (if the person's truth cannot be determined). For example, <unknown, no, yes> or <yes, yes, yes>.\"}","display_format":"text","language":"","answer_status":"published","assets":[],"source_url":"https://github.com/google-deepmind/bbeh","history":"initial import","indexing_mode":"index","subproblems":[],"grids":[]}