{"kind":"task","effective_mode":"full","benchmark":{"kind":"benchmark","effective_mode":"full","slug":"bbeh","formal_name":"BIG-Bench Extra Hard (BBEH)","introduction":"BIG-Bench Extra Hard evaluates language-model reasoning through challenging task categories. It offers full and mini distributions; this import contains the full distribution: 4,520 questions across all 23 task categories.","introduction_ja":"","introduction_en":"","category":"Category not supplied","task_count":null,"acquisition_status":"Acquisition status not supplied","official_url":"https://github.com/google-deepmind/bbeh","indexing_mode":"index","profile":{"resources":[],"task_format":"","scoring":"","metric":"","size":"","answer_access":"","license":"","citation":"","maintainer":"","released":"","why_hard":"","related":[],"history":"","construction":"","limitations":"","saturation":"","scores":[]}},"task_id":"4bbd028b-2550-524b-ac92-0deee4502b05","task_key":"bbeh~5fweb~5fof~5flies--full--4bbd028b-2550-524b-ac92-0deee4502b05","task_revision_id":"1","upstream_id":"","short_description":"In this question, assume each person either always tells the truth or always…","config":"bbeh_web_of_lies","split":"full","body":"{\"input\":\"In this question, assume each person either always tells the truth or always lies. Oriana says McKenzie lies. Muriel says Mallory lies. McKenzie says Oriana lies. Phoebe says exactly one of Shenna, Murphy and Odin tells the truth.Conception says exactly two of Tamika, Oriana and McKenzie tells the truth. Martin says Osian tells the truth. Miriam says Ozzie tells the truth. Willian says Bernita lies. Inga says exactly two of Willian, Mallory and Muriel tells the truth. Jamey says Sima tells the truth. Ottoline says Osian lies. Fletcher says Phoebe tells the truth. Crista lies. Bernita says Lorine tells the truth. Amberly says either exactly one of Vina, Melvin and Oona tells the truth, or all three of them. Sima says only one of Conception, Martin and Ottoline tells the truth. Oona says Melvin tells the truth. Maya says Ozzy tells the truth. Tamika says either exactly two of Antwan, Ozzie and Miriam tell the truth or none of them. Kristian says Jamey tells the truth. Odin says Murphy lies. Melvin says Oona tells the truth. Lorine says either all three of Kristian, Maya and Ozzy tell the truth or only one of them. Vina says Inga tells the truth. Ozzie says Miriam tells the truth. Antwan says Fletcher tells the truth. Ozzy says Maya tells the truth. Mallory says Muriel lies. Jerry says Crista tells the truth. Murphy says Odin lies. Shenna says Jerry tells the truth. Do Lorine, Martin, and Amberly tell the truth? Your answer should be a list of three words separated by commas. Each of them must be either yes (if the person tells the truth), no (if the person lies), or unknown (if the person's truth cannot be determined). For example, <unknown, no, yes> or <yes, yes, yes>.\"}","display_format":"text","language":"","answer_status":"published","assets":[],"source_url":"https://github.com/google-deepmind/bbeh","history":"initial import","indexing_mode":"index","subproblems":[],"grids":[]}