{"kind":"task","effective_mode":"full","benchmark":{"kind":"benchmark","effective_mode":"full","slug":"bbeh","formal_name":"BIG-Bench Extra Hard (BBEH)","introduction":"BIG-Bench Extra Hard evaluates language-model reasoning through challenging task categories. It offers full and mini distributions; this import contains the full distribution: 4,520 questions across all 23 task categories.","introduction_ja":"","introduction_en":"","category":"Category not supplied","task_count":null,"acquisition_status":"Acquisition status not supplied","official_url":"https://github.com/google-deepmind/bbeh","indexing_mode":"index","profile":{"resources":[],"task_format":"","scoring":"","metric":"","size":"","answer_access":"","license":"","citation":"","maintainer":"","released":"","why_hard":"","related":[],"history":"","construction":"","limitations":"","saturation":"","scores":[]}},"task_id":"88ebd18b-cbe6-5798-b4c0-013a3b955989","task_key":"bbeh~5fweb~5fof~5flies--full--88ebd18b-cbe6-5798-b4c0-013a3b955989","task_revision_id":"1","upstream_id":"","short_description":"In this question, assume each person either always tells the truth or always…","config":"bbeh_web_of_lies","split":"full","body":"{\"input\":\"In this question, assume each person either always tells the truth or always lies. Margaret says McKenzie tells the truth. Elanor says Shenna tells the truth. Orson says Oriana lies. Oliver says Maybelle lies. Shenna says either exactly one of Kristian, Mateo and Opal tells the truth, or all three of them. Crista says either all three of Delfina, Mona and Morris tell the truth or only one of them. Kristian says either exactly two of Crista, Odessa and Matteo tell the truth or none of them. Murray says Melanie lies. Ka says exactly one of Andree, Miller and Maynard tell the truth. Inga says either all three of Audrie, Owen and Virginia tell the truth or only one of them. Delfina tells the truth. Omar says Natasha tells the truth. Mike says Melanie tells the truth. McKenzie says Margaret tells the truth. Matteo says Odessa tells the truth. Jaymie says either exactly one of Antwan, Maria and Marina tells the truth, or all three of them. John says Ofelia tells the truth. Odessa says Matteo tells the truth. Andree says exactly two of Rashida, Orson and Oriana tell the truth. Sima says either exactly one of Sherrie, John and Ofelia tells the truth, or all three of them. Opal says Mateo tells the truth. Rashida says only one of Jim, Mike and Murray tells the truth. Natasha says Omar tells the truth. Maynard says Miller lies. Antwan says either exactly one of Sima, McKenzie and Margaret tells the truth, or all three of them. Virginia says Owen tells the truth. Jim says Elanor tells the truth. Ryan says Ka lies. Miller says Maynard lies. Mateo says Opal tells the truth. Odell says Maybelle tells the truth. Morris says Mona tells the truth. Sherrie says Inga, Odell and Oliver all tell the truth. Marina says Maria tells the truth. Audrie says Ryan tells the truth. Mona says Morris tells the truth. Ofelia says John tells the truth. Oriana says Orson lies. Owen says Virginia tells the truth. Helene says either all three of Jaymie, Omar and Natasha lie, or two of them tell the truth. Maria says Marina tells the truth. Do Sima, Audrie, and Helene tell the truth? Your answer should be a list of three words separated by commas. Each of them must be either yes (if the person tells the truth), no (if the person lies), or unknown (if the person's truth cannot be determined). For example, <unknown, no, yes> or <yes, yes, yes>.\"}","display_format":"text","language":"","answer_status":"published","assets":[],"source_url":"https://github.com/google-deepmind/bbeh","history":"initial import","indexing_mode":"index","subproblems":[],"grids":[]}