{"kind":"task","effective_mode":"full","benchmark":{"kind":"benchmark","effective_mode":"full","slug":"bbeh","formal_name":"BIG-Bench Extra Hard (BBEH)","introduction":"BIG-Bench Extra Hard evaluates language-model reasoning through challenging task categories. It offers full and mini distributions; this import contains the full distribution: 4,520 questions across all 23 task categories.","introduction_ja":"","introduction_en":"","category":"Category not supplied","task_count":null,"acquisition_status":"Acquisition status not supplied","official_url":"https://github.com/google-deepmind/bbeh","indexing_mode":"index","profile":{"resources":[],"task_format":"","scoring":"","metric":"","size":"","answer_access":"","license":"","citation":"","maintainer":"","released":"","why_hard":"","related":[],"history":"","construction":"","limitations":"","saturation":"","scores":[]}},"task_id":"ef618587-7c93-5f93-8a0f-687febdb2faa","task_key":"bbeh~5fweb~5fof~5flies--full--ef618587-7c93-5f93-8a0f-687febdb2faa","task_revision_id":"1","upstream_id":"","short_description":"In this question, assume each person either always tells the truth or always…","config":"bbeh_web_of_lies","split":"full","body":"{\"input\":\"In this question, assume each person either always tells the truth or always lies. Kristian says only one of Audrie, Mark and Max tells the truth. Mark says Merlin tells the truth. Conception says only one of Antwan, Nishanth and Ovid tells the truth. Miranda says Maximilian tells the truth. Mason says Oakley lies. Rashida says Shaunda, Orlando and Osvaldo all tell the truth. Christie says Crista, Marlon and Odette all tell the truth. Orly says Meredith lies. Osvaldo says Natasha lies. Marty says Mary tells the truth. Jim says exactly two of Kristian, Oona and Olive tell the truth. Willian says Jamey lies. Mary says Marty tells the truth. Mavis says Disha tells the truth. Ka says only one of Elanor, Margaret and Murray tells the truth. McKenzie says Otis tells the truth. Ovid says Lalit lies. Delfina says only one of Jaymie, Olivia and Otto lies. Meredith says Orly lies. Olive says Oona lies. Nishanth says Lalit tells the truth. Jaymie says Fletcher lies. Shaunda says only one of Delfina, Mavis and Orson tells the truth. Margaret says Mariana tells the truth. Murray says Mariana lies. Otis says McKenzie tells the truth. Delbert says either all three of Vina, Mary and Marty lie, or two of them tell the truth. Elanor says either exactly one of Christie, Michelle and Orin tells the truth, or all three of them. Odette says Marina lies. Audrie says exactly one of Ka, Orly and Meredith tell the truth. Odile says Odell tells the truth. Marlon says Marina tells the truth. Odell says Odile tells the truth. Tamika says Rashida tells the truth. Owen says Maya tells the truth. Shenna says either exactly two of Gwenn, Odell and Odile tell the truth or none of them. Crista says only one of Kandi, Miranda and Morris tells the truth. Kandi says Willian lies. Odessa says Oakley tells the truth. Phoebe says Delbert lies. Michelle says Orin tells the truth. Morris says Maximilian lies. Oona says Olive lies. Vina lies. Orson says Disha lies. Maya says Owen tells the truth. Otto says Maynard lies. Antwan says either exactly two of Jim, Otis and McKenzie tell the truth or none of them. Jamey says only one of Shenna, Mona and Osborn lies. Orin says Michelle tells the truth. Orlando says Natasha tells the truth. Olivia says Maynard tells the truth. Gwenn says only one of Phoebe, Odessa and Mason tells the truth. Max says Merlin lies. Mona says Milo tells the truth. Fletcher says either all three of Conception, Maya and Owen tell the truth or only one of them. Osborn says Milo lies. Do Elanor, Margaret, and Tamika tell the truth? Your answer should be a list of three words separated by commas. Each of them must be either yes (if the person tells the truth), no (if the person lies), or unknown (if the person's truth cannot be determined). For example, <unknown, no, yes> or <yes, yes, yes>.\"}","display_format":"text","language":"","answer_status":"published","assets":[],"source_url":"https://github.com/google-deepmind/bbeh","history":"initial import","indexing_mode":"index","subproblems":[],"grids":[]}