{"kind":"task","effective_mode":"full","benchmark":{"kind":"benchmark","effective_mode":"full","slug":"bbeh","formal_name":"BIG-Bench Extra Hard (BBEH)","introduction":"BIG-Bench Extra Hard evaluates language-model reasoning through challenging task categories. It offers full and mini distributions; this import contains the full distribution: 4,520 questions across all 23 task categories.","introduction_ja":"","introduction_en":"","category":"Category not supplied","task_count":null,"acquisition_status":"Acquisition status not supplied","official_url":"https://github.com/google-deepmind/bbeh","indexing_mode":"index","profile":{"resources":[],"task_format":"","scoring":"","metric":"","size":"","answer_access":"","license":"","citation":"","maintainer":"","released":"","why_hard":"","related":[],"history":"","construction":"","limitations":"","saturation":"","scores":[]}},"task_id":"7a191914-53b8-5489-b101-d43e178bd563","task_key":"bbeh~5fweb~5fof~5flies--full--7a191914-53b8-5489-b101-d43e178bd563","task_revision_id":"1","upstream_id":"","short_description":"In this question, assume each person either always tells the truth or always…","config":"bbeh_web_of_lies","split":"full","body":"{\"input\":\"In this question, assume each person either always tells the truth or always lies. Octavian says Mavis tells the truth. Michael says Miriam lies. Osian says Martha tells the truth. Delfina says exactly two of Sima, Minnie and Mercedes tell the truth. Odile says Maxim lies. Disha says Oprah tells the truth. Orlando says Olive lies. Mateo says Martha lies. Andree says either all three of Delfina, Ollie and Moses tell the truth or only one of them. Shaunda says Rashida tells the truth. Minnie says Mercedes lies. Yoland says Gwenn tells the truth. Ka says either all three of Sherrie, Disha and Oprah tell the truth or only one of them. Moses says Ollie tells the truth. Rashida says exactly two of Inga, Odile and Maxim tell the truth. Raymond says Shaunda, Osian and Mateo all tell the truth. Inga tells the truth. Olive says Orlando lies. Willian says Raymond lies. Ollie says Moses tells the truth. Maxim says Odile lies. Maurice says Ocean tells the truth. Mavis says Octavian tells the truth. Alejandro says only one of Ka, Maurice and Oriana lies. Sherrie says either all three of Kristian, Mavis and Octavian lie, or two of them tell the truth. Mercedes says Minnie lies. Sima says only one of Alexis, Owen and Michael lies. Oprah says Disha tells the truth. Gwenn says exactly two of Alejandro, Orlando and Olive tell the truth. Alexis says Willian tells the truth. Owen says Miriam tells the truth. Kristian says Andree lies. Oriana says Ocean lies. Do Alejandro, Ka, and Yoland tell the truth? Your answer should be a list of three words separated by commas. Each of them must be either yes (if the person tells the truth), no (if the person lies), or unknown (if the person's truth cannot be determined). For example, <unknown, no, yes> or <yes, yes, yes>.\"}","display_format":"text","language":"","answer_status":"published","assets":[],"source_url":"https://github.com/google-deepmind/bbeh","history":"initial import","indexing_mode":"index","subproblems":[],"grids":[]}