{"kind":"task","effective_mode":"full","benchmark":{"kind":"benchmark","effective_mode":"full","slug":"bbeh","formal_name":"BIG-Bench Extra Hard (BBEH)","introduction":"BIG-Bench Extra Hard evaluates language-model reasoning through challenging task categories. It offers full and mini distributions; this import contains the full distribution: 4,520 questions across all 23 task categories.","introduction_ja":"","introduction_en":"","category":"Category not supplied","task_count":null,"acquisition_status":"Acquisition status not supplied","official_url":"https://github.com/google-deepmind/bbeh","indexing_mode":"index","profile":{"resources":[],"task_format":"","scoring":"","metric":"","size":"","answer_access":"","license":"","citation":"","maintainer":"","released":"","why_hard":"","related":[],"history":"","construction":"","limitations":"","saturation":"","scores":[]}},"task_id":"4326b151-a4d5-5082-a016-7223328dfc1f","task_key":"bbeh~5fweb~5fof~5flies--full--4326b151-a4d5-5082-a016-7223328dfc1f","task_revision_id":"1","upstream_id":"","short_description":"In this question, assume each person either always tells the truth or always…","config":"bbeh_web_of_lies","split":"full","body":"{\"input\":\"In this question, assume each person either always tells the truth or always lies. Delfina says either all three of Lorine, Mindy and Oona lie, or two of them tell the truth. Fidel says either all three of Sherrie, Owen and Murphy tell the truth or only one of them. Mona says Ozzie tells the truth. Mina says Ocean tells the truth. Vernell says only one of Fidel, Octavian and Mickey lies. Tamika tells the truth. Mary says Olga tells the truth. Delbert says either exactly two of Vernell, Maria and Miguel tell the truth or none of them. Maybelle says Oda tells the truth. Mariah says Marty lies. Owen says Murphy tells the truth. Oscar says Maurice lies. Odette says Misty tells the truth. Odell says Octavia tells the truth. Madison says Olga lies. Sherrie says Fletcher lies. Mindy says Oona tells the truth. Mehran says Matthew tells the truth. Malcolm says Ozzy tells the truth. Jim says Elanor lies. Alejandro says either all three of Kandi, Ozzy and Malcolm tell the truth or only one of them. Elanor says either all three of Ka, Odette and Misty tell the truth or only one of them. Octavia says Odell tells the truth. Misty says Odette tells the truth. Oswald says Oleg tells the truth. Oona says Mindy tells the truth. Obadiah says McKenzie tells the truth. McKenzie says Obadiah tells the truth. Bernita says either exactly two of Delfina, Octavia and Odell tell the truth or none of them. Ozzie says Mona tells the truth. Kandi says Delbert lies. Mark says Odin lies. Shalonda says only one of Tamika, Maybelle and Nicole lies. Willian says either all three of Helene, Ozzie and Mona lie, or two of them tell the truth. Ocean says Mina tells the truth. Ozzy says Malcolm tells the truth. Oran says Odin tells the truth. Nishanth says Maurice tells the truth. Murphy says Owen tells the truth. Jamey says Shaunda, Mary and Madison all tell the truth. Vina says either exactly one of Bernita, McKenzie and Obadiah tells the truth, or all three of them. Shaunda says either all three of Amberly, Oleg and Oswald lie, or two of them tell the truth. Lorine says either exactly one of Jim, Mina and Ocean tells the truth, or all three of them. Oleg says Oswald tells the truth. Otto says Matthew lies. Amberly says only one of Vina, Oran and Mark lies. Fletcher says Willian, Nishanth and Oscar all tell the truth. Mickey says Oliver lies. Ka says only one of Shalonda, Mehran and Otto lies. Helene says exactly two of Jamey, Mariah and Marty tells the truth. Marty says Mariah lies. Maria says Miguel tells the truth. Nicole says Oda lies. Miguel says Maria tells the truth. Octavian says Oliver tells the truth. Do Shaunda, Nishanth, and Alejandro tell the truth? Your answer should be a list of three words separated by commas. Each of them must be either yes (if the person tells the truth), no (if the person lies), or unknown (if the person's truth cannot be determined). For example, <unknown, no, yes> or <yes, yes, yes>.\"}","display_format":"text","language":"","answer_status":"published","assets":[],"source_url":"https://github.com/google-deepmind/bbeh","history":"initial import","indexing_mode":"index","subproblems":[],"grids":[]}