{"kind":"task","effective_mode":"full","benchmark":{"kind":"benchmark","effective_mode":"full","slug":"bbeh","formal_name":"BIG-Bench Extra Hard (BBEH)","introduction":"BIG-Bench Extra Hard evaluates language-model reasoning through challenging task categories. It offers full and mini distributions; this import contains the full distribution: 4,520 questions across all 23 task categories.","introduction_ja":"","introduction_en":"","category":"Category not supplied","task_count":null,"acquisition_status":"Acquisition status not supplied","official_url":"https://github.com/google-deepmind/bbeh","indexing_mode":"index","profile":{"resources":[],"task_format":"","scoring":"","metric":"","size":"","answer_access":"","license":"","citation":"","maintainer":"","released":"","why_hard":"","related":[],"history":"","construction":"","limitations":"","saturation":"","scores":[]}},"task_id":"759a12d9-914f-5203-9037-8e8ff30052ee","task_key":"bbeh~5fweb~5fof~5flies--full--759a12d9-914f-5203-9037-8e8ff30052ee","task_revision_id":"1","upstream_id":"","short_description":"In this question, assume each person either always tells the truth or always…","config":"bbeh_web_of_lies","split":"full","body":"{\"input\":\"In this question, assume each person either always tells the truth or always lies. Shalonda says exactly one of Bernita, Maximilian and Oda tell the truth. Lorine says only one of Alejandro, Murphy and Marsha tells the truth. Mickey says Olympia tells the truth. Mia says Matteo lies. Vernell says Leda tells the truth. Marsha says Ozzie lies. Conception says exactly one of Sima, Meadow and Virginia tells the truth.Merlin says Oakley tells the truth. Murphy says Ozzie tells the truth. Onyx says Oscar lies. Fidel says exactly two of Shenna, Mia and Matteo tell the truth. Owyn says Michael tells the truth. Jaymie says either all three of Antwan, Michael and Owyn tell the truth or only one of them. Antwan says exactly one of Willian, Oprah and Mary tells the truth.Meadow says Virginia lies. Yoland lies. Muriel says Miriam tells the truth. Matteo says Mia lies. Raymond says Shalonda tells the truth. Sima says Raymond tells the truth. Michelle says Osvaldo lies. Rashida says Lorine tells the truth. Crista says exactly one of Gwenn, Mandy and Olga tells the truth.Dallas says either all three of Phoebe, Mickey and Olympia tell the truth or only one of them. Margaret says Osian tells the truth. Oakleigh says Melvin lies. Shaunda says Ka, Mariah and Michelle all tell the truth. Osian says Margaret tells the truth. Bernita says Jamey lies. Olympia says Mickey tells the truth. Ka says Rashida tells the truth. Willian says Dallas lies. Mary says Oprah lies. Phoebe says either all three of Vernell, Margaret and Osian tell the truth or only one of them. Maximilian says Oda lies. Vina says Audrie lies. Mariah says Osvaldo tells the truth. Maya says Mariana tells the truth. Leda says either all three of Crista, Merlin and Oakley tell the truth or only one of them. Audrie says only one of Shaunda, Muriel and Mallory lies. Olga says Mandy lies. Oda says Maximilian lies. Amberly says only one of Jaymie, Disha and Oakleigh lies. Virginia says Meadow lies. Alejandro says either exactly one of Yoland, Mariana and Maya tells the truth, or all three of them. Disha says Melvin tells the truth. Oprah says Mary lies. Oscar says Onyx lies. Mandy says Olga lies. Mariana says Maya tells the truth. Owen says Martha lies. Gwenn says exactly one of Fidel, Onyx and Oscar tell the truth. Shenna says Conception tells the truth. Oakley says Merlin tells the truth. Michael says Owyn tells the truth. Mallory says Miriam lies. Martha says Owen lies. Jamey says exactly one of Vina, Martha and Owen tell the truth. Do Leda, Murphy, and Amberly tell the truth? Your answer should be a list of three words separated by commas. Each of them must be either yes (if the person tells the truth), no (if the person lies), or unknown (if the person's truth cannot be determined). For example, <unknown, no, yes> or <yes, yes, yes>.\"}","display_format":"text","language":"","answer_status":"published","assets":[],"source_url":"https://github.com/google-deepmind/bbeh","history":"initial import","indexing_mode":"index","subproblems":[],"grids":[]}