{"kind":"task","effective_mode":"full","benchmark":{"kind":"benchmark","effective_mode":"full","slug":"bbeh","formal_name":"BIG-Bench Extra Hard (BBEH)","introduction":"BIG-Bench Extra Hard evaluates language-model reasoning through challenging task categories. It offers full and mini distributions; this import contains the full distribution: 4,520 questions across all 23 task categories.","introduction_ja":"","introduction_en":"","category":"Category not supplied","task_count":null,"acquisition_status":"Acquisition status not supplied","official_url":"https://github.com/google-deepmind/bbeh","indexing_mode":"index","profile":{"resources":[],"task_format":"","scoring":"","metric":"","size":"","answer_access":"","license":"","citation":"","maintainer":"","released":"","why_hard":"","related":[],"history":"","construction":"","limitations":"","saturation":"","scores":[]}},"task_id":"92091051-5b2d-5a02-8efc-1a9a73900381","task_key":"bbeh~5fweb~5fof~5flies--full--92091051-5b2d-5a02-8efc-1a9a73900381","task_revision_id":"1","upstream_id":"","short_description":"In this question, assume each person either always tells the truth or always…","config":"bbeh_web_of_lies","split":"full","body":"{\"input\":\"In this question, assume each person either always tells the truth or always lies. Osian says Mike tells the truth. Minerva says Marvin tells the truth. Michaela says Marina lies. Olaf says Miguel lies. Miguel says Olaf lies. Delfina says exactly two of Lorine, Michaela and Marina tells the truth. Ollie says Mason tells the truth. Othello says Mark tells the truth. Mariah says Oswald tells the truth. Delbert says only one of Conception, Martin and Martha lies. Marvin says Minerva tells the truth. Sal says either all three of Andree, Othello and Mark lie, or two of them tell the truth. Vernell says Tamika lies. Mariana says Maria tells the truth. Maya says Mason lies. Moses says Osmond tells the truth. Audrie says only one of Inga, Osian and Malcolm lies. Osborne says Olympia tells the truth. Shalonda says Fidel, Marty and Marcus all tell the truth. Fidel says exactly one of Kristian, Monica and Murphy tells the truth.Malcolm says Mike lies. Marina says Michaela lies. Odessa says Odette tells the truth. Sherrie says Vernell lies. Mehran says Maria lies. Martin says Miles tells the truth. Amberly says Dallas lies. Christie says exactly two of Phoebe, Miguel and Olaf tell the truth. Crista says either exactly two of Delbert, Odette and Odessa tell the truth or none of them. Kandi tells the truth. Andree says only one of Ryan, Moses and Melody lies. Mackenzie says Oriana lies. Uri says Matteo tells the truth. Obadiah says Oriana tells the truth. Ozzy says Mandy tells the truth. Mercedes says Orson lies. Alexis says only one of Shenna, Obadiah and Mackenzie lies. Inga says Kandi, Mariana and Mehran all tell the truth. Willian says either all three of Delfina, Uri and Matteo lie, or two of them tell the truth. Antwan says either all three of Sal, Oswald and Mariah tell the truth or only one of them. Vina says exactly two of Alexis, Megan and Milton tells the truth. Nishanth says Orhan lies. Kristian says only one of Crista, Orly and Nishanth lies. Melody says Osmond lies. Mark says Othello tells the truth. Phoebe says either all three of Antwan, Mandy and Ozzy tell the truth or only one of them. Conception says exactly one of Vina, Maurice and Osvaldo tells the truth.Matteo says Uri tells the truth. Marcus says Muriel lies. Oswald says Mariah tells the truth. Ryan says either exactly two of Sherrie, Minerva and Marvin tell the truth or none of them. Odette says Odessa tells the truth. Marty says Muriel tells the truth. Tamika says only one of Yoland, Ollie and Maya tells the truth. Dallas says Audrie tells the truth. Milton says Megan lies. Orly says Orhan tells the truth. Sima says exactly two of Amberly, Mercedes and Orson tell the truth. Mandy says Ozzy tells the truth. Monica says Murphy lies. Orlando says Oscar tells the truth. Mina says Ottoline tells the truth. Ottoline says Mina tells the truth. Murphy says Monica lies. Shenna says either exactly two of Christie, Ottoline and Mina tell the truth or none of them. Lorine says either exactly one of Shalonda, Orlando and Oscar tells the truth, or all three of them. Oscar says Orlando tells the truth. Yoland says Sima, Osborne and Murray all tell the truth. Maurice says Osvaldo lies. Murray says Olympia lies. Martha says Miles lies. Orson says Mercedes lies. Osvaldo says Maurice lies. Megan says Milton lies. Do Sal, Matteo, and Willian tell the truth? Your answer should be a list of three words separated by commas. Each of them must be either yes (if the person tells the truth), no (if the person lies), or unknown (if the person's truth cannot be determined). For example, <unknown, no, yes> or <yes, yes, yes>.\"}","display_format":"text","language":"","answer_status":"published","assets":[],"source_url":"https://github.com/google-deepmind/bbeh","history":"initial import","indexing_mode":"index","subproblems":[],"grids":[]}