{"kind":"task","effective_mode":"full","benchmark":{"kind":"benchmark","effective_mode":"full","slug":"bbeh","formal_name":"BIG-Bench Extra Hard (BBEH)","introduction":"BIG-Bench Extra Hard evaluates language-model reasoning through challenging task categories. It offers full and mini distributions; this import contains the full distribution: 4,520 questions across all 23 task categories.","introduction_ja":"","introduction_en":"","category":"Category not supplied","task_count":null,"acquisition_status":"Acquisition status not supplied","official_url":"https://github.com/google-deepmind/bbeh","indexing_mode":"index","profile":{"resources":[],"task_format":"","scoring":"","metric":"","size":"","answer_access":"","license":"","citation":"","maintainer":"","released":"","why_hard":"","related":[],"history":"","construction":"","limitations":"","saturation":"","scores":[]}},"task_id":"58ab6194-94c7-5fee-9fc2-ec5485e1a195","task_key":"bbeh~5fweb~5fof~5flies--full--58ab6194-94c7-5fee-9fc2-ec5485e1a195","task_revision_id":"1","upstream_id":"","short_description":"In this question, assume each person either always tells the truth or always…","config":"bbeh_web_of_lies","split":"full","body":"{\"input\":\"In this question, assume each person either always tells the truth or always lies. Osmond says Mallory tells the truth. Bernita says Phoebe lies. Leda says either exactly two of Fidel, Mandy and Orla tell the truth or none of them. Oswin says Octavia lies. Sherrie says either exactly two of Yoland, Mitchell and Orly tell the truth or none of them. Kandi says only one of Christie, Mason and Maynard lies. Christie lies. Matthew says Disha lies. Melinda says Meadow lies. Meadow says Melinda lies. Owen says Marty lies. Gwenn says only one of Delfina, McKenzie and Owen lies. Maynard says Ozzy lies. Orla says Mandy tells the truth. Mason says Ozzy tells the truth. Lorine says only one of Crista, Martha and Matthew lies. Octavia says Oswin lies. Nishanth says Max lies. Jamey says either all three of Sherrie, Miguel and Onyx lie, or two of them tell the truth. Mitchell says Orly tells the truth. Crista says exactly two of Vernell, Max and Nishanth tells the truth. Delfina says exactly one of Bernita, Monica and Michelle tell the truth. Phoebe says exactly two of Lorine, Olympia and Oriana tells the truth. Miguel says Onyx tells the truth. Martha says Disha tells the truth. McKenzie says Marty tells the truth. Vernell says Kandi tells the truth. Shenna says exactly two of Jamey, Octavia and Oswin tell the truth. Mandy says Orla tells the truth. Michelle says Monica lies. Onyx says Miguel tells the truth. Monica says Michelle lies. Max says Nishanth lies. Orly says Mitchell tells the truth. Maxim says Mallory lies. Fidel says only one of Alejandro, Osmond and Maxim lies. Olympia says Oriana lies. Alejandro says Shenna tells the truth. Oriana says Olympia lies. Yoland says exactly one of Gwenn, Melinda and Meadow tells the truth.Do Shenna, Alejandro, and Leda tell the truth? Your answer should be a list of three words separated by commas. Each of them must be either yes (if the person tells the truth), no (if the person lies), or unknown (if the person's truth cannot be determined). For example, <unknown, no, yes> or <yes, yes, yes>.\"}","display_format":"text","language":"","answer_status":"published","assets":[],"source_url":"https://github.com/google-deepmind/bbeh","history":"initial import","indexing_mode":"index","subproblems":[],"grids":[]}