{"kind":"task","effective_mode":"full","benchmark":{"kind":"benchmark","effective_mode":"full","slug":"bbeh","formal_name":"BIG-Bench Extra Hard (BBEH)","introduction":"BIG-Bench Extra Hard evaluates language-model reasoning through challenging task categories. It offers full and mini distributions; this import contains the full distribution: 4,520 questions across all 23 task categories.","introduction_ja":"","introduction_en":"","category":"Category not supplied","task_count":null,"acquisition_status":"Acquisition status not supplied","official_url":"https://github.com/google-deepmind/bbeh","indexing_mode":"index","profile":{"resources":[],"task_format":"","scoring":"","metric":"","size":"","answer_access":"","license":"","citation":"","maintainer":"","released":"","why_hard":"","related":[],"history":"","construction":"","limitations":"","saturation":"","scores":[]}},"task_id":"6b505441-0cfa-57cc-b0d9-e2f1da0c8784","task_key":"bbeh~5fweb~5fof~5flies--full--6b505441-0cfa-57cc-b0d9-e2f1da0c8784","task_revision_id":"1","upstream_id":"","short_description":"In this question, assume each person either always tells the truth or always…","config":"bbeh_web_of_lies","split":"full","body":"{\"input\":\"In this question, assume each person either always tells the truth or always lies. Vina says Sal tells the truth. Sanket says Octavian lies. Sima lies. Sherrie says only one of Vina, Melinda and Oakleigh lies. Mitchell says Marty lies. Maria says Disha tells the truth. Marina says Olga tells the truth. Murray says Ofelia tells the truth. Onyx says Olaf tells the truth. Owyn says Marty tells the truth. Phoebe says either all three of Lorine, Nishanth and Oakley lie, or two of them tell the truth. Jaymie says either exactly one of Rashida, Disha and Maria tells the truth, or all three of them. Marlon says Odessa lies. Fidel says only one of Sherrie, Owyn and Mitchell tells the truth. Muriel says Opal tells the truth. Milo says Obadiah lies. Disha says Maria tells the truth. Tamika says either all three of Sima, Odette and Monica lie, or two of them tell the truth. Opal says Muriel tells the truth. Obadiah says Milo lies. Raymond says either exactly two of Phoebe, Murray and Ofelia tell the truth or none of them. Osborn says Ottoline tells the truth. Lorine says either exactly one of Shaunda, Muriel and Opal tells the truth, or all three of them. Odessa says Marlon lies. Mario says Miriam tells the truth. Leda says Jaymie lies. Orin says Lalit tells the truth. Oakley says Nishanth tells the truth. Osvaldo says Olympia lies. Rashida says exactly two of Raymond, Osvaldo and Olympia tells the truth. Miriam says Mario tells the truth. Octavian says Sanket lies. Shaunda says either exactly one of Inga, Osborn and Ottoline tells the truth, or all three of them. Mehran says Osborne tells the truth. Melinda says Marshall tells the truth. Ofelia says Murray tells the truth. Fletcher says either exactly two of Leda, Lalit and Orin tell the truth or none of them. Sal says only one of Fletcher, Oswald and Michelle lies. Osborne says Mehran tells the truth. Oakleigh says Marshall lies. Audrie says Willian, Meredith and Madeleine all tell the truth. Olympia says Osvaldo lies. Willian says either all three of Conception, Osborne and Mehran lie, or two of them tell the truth. Ryan says exactly two of Audrie, Sanket and Octavian tells the truth. Odette says Monica tells the truth. Meredith says Minerva tells the truth. Madeleine says Minerva lies. Olga says Marina tells the truth. Amberly says either exactly two of Elanor, Miriam and Mario tell the truth or none of them. Michelle says Odin lies. Nishanth says Oakley tells the truth. Ka says either exactly two of Fidel, Marina and Olga tell the truth or none of them. Lalit says Orin tells the truth. Elanor says Ryan lies. Inga says Delfina, Onyx and Kate all tell the truth. Oswald says Odin tells the truth. Delfina says exactly two of Tamika, Obadiah and Milo tell the truth. Monica says Odette tells the truth. Kate says Olaf lies. Ottoline says Osborn tells the truth. Conception says exactly two of Ka, Odessa and Marlon tells the truth. Do Sal, Mitchell, and Amberly tell the truth? Your answer should be a list of three words separated by commas. Each of them must be either yes (if the person tells the truth), no (if the person lies), or unknown (if the person's truth cannot be determined). For example, <unknown, no, yes> or <yes, yes, yes>.\"}","display_format":"text","language":"","answer_status":"published","assets":[],"source_url":"https://github.com/google-deepmind/bbeh","history":"initial import","indexing_mode":"index","subproblems":[],"grids":[]}