{"kind":"task","effective_mode":"full","benchmark":{"kind":"benchmark","effective_mode":"full","slug":"bbeh","formal_name":"BIG-Bench Extra Hard (BBEH)","introduction":"BIG-Bench Extra Hard evaluates language-model reasoning through challenging task categories. It offers full and mini distributions; this import contains the full distribution: 4,520 questions across all 23 task categories.","introduction_ja":"","introduction_en":"","category":"Category not supplied","task_count":null,"acquisition_status":"Acquisition status not supplied","official_url":"https://github.com/google-deepmind/bbeh","indexing_mode":"index","profile":{"resources":[],"task_format":"","scoring":"","metric":"","size":"","answer_access":"","license":"","citation":"","maintainer":"","released":"","why_hard":"","related":[],"history":"","construction":"","limitations":"","saturation":"","scores":[]}},"task_id":"09cd2148-3845-543d-b478-65343fb52456","task_key":"bbeh~5fweb~5fof~5flies--full--09cd2148-3845-543d-b478-65343fb52456","task_revision_id":"1","upstream_id":"","short_description":"In this question, assume each person either always tells the truth or always…","config":"bbeh_web_of_lies","split":"full","body":"{\"input\":\"In this question, assume each person either always tells the truth or always lies. Orla says Otto tells the truth. Marlon says Olaf lies. Otto says Orla tells the truth. Odessa says Milton lies. Morris says Oakleigh lies. Antwan says either exactly two of Raymond, Osmond and Molly tell the truth or none of them. Jerry says exactly one of Inga, Mario and Nicole tell the truth. Raymond tells the truth. Marvin says McKenzie lies. Martha says Osvaldo lies. Bernita says Shaunda lies. Ocean says Maria tells the truth. Owen says Oleg tells the truth. Oriana says McKenzie tells the truth. Alexis says Rashida lies. Osvaldo says Martha lies. Rashida says exactly two of Sherrie, Marlon and Olaf tell the truth. Oleg says Owen tells the truth. Oakleigh says Morris lies. Jim says only one of Amberly, Ocean and Marsha lies. Amberly says Delbert tells the truth. Vina says exactly two of Bernita, Monica and Maurice tells the truth. Milton says Odessa lies. Dallas says exactly two of Alexis, Morris and Oakleigh tells the truth. Monica says Maurice lies. Marsha says Maria lies. Sherrie says exactly one of Jim, Milton and Odessa tells the truth.Inga says only one of Dallas, Oriana and Marvin tells the truth. Shaunda says either all three of Antwan, Orla and Otto lie, or two of them tell the truth. Olaf says Marlon lies. Mario says Nicole lies. Molly says Osmond tells the truth. Willian says exactly one of Jerry, Martha and Osvaldo tells the truth.Maurice says Monica lies. Delbert says either exactly one of Vina, Owen and Oleg tells the truth, or all three of them. Nicole says Mario lies. Osmond says Molly tells the truth. Do Dallas, Alexis, and Willian tell the truth? Your answer should be a list of three words separated by commas. Each of them must be either yes (if the person tells the truth), no (if the person lies), or unknown (if the person's truth cannot be determined). For example, <unknown, no, yes> or <yes, yes, yes>.\"}","display_format":"text","language":"","answer_status":"published","assets":[],"source_url":"https://github.com/google-deepmind/bbeh","history":"initial import","indexing_mode":"index","subproblems":[],"grids":[]}