{"kind":"task","effective_mode":"full","benchmark":{"kind":"benchmark","effective_mode":"full","slug":"bbeh","formal_name":"BIG-Bench Extra Hard (BBEH)","introduction":"BIG-Bench Extra Hard evaluates language-model reasoning through challenging task categories. It offers full and mini distributions; this import contains the full distribution: 4,520 questions across all 23 task categories.","introduction_ja":"","introduction_en":"","category":"Category not supplied","task_count":null,"acquisition_status":"Acquisition status not supplied","official_url":"https://github.com/google-deepmind/bbeh","indexing_mode":"index","profile":{"resources":[],"task_format":"","scoring":"","metric":"","size":"","answer_access":"","license":"","citation":"","maintainer":"","released":"","why_hard":"","related":[],"history":"","construction":"","limitations":"","saturation":"","scores":[]}},"task_id":"c5e21c1d-7caf-53bf-b6d9-cc6d0d56be31","task_key":"bbeh~5fweb~5fof~5flies--full--c5e21c1d-7caf-53bf-b6d9-cc6d0d56be31","task_revision_id":"1","upstream_id":"","short_description":"In this question, assume each person either always tells the truth or always…","config":"bbeh_web_of_lies","split":"full","body":"{\"input\":\"In this question, assume each person either always tells the truth or always lies. Melody says Oswald tells the truth. Phoebe says Shaunda, Ophelia and Osvaldo all tell the truth. Orhan says Minerva lies. Shalonda says Gwenn tells the truth. Miller says Margaret tells the truth. Delfina tells the truth. Ocean says Octavius lies. Osmond says Octavia lies. Amberly says Kristian tells the truth. Minerva says Orhan lies. Oakleigh says Olympia lies. Gwenn says either exactly two of Jerry, Margaret and Miller tell the truth or none of them. Olympia says Oakleigh lies. Orin says Mark lies. Owyn says Mustafa lies. Yoland says either all three of Fidel, Melody and Oswald lie, or two of them tell the truth. Mariana says Nicole lies. Tamika says exactly two of Shalonda, Nicole and Mariana tells the truth. Oriana says Octavia tells the truth. Alejandro says Lorine tells the truth. Antwan says only one of Fletcher, Obadiah and Merlin lies. Merlin says Misty lies. Jaymie says Alejandro, Oriana and Osmond all tell the truth. Fletcher says exactly two of Jaymie, Olympia and Oakleigh tells the truth. Ophelia says Melvin tells the truth. Dallas says Amberly lies. Fidel says exactly one of Delfina, Orin and Mark tell the truth. Shaunda says Crista lies. Margaret says Miller tells the truth. Ryan says exactly two of Delbert, Mustafa and Owyn tell the truth. Delbert says Phoebe lies. Obadiah says Misty tells the truth. Mustafa says Owyn lies. Kristian says exactly two of Tamika, Minerva and Orhan tells the truth. Nicole says Mariana lies. Lorine says exactly two of Yoland, Octavius and Ocean tell the truth. Octavius says Ocean lies. Jerry says Ryan tells the truth. Oswald says Melody tells the truth. Crista says Antwan tells the truth. Osvaldo says Melvin lies. Mark says Orin lies. Do Kristian, Orin, and Dallas tell the truth? Your answer should be a list of three words separated by commas. Each of them must be either yes (if the person tells the truth), no (if the person lies), or unknown (if the person's truth cannot be determined). For example, <unknown, no, yes> or <yes, yes, yes>.\"}","display_format":"text","language":"","answer_status":"published","assets":[],"source_url":"https://github.com/google-deepmind/bbeh","history":"initial import","indexing_mode":"index","subproblems":[],"grids":[]}