{"kind":"task","effective_mode":"full","benchmark":{"kind":"benchmark","effective_mode":"full","slug":"bbeh","formal_name":"BIG-Bench Extra Hard (BBEH)","introduction":"BIG-Bench Extra Hard evaluates language-model reasoning through challenging task categories. It offers full and mini distributions; this import contains the full distribution: 4,520 questions across all 23 task categories.","introduction_ja":"","introduction_en":"","category":"Category not supplied","task_count":null,"acquisition_status":"Acquisition status not supplied","official_url":"https://github.com/google-deepmind/bbeh","indexing_mode":"index","profile":{"resources":[],"task_format":"","scoring":"","metric":"","size":"","answer_access":"","license":"","citation":"","maintainer":"","released":"","why_hard":"","related":[],"history":"","construction":"","limitations":"","saturation":"","scores":[]}},"task_id":"757f2859-8605-505e-ad1d-a8579aaa3030","task_key":"bbeh~5fweb~5fof~5flies--full--757f2859-8605-505e-ad1d-a8579aaa3030","task_revision_id":"1","upstream_id":"","short_description":"In this question, assume each person either always tells the truth or always…","config":"bbeh_web_of_lies","split":"full","body":"{\"input\":\"In this question, assume each person either always tells the truth or always lies. Onyx says Disha lies. Merlin says Oscar tells the truth. Vernell says only one of Audrie, Ozias and Molly tells the truth. Fletcher says only one of Shaunda, Mercedes and Onyx lies. Otis says Ovid lies. Oscar says Merlin tells the truth. Maybelle says Ovid tells the truth. Lorine says either exactly two of Amberly, Odysseus and Miller tell the truth or none of them. Miller says Odysseus tells the truth. Olympia says Maximilian tells the truth. Audrie lies. Mercedes says Disha tells the truth. Ozias says Miguel tells the truth. Osbert says Marsha tells the truth. Odysseus says Miller tells the truth. Hritik says Osvaldo tells the truth. Osborne says Morris lies. Dallas says Tamika lies. Maximilian says Olympia tells the truth. Mariah says Melanie tells the truth. Ka says exactly one of Lorine, Morris and Osborne tells the truth.Melanie says Mariah tells the truth. Melinda says Oden tells the truth. Ocean says Oswin lies. Elanor says either all three of Bernita, Osvaldo and Hritik tell the truth or only one of them. Oswald says Maynard lies. Osvaldo says Hritik tells the truth. Vina says exactly one of Delfina, Maynard and Oswald tell the truth. Bernita says either all three of Dallas, Merlin and Oscar tell the truth or only one of them. Amberly says either all three of Gwenn, Olympia and Maximilian tell the truth or only one of them. Shaunda says only one of Andree, Maybelle and Otis lies. Tamika says either exactly one of Vina, Melinda and Oden tells the truth, or all three of them. Sal says Fletcher tells the truth. Marsha says Osbert tells the truth. Delfina says exactly two of Vernell, Ocean and Oswin tell the truth. Morris says Osborne lies. Maynard says Oswald lies. Andree says either all three of Christie, Melanie and Mariah tell the truth or only one of them. Christie says either all three of Elanor, Marsha and Osbert tell the truth or only one of them. Oswin says Ocean lies. Oden says Melinda tells the truth. Molly says Miguel lies. Gwenn says Sal tells the truth. Do Sal, Otis, and Ka tell the truth? Your answer should be a list of three words separated by commas. Each of them must be either yes (if the person tells the truth), no (if the person lies), or unknown (if the person's truth cannot be determined). For example, <unknown, no, yes> or <yes, yes, yes>.\"}","display_format":"text","language":"","answer_status":"published","assets":[],"source_url":"https://github.com/google-deepmind/bbeh","history":"initial import","indexing_mode":"index","subproblems":[],"grids":[]}