{"kind":"task","effective_mode":"full","benchmark":{"kind":"benchmark","effective_mode":"full","slug":"bbeh","formal_name":"BIG-Bench Extra Hard (BBEH)","introduction":"BIG-Bench Extra Hard evaluates language-model reasoning through challenging task categories. It offers full and mini distributions; this import contains the full distribution: 4,520 questions across all 23 task categories.","introduction_ja":"","introduction_en":"","category":"Category not supplied","task_count":null,"acquisition_status":"Acquisition status not supplied","official_url":"https://github.com/google-deepmind/bbeh","indexing_mode":"index","profile":{"resources":[],"task_format":"","scoring":"","metric":"","size":"","answer_access":"","license":"","citation":"","maintainer":"","released":"","why_hard":"","related":[],"history":"","construction":"","limitations":"","saturation":"","scores":[]}},"task_id":"b26b64a4-d760-54f5-8bde-a4b914a22254","task_key":"bbeh~5fweb~5fof~5flies--full--b26b64a4-d760-54f5-8bde-a4b914a22254","task_revision_id":"1","upstream_id":"","short_description":"In this question, assume each person either always tells the truth or always…","config":"bbeh_web_of_lies","split":"full","body":"{\"input\":\"In this question, assume each person either always tells the truth or always lies. Melvin says Mark lies. Tamika says Phoebe tells the truth. Oscar says Michelle lies. Mavis says Mark tells the truth. Phoebe says Delbert lies. Bernita says only one of Alexis, Margaret and Mohamed lies. Mohamed says Merlin lies. Raymond says Antwan tells the truth. Milton says Oswin tells the truth. Othello says John lies. Antwan says exactly one of Andree, Oswald and Madeleine tell the truth. Michaela says Maxim tells the truth. Muriel says Nicole tells the truth. Uri says Michelle tells the truth. Lorine says exactly two of Elanor, Oliver and Orlando tell the truth. Orlando says Oliver lies. Christie says Jerry lies. Kandi says exactly two of Christie, Octavio and Olivia tell the truth. Odysseus says Ozzie tells the truth. Nicole says Muriel tells the truth. Shaunda says only one of Lorine, Mallory and Othello lies. Oswald says Madeleine lies. Olivia says Octavio lies. Andree says only one of Tamika, Mavis and Melvin lies. Margaret says Merlin tells the truth. Shalonda says Yoland, Milton and Malcolm all tell the truth. Osbert says Lalit lies. Madeleine says Oswald lies. Vina says either all three of Kandi, Muriel and Nicole tell the truth or only one of them. Yoland says only one of Vina, Uri and Oscar lies. Malcolm says Oswin lies. Oona says Maxim lies. Vernell lies. Ozzie says Odysseus tells the truth. Octavio says Olivia lies. Mallory says John tells the truth. Delbert says exactly two of Vernell, Lalit and Osbert tells the truth. Lalit says Osbert lies. Elanor says only one of Bernita, Michaela and Oona tells the truth. Jerry says either exactly two of Shaunda, Ozzie and Odysseus tell the truth or none of them. Alexis says Raymond tells the truth. Oliver says Orlando lies. Do Shaunda, Orlando, and Shalonda tell the truth? Your answer should be a list of three words separated by commas. Each of them must be either yes (if the person tells the truth), no (if the person lies), or unknown (if the person's truth cannot be determined). For example, <unknown, no, yes> or <yes, yes, yes>.\"}","display_format":"text","language":"","answer_status":"published","assets":[],"source_url":"https://github.com/google-deepmind/bbeh","history":"initial import","indexing_mode":"index","subproblems":[],"grids":[]}