{"kind":"task","effective_mode":"full","benchmark":{"kind":"benchmark","effective_mode":"full","slug":"bbeh","formal_name":"BIG-Bench Extra Hard (BBEH)","introduction":"BIG-Bench Extra Hard evaluates language-model reasoning through challenging task categories. It offers full and mini distributions; this import contains the full distribution: 4,520 questions across all 23 task categories.","introduction_ja":"","introduction_en":"","category":"Category not supplied","task_count":null,"acquisition_status":"Acquisition status not supplied","official_url":"https://github.com/google-deepmind/bbeh","indexing_mode":"index","profile":{"resources":[],"task_format":"","scoring":"","metric":"","size":"","answer_access":"","license":"","citation":"","maintainer":"","released":"","why_hard":"","related":[],"history":"","construction":"","limitations":"","saturation":"","scores":[]}},"task_id":"7c1d21d2-dacf-537f-ab42-552dda0988f4","task_key":"bbeh~5fweb~5fof~5flies--full--7c1d21d2-dacf-537f-ab42-552dda0988f4","task_revision_id":"1","upstream_id":"","short_description":"In this question, assume each person either always tells the truth or always…","config":"bbeh_web_of_lies","split":"full","body":"{\"input\":\"In this question, assume each person either always tells the truth or always lies. Millie says Orson lies. Othello says John tells the truth. Shaunda says exactly two of Delbert, Mariana and Milton tell the truth. Maxwell says Ollie tells the truth. Mariana says Milton lies. Delbert says Conception tells the truth. Ozzy says McKenzie lies. Mustafa says Oran lies. Mona says Micah tells the truth. Melody says Mike lies. Orlando says Mercedes tells the truth. Maya says Ophelia tells the truth. Dallas says either all three of Shalonda, Oriana and Lalit tell the truth or only one of them. Ocean says Miranda tells the truth. Osborn says Mindy lies. Millicent says John lies. Orson says Millie lies. Tamika says only one of Ryan, Owen and Morris lies. Shalonda says exactly two of Willian, Maybelle and Maximilian tell the truth. Maynard says Mindy tells the truth. Maybelle says Maximilian lies. Monica says Ophelia lies. Willian says exactly one of Vernell, Owyn and Octavian tells the truth.Andree says Leda tells the truth. Delfina says Shaunda, Maynard and Osborn all tell the truth. Inga says Rashida lies. Mark says Mike tells the truth. Conception says only one of Jaymie, Martin and Odin lies. Oran says Mustafa lies. Crista says exactly one of Sal, Oda and Virginia tells the truth.Audrie says exactly one of Ka, Oran and Mustafa tell the truth. McKenzie says Ozzy lies. Oda says Virginia lies. Leda tells the truth. Rashida says exactly one of Helene, Millie and Orson tells the truth.Jaymie says only one of Inga, Othello and Millicent tells the truth. Owyn says Octavian lies. Margaret says Mercedes lies. Oriana says Lalit tells the truth. Sal says Andree lies. Martin says Miriam tells the truth. Shenna says only one of Dallas, Maya and Monica lies. Odin says Miriam lies. Micah says Mona tells the truth. Morris says Ozzie lies. Virginia says Oda lies. Octavian says Owyn lies. Lalit says Oriana tells the truth. Owen says Ozzie tells the truth. Milton says Mariana lies. Sherrie says exactly two of Teressa, Ozzy and McKenzie tell the truth. Teressa says either all three of Delfina, Ollie and Maxwell lie, or two of them tell the truth. Miranda says Ocean tells the truth. Antwan says Shenna, Orlando and Margaret all tell the truth. Ka says only one of Sherrie, Mark and Melody lies. Fletcher says either exactly one of Antwan, Mona and Micah tells the truth, or all three of them. Maximilian says Maybelle lies. Ollie says Maxwell tells the truth. Ryan says Audrie lies. Helene says Fletcher tells the truth. Vernell says either exactly two of Crista, Ocean and Miranda tell the truth or none of them. Do Jaymie, Ka, and Tamika tell the truth? Your answer should be a list of three words separated by commas. Each of them must be either yes (if the person tells the truth), no (if the person lies), or unknown (if the person's truth cannot be determined). For example, <unknown, no, yes> or <yes, yes, yes>.\"}","display_format":"text","language":"","answer_status":"published","assets":[],"source_url":"https://github.com/google-deepmind/bbeh","history":"initial import","indexing_mode":"index","subproblems":[],"grids":[]}