{"kind":"task","effective_mode":"full","benchmark":{"kind":"benchmark","effective_mode":"full","slug":"bbeh","formal_name":"BIG-Bench Extra Hard (BBEH)","introduction":"BIG-Bench Extra Hard evaluates language-model reasoning through challenging task categories. It offers full and mini distributions; this import contains the full distribution: 4,520 questions across all 23 task categories.","introduction_ja":"","introduction_en":"","category":"Category not supplied","task_count":null,"acquisition_status":"Acquisition status not supplied","official_url":"https://github.com/google-deepmind/bbeh","indexing_mode":"index","profile":{"resources":[],"task_format":"","scoring":"","metric":"","size":"","answer_access":"","license":"","citation":"","maintainer":"","released":"","why_hard":"","related":[],"history":"","construction":"","limitations":"","saturation":"","scores":[]}},"task_id":"27ad05e0-b745-5812-b85c-564bb25cf494","task_key":"bbeh~5fweb~5fof~5flies--full--27ad05e0-b745-5812-b85c-564bb25cf494","task_revision_id":"1","upstream_id":"","short_description":"In this question, assume each person either always tells the truth or always…","config":"bbeh_web_of_lies","split":"full","body":"{\"input\":\"In this question, assume each person either always tells the truth or always lies. Gwenn says Amberly lies. Sanket says Muhammad lies. Orin says Owyn tells the truth. Murphy says Maurice tells the truth. Jamey says either exactly two of Kandi, Orson and Orly tell the truth or none of them. Shenna says Jerry tells the truth. Sal says Jamey lies. Delbert says only one of Dallas, Molly and Osborn lies. Molly says Matthew tells the truth. Maxim says Michelle lies. Ollie says Mallory lies. Morris says Miles lies. Maurice says Murphy tells the truth. Marsha says John tells the truth. Yoland says either all three of Lorine, Maria and Orhan tell the truth or only one of them. Odysseus says McKenzie lies. Oswald says Odell lies. Phoebe says Leda lies. Orly says Orson tells the truth. Odile says Miguel tells the truth. Rashida says Shenna, Marsha and Maxwell all tell the truth. Mehran says Miles tells the truth. Raymond says Gwenn, Odin and Odysseus all tell the truth. Kandi says Delbert lies. Maxwell says John lies. Orhan says Maria tells the truth. Christie says Rashida tells the truth. Odette says Megan tells the truth. Vina says Tamika tells the truth. Murray says Octavia lies. Hritik says Miguel lies. Conception says either all three of Helene, Owyn and Orin tell the truth or only one of them. Octavia says Murray lies. Jerry says Shaunda lies. Leda tells the truth. Owyn says Orin tells the truth. Odin says McKenzie tells the truth. Orion says Ozzy tells the truth. Oscar says Millicent lies. Tamika says exactly two of Inga, Murray and Octavia tell the truth. Marina says Ophelia lies. Dallas says exactly one of Conception, Ollie and Mallory tell the truth. Uri says Oden tells the truth. Alejandro says only one of Yoland, Ocean and Oswald lies. Mindy says Oden lies. Osborn says Matthew lies. Ocean says Odell tells the truth. Teressa says exactly two of Vernell, Sanket and Muhammad tell the truth. Ka says only one of Andree, Orion and Mona lies. Inga says Elanor tells the truth. Miller says Ophelia tells the truth. Mickey says Megan lies. Muhammad says Sanket lies. Michelle says Maxim lies. Andree says only one of Raymond, Mehran and Morris tells the truth. Shaunda says Teressa, Uri and Mindy all tell the truth. Orson says Orly tells the truth. Amberly says exactly one of Fidel, Millicent and Oscar tell the truth. Lorine says either all three of Phoebe, Maurice and Murphy lie, or two of them tell the truth. Mallory says Ollie lies. Vernell says only one of Alejandro, Miller and Marina lies. Elanor says only one of Shalonda, Odile and Hritik lies. Shalonda says exactly two of Ka, Maxim and Michelle tell the truth. Mona says Ozzy lies. Millicent says Oscar lies. Fidel says only one of Christie, Odette and Mickey tells the truth. Maria says Orhan tells the truth. Helene says Vina tells the truth. Do Shalonda, Fidel, and Sal tell the truth? Your answer should be a list of three words separated by commas. Each of them must be either yes (if the person tells the truth), no (if the person lies), or unknown (if the person's truth cannot be determined). For example, <unknown, no, yes> or <yes, yes, yes>.\"}","display_format":"text","language":"","answer_status":"published","assets":[],"source_url":"https://github.com/google-deepmind/bbeh","history":"initial import","indexing_mode":"index","subproblems":[],"grids":[]}