{"kind":"task","effective_mode":"full","benchmark":{"kind":"benchmark","effective_mode":"full","slug":"bbeh","formal_name":"BIG-Bench Extra Hard (BBEH)","introduction":"BIG-Bench Extra Hard evaluates language-model reasoning through challenging task categories. It offers full and mini distributions; this import contains the full distribution: 4,520 questions across all 23 task categories.","introduction_ja":"","introduction_en":"","category":"Category not supplied","task_count":null,"acquisition_status":"Acquisition status not supplied","official_url":"https://github.com/google-deepmind/bbeh","indexing_mode":"index","profile":{"resources":[],"task_format":"","scoring":"","metric":"","size":"","answer_access":"","license":"","citation":"","maintainer":"","released":"","why_hard":"","related":[],"history":"","construction":"","limitations":"","saturation":"","scores":[]}},"task_id":"31bf4e1e-bc7a-5454-9eb4-037364f72365","task_key":"bbeh~5fweb~5fof~5flies--full--31bf4e1e-bc7a-5454-9eb4-037364f72365","task_revision_id":"1","upstream_id":"","short_description":"In this question, assume each person either always tells the truth or always…","config":"bbeh_web_of_lies","split":"full","body":"{\"input\":\"In this question, assume each person either always tells the truth or always lies. Delfina says only one of Shenna, Osbert and Matthew tells the truth. Amberly says either all three of Audrie, Martin and Uri lie, or two of them tell the truth. Oswald says Mina lies. Maybelle says Miguel lies. Elanor says only one of Shalonda, Margaret and Mario lies. Othello says Ophelia lies. Minerva says Mallory tells the truth. Meadow says Maxim tells the truth. Mallory says Minerva tells the truth. Yoland says exactly two of Andree, Osian and Omar tell the truth. Jamey says Amberly lies. Orhan says Mark tells the truth. Sanket says Oriana tells the truth. Jim says Raymond tells the truth. Odin says Mina tells the truth. Maria says Melody tells the truth. Sal says either exactly one of Elanor, Orhan and Mark tells the truth, or all three of them. Andree says Jamey lies. McKenzie says Ofelia lies. Audrie says either exactly two of Sima, Maxim and Meadow tell the truth or none of them. Jerry says either all three of Crista, Minerva and Mallory lie, or two of them tell the truth. Martin says Uri tells the truth. Sherrie says Ryan lies. Matthew says Melinda lies. Fletcher says only one of Bernita, Mustafa and Maybelle tells the truth. Osbert says Melinda tells the truth. Uri says Martin tells the truth. Shaunda says Jim tells the truth. Octavio says Kate lies. Shalonda says exactly one of Fletcher, Othello and Ophelia tell the truth. Mark says Orhan tells the truth. Mustafa says Miguel tells the truth. Crista says Helene, Sanket and Odysseus all tell the truth. Omar says Osian lies. Ophelia says Othello lies. Oleg says Osvaldo tells the truth. Shenna says exactly two of Shaunda, Octavio and Kate tells the truth. Sima says only one of Sherrie, Odin and Oswald lies. Margaret says Osmond tells the truth. Bernita says Yoland lies. Oswin says Osric tells the truth. Mario says Osmond lies. Orly says Mitchell tells the truth. Helene says either all three of Christie, Osvaldo and Oleg lie, or two of them tell the truth. Odysseus says Oriana lies. Murray says Mitchell lies. Osvaldo says Oleg tells the truth. Raymond says only one of Sal, Orly and Murray lies. Ryan lies. Kate says Octavio lies. Melody says Maria tells the truth. Marvin says Osric lies. Rashida says either exactly one of Lorine, Melody and Maria tells the truth, or all three of them. Ofelia says McKenzie lies. Maxim says Meadow tells the truth. Osian says Omar lies. Christie says exactly two of Rashida, Ofelia and McKenzie tells the truth. Lorine says Delfina, Oswin and Marvin all tell the truth. Do Delfina, Crista, and Jerry tell the truth? Your answer should be a list of three words separated by commas. Each of them must be either yes (if the person tells the truth), no (if the person lies), or unknown (if the person's truth cannot be determined). For example, <unknown, no, yes> or <yes, yes, yes>.\"}","display_format":"text","language":"","answer_status":"published","assets":[],"source_url":"https://github.com/google-deepmind/bbeh","history":"initial import","indexing_mode":"index","subproblems":[],"grids":[]}