{"kind":"task","effective_mode":"full","benchmark":{"kind":"benchmark","effective_mode":"full","slug":"bbeh","formal_name":"BIG-Bench Extra Hard (BBEH)","introduction":"BIG-Bench Extra Hard evaluates language-model reasoning through challenging task categories. It offers full and mini distributions; this import contains the full distribution: 4,520 questions across all 23 task categories.","introduction_ja":"","introduction_en":"","category":"Category not supplied","task_count":null,"acquisition_status":"Acquisition status not supplied","official_url":"https://github.com/google-deepmind/bbeh","indexing_mode":"index","profile":{"resources":[],"task_format":"","scoring":"","metric":"","size":"","answer_access":"","license":"","citation":"","maintainer":"","released":"","why_hard":"","related":[],"history":"","construction":"","limitations":"","saturation":"","scores":[]}},"task_id":"01df78a6-412f-5c2b-bb18-3f8826a53bf9","task_key":"bbeh~5fweb~5fof~5flies--full--01df78a6-412f-5c2b-bb18-3f8826a53bf9","task_revision_id":"1","upstream_id":"","short_description":"In this question, assume each person either always tells the truth or always…","config":"bbeh_web_of_lies","split":"full","body":"{\"input\":\"In this question, assume each person either always tells the truth or always lies. Madeleine says Onyx lies. Oswin says Oprah lies. Ottoline says Oakleigh lies. Delfina says exactly one of Kristian, Madeleine and Onyx tell the truth. Virginia says Mina lies. Orly says Mohamed tells the truth. Oakleigh says Ottoline lies. Michelle says Ophelia tells the truth. Mickey says Disha lies. Antwan says only one of Bernita, Mitchell and Otto tells the truth. Ophelia says Michelle tells the truth. Sal says either exactly two of Rashida, Martha and Mona tell the truth or none of them. Monica says Disha tells the truth. Ryan says exactly one of Amberly, Mina and Virginia tell the truth. Jaymie says Delfina lies. Fletcher says Tamika tells the truth. Oprah says Oswin lies. Max says Oscar tells the truth. Rashida says only one of Fletcher, Monica and Mickey lies. Amberly says Sima tells the truth. Bernita says exactly two of Christie, Oswin and Oprah tells the truth. Martha says Mona tells the truth. Kristian says either all three of Teressa, Max and Oscar tell the truth or only one of them. Mitchell says Murray tells the truth. Sima says either all three of Dallas, Mohamed and Orly tell the truth or only one of them. Tamika says exactly two of Antwan, Oakleigh and Ottoline tell the truth. Onyx says Madeleine lies. Teressa says exactly one of Sal, Octavian and Margaret tells the truth.Mohamed says Orly tells the truth. Mona says Martha tells the truth. Christie says either exactly two of Jerry, Ophelia and Michelle tell the truth or none of them. Otto says Murray lies. Margaret says Octavian lies. Oscar says Max tells the truth. Jerry says Ryan lies. Octavian says Margaret lies. Mina says Virginia lies. Dallas tells the truth. Do Kristian, Sal, and Jaymie tell the truth? Your answer should be a list of three words separated by commas. Each of them must be either yes (if the person tells the truth), no (if the person lies), or unknown (if the person's truth cannot be determined). For example, <unknown, no, yes> or <yes, yes, yes>.\"}","display_format":"text","language":"","answer_status":"published","assets":[],"source_url":"https://github.com/google-deepmind/bbeh","history":"initial import","indexing_mode":"index","subproblems":[],"grids":[]}