{"kind":"task","effective_mode":"full","benchmark":{"kind":"benchmark","effective_mode":"full","slug":"bbeh","formal_name":"BIG-Bench Extra Hard (BBEH)","introduction":"BIG-Bench Extra Hard evaluates language-model reasoning through challenging task categories. It offers full and mini distributions; this import contains the full distribution: 4,520 questions across all 23 task categories.","introduction_ja":"","introduction_en":"","category":"Category not supplied","task_count":null,"acquisition_status":"Acquisition status not supplied","official_url":"https://github.com/google-deepmind/bbeh","indexing_mode":"index","profile":{"resources":[],"task_format":"","scoring":"","metric":"","size":"","answer_access":"","license":"","citation":"","maintainer":"","released":"","why_hard":"","related":[],"history":"","construction":"","limitations":"","saturation":"","scores":[]}},"task_id":"655de82d-8ae5-5e31-9e87-3c624722f8e9","task_key":"bbeh~5fweb~5fof~5flies--full--655de82d-8ae5-5e31-9e87-3c624722f8e9","task_revision_id":"1","upstream_id":"","short_description":"In this question, assume each person either always tells the truth or always…","config":"bbeh_web_of_lies","split":"full","body":"{\"input\":\"In this question, assume each person either always tells the truth or always lies. Jerry says either all three of Phoebe, Lalit and Miles tell the truth or only one of them. Sima lies. Jamey says exactly one of Ryan, Maxim and Orlando tell the truth. Andree says only one of Ka, Madeleine and Odin tells the truth. Michelle says Mariana lies. Moses says Obadiah tells the truth. Miles says Lalit tells the truth. Odin says Osborne lies. Maxim says Orlando lies. Ovid says Mavis tells the truth. Millicent says John lies. Elanor says either exactly one of Vernell, Mario and Olive tells the truth, or all three of them. Kandi says Kristian, Morris and Meredith all tell the truth. Lalit says Miles tells the truth. Dallas says either exactly one of Conception, Bahar and Mercedes tells the truth, or all three of them. John says Millicent lies. Vernell says either exactly two of Inga, Monica and Marshall tell the truth or none of them. Teressa says Elanor tells the truth. Antwan says exactly two of Amberly, Mariana and Michelle tells the truth. Maynard says Marty tells the truth. Ka says Fletcher tells the truth. Mandy says Mohamed lies. Madeleine says Osborne tells the truth. Orlando says Maxim lies. Mariah says Nishanth tells the truth. Willian says either all three of Andree, Nishanth and Mariah tell the truth or only one of them. Mario says Olive tells the truth. Meredith says Oran lies. Alejandro says only one of Dallas, Oakley and Orhan tells the truth. Phoebe says Sima lies. Oakley says Madison tells the truth. Orla says Obadiah lies. Mariana says Michelle lies. Misty says Virginia tells the truth. Uri says Oprah lies. Olive says Mario tells the truth. Inga says only one of Jerry, Moses and Orla lies. Mohamed says Mandy lies. Raymond says exactly two of Tamika, Mohamed and Mandy tells the truth. Tamika says Shalonda tells the truth. Orhan says Madison lies. Nishanth says Mariah tells the truth. Monica says Marshall tells the truth. Mercedes says Bahar tells the truth. Shalonda says Alejandro, Misty and Marvin all tell the truth. Ozzie says Marty lies. Morris says Oran tells the truth. Conception says exactly two of Teressa, Millicent and John tells the truth. Marvin says Virginia lies. Fletcher says only one of Kandi, Maynard and Ozzie lies. Bahar says Mercedes tells the truth. Marshall says Monica tells the truth. Amberly says exactly one of Willian, Uri and Oprah tell the truth. Oprah says Uri lies. Kristian says either all three of Raymond, Mavis and Ovid lie, or two of them tell the truth. Ryan says Leda tells the truth. Mavis says Ovid tells the truth. Leda says Antwan lies. Do Willian, Odin, and Jamey tell the truth? Your answer should be a list of three words separated by commas. Each of them must be either yes (if the person tells the truth), no (if the person lies), or unknown (if the person's truth cannot be determined). For example, <unknown, no, yes> or <yes, yes, yes>.\"}","display_format":"text","language":"","answer_status":"published","assets":[],"source_url":"https://github.com/google-deepmind/bbeh","history":"initial import","indexing_mode":"index","subproblems":[],"grids":[]}