{"kind":"task","effective_mode":"full","benchmark":{"kind":"benchmark","effective_mode":"full","slug":"bbeh","formal_name":"BIG-Bench Extra Hard (BBEH)","introduction":"BIG-Bench Extra Hard evaluates language-model reasoning through challenging task categories. It offers full and mini distributions; this import contains the full distribution: 4,520 questions across all 23 task categories.","introduction_ja":"","introduction_en":"","category":"Category not supplied","task_count":null,"acquisition_status":"Acquisition status not supplied","official_url":"https://github.com/google-deepmind/bbeh","indexing_mode":"index","profile":{"resources":[],"task_format":"","scoring":"","metric":"","size":"","answer_access":"","license":"","citation":"","maintainer":"","released":"","why_hard":"","related":[],"history":"","construction":"","limitations":"","saturation":"","scores":[]}},"task_id":"d3fca7e8-549c-5121-b68b-73fc418cbfe7","task_key":"bbeh~5fweb~5fof~5flies--full--d3fca7e8-549c-5121-b68b-73fc418cbfe7","task_revision_id":"1","upstream_id":"","short_description":"In this question, assume each person either always tells the truth or always…","config":"bbeh_web_of_lies","split":"full","body":"{\"input\":\"In this question, assume each person either always tells the truth or always lies. Orson says Orla lies. Jamey says exactly two of Elanor, Oriana and Otis tell the truth. Rashida says exactly two of Alejandro, Orin and Olive tell the truth. Oriana says Otis lies. Bahar says Mackenzie tells the truth. Sal says Vina tells the truth. Vernell says Jerry lies. Sima says only one of Gwenn, Murphy and Orson lies. Milton says Melinda lies. Dallas says exactly two of Sherrie, Margaret and Ozzy tell the truth. Ozzy says Margaret lies. Orin says Olive lies. Marsha says Megan lies. Olive says Orin lies. Fletcher says only one of Jim, Maria and Oden lies. Jerry says either exactly two of Tamika, Bahar and Mackenzie tell the truth or none of them. Jim says exactly two of Jamey, Melinda and Milton tell the truth. Tamika says Sima lies. Helene says Dallas tells the truth. Alejandro says Fletcher tells the truth. Oden says Mercedes lies. Oprah says Maynard lies. Sherrie says only one of Vernell, Maybelle and Odell lies. Elanor says Shalonda lies. Otis says Oriana lies. Obadiah says Olga tells the truth. Moses says Orlando tells the truth. Olga says Obadiah tells the truth. Shalonda says either exactly two of Andree, Obadiah and Olga tell the truth or none of them. Alexis says exactly two of Rashida, Maynard and Oprah tell the truth. Megan says Marsha lies. Melinda says Milton lies. Murphy says Orla tells the truth. Mackenzie says Bahar tells the truth. Miles says Orlando lies. Odell says Hritik lies. Maria says Mercedes tells the truth. Andree tells the truth. Maybelle says Hritik tells the truth. Vina says exactly one of Helene, Marsha and Megan tell the truth. Margaret says Ozzy lies. Gwenn says Alexis, Moses and Miles all tell the truth. Maynard says Oprah lies. Do Vina, Sherrie, and Sal tell the truth? Your answer should be a list of three words separated by commas. Each of them must be either yes (if the person tells the truth), no (if the person lies), or unknown (if the person's truth cannot be determined). For example, <unknown, no, yes> or <yes, yes, yes>.\"}","display_format":"text","language":"","answer_status":"published","assets":[],"source_url":"https://github.com/google-deepmind/bbeh","history":"initial import","indexing_mode":"index","subproblems":[],"grids":[]}