{"kind":"task","effective_mode":"full","benchmark":{"kind":"benchmark","effective_mode":"full","slug":"bbeh","formal_name":"BIG-Bench Extra Hard (BBEH)","introduction":"BIG-Bench Extra Hard evaluates language-model reasoning through challenging task categories. It offers full and mini distributions; this import contains the full distribution: 4,520 questions across all 23 task categories.","introduction_ja":"","introduction_en":"","category":"Category not supplied","task_count":null,"acquisition_status":"Acquisition status not supplied","official_url":"https://github.com/google-deepmind/bbeh","indexing_mode":"index","profile":{"resources":[],"task_format":"","scoring":"","metric":"","size":"","answer_access":"","license":"","citation":"","maintainer":"","released":"","why_hard":"","related":[],"history":"","construction":"","limitations":"","saturation":"","scores":[]}},"task_id":"97273954-a7c7-5ffe-af80-aaef44840dd9","task_key":"bbeh~5fweb~5fof~5flies--full--97273954-a7c7-5ffe-af80-aaef44840dd9","task_revision_id":"1","upstream_id":"","short_description":"In this question, assume each person either always tells the truth or always…","config":"bbeh_web_of_lies","split":"full","body":"{\"input\":\"In this question, assume each person either always tells the truth or always lies. Orla says Maybelle tells the truth. Mia says Natasha lies. Mickey says Margaret tells the truth. Amberly says either all three of Delbert, Margaret and Mickey lie, or two of them tell the truth. Misty says Osian tells the truth. Matthew says Osian lies. Octavian says Maverick lies. Octavius says Muhammad tells the truth. Mackenzie says Oakleigh lies. Raymond says Vina lies. Olaf says Olga tells the truth. Mariana says Natasha tells the truth. Muhammad says Octavius tells the truth. Osbert says Ozzie lies. Olga says Olaf tells the truth. Leda says either all three of Jaymie, Olga and Olaf tell the truth or only one of them. Elanor says exactly two of Kandi, Melody and Oswin tell the truth. Maxwell says Molly tells the truth. Antwan says only one of Alexis, Owyn and Maynard tells the truth. Delbert says exactly one of Leda, Maverick and Octavian tell the truth. Yoland says either exactly two of Fletcher, Onyx and Millicent tell the truth or none of them. Maynard says Oprah lies. Ozzie says Osbert lies. Maybelle says Orla tells the truth. Maverick says Octavian lies. Margaret says Mickey tells the truth. Tamika says either all three of Antwan, Mehran and Olympia tell the truth or only one of them. Kandi says Yoland lies. Oakleigh says Mackenzie lies. Oswin says Melody lies. Shalonda says exactly two of Elanor, Mackenzie and Oakleigh tell the truth. Olympia says Mehran tells the truth. Alexis says either exactly two of Sal, Orla and Maybelle tell the truth or none of them. Fletcher says exactly two of Teressa, Osbert and Ozzie tell the truth. Millicent says Onyx tells the truth. Teressa tells the truth. Owyn says Oprah tells the truth. Melody says Oswin lies. Helene says Shalonda, Misty and Matthew all tell the truth. Michaela says Molly lies. Onyx says Millicent tells the truth. Vina says either exactly one of Helene, Muhammad and Octavius tells the truth, or all three of them. Sal says only one of Raymond, Maxwell and Michaela lies. Jaymie says only one of Tamika, Mariana and Mia lies. Mehran says Olympia tells the truth. Do Alexis, Antwan, and Amberly tell the truth? Your answer should be a list of three words separated by commas. Each of them must be either yes (if the person tells the truth), no (if the person lies), or unknown (if the person's truth cannot be determined). For example, <unknown, no, yes> or <yes, yes, yes>.\"}","display_format":"text","language":"","answer_status":"published","assets":[],"source_url":"https://github.com/google-deepmind/bbeh","history":"initial import","indexing_mode":"index","subproblems":[],"grids":[]}