{"kind":"task","effective_mode":"full","benchmark":{"kind":"benchmark","effective_mode":"full","slug":"bbeh","formal_name":"BIG-Bench Extra Hard (BBEH)","introduction":"BIG-Bench Extra Hard evaluates language-model reasoning through challenging task categories. It offers full and mini distributions; this import contains the full distribution: 4,520 questions across all 23 task categories.","introduction_ja":"","introduction_en":"","category":"Category not supplied","task_count":null,"acquisition_status":"Acquisition status not supplied","official_url":"https://github.com/google-deepmind/bbeh","indexing_mode":"index","profile":{"resources":[],"task_format":"","scoring":"","metric":"","size":"","answer_access":"","license":"","citation":"","maintainer":"","released":"","why_hard":"","related":[],"history":"","construction":"","limitations":"","saturation":"","scores":[]}},"task_id":"8418dee6-ba1e-59fb-b152-0c4ffe517363","task_key":"bbeh~5fweb~5fof~5flies--full--8418dee6-ba1e-59fb-b152-0c4ffe517363","task_revision_id":"1","upstream_id":"","short_description":"In this question, assume each person either always tells the truth or always…","config":"bbeh_web_of_lies","split":"full","body":"{\"input\":\"In this question, assume each person either always tells the truth or always lies. Rashida says Yoland lies. Jim says either all three of Conception, Muriel and Natasha lie, or two of them tell the truth. Sal says exactly one of Gwenn, Mavis and Obadiah tells the truth.Alejandro says exactly two of Kandi, Murphy and Octavius tell the truth. Conception says exactly one of Alejandro, Mehran and Opal tell the truth. Mercedes says Miller tells the truth. Oda says Orin tells the truth. Ottoline says Sanket lies. Yoland says either all three of Vina, Ozzy and Orhan lie, or two of them tell the truth. Mina says Orin lies. Mehran says Opal lies. Murphy says Octavius lies. Mario says Millicent lies. Margaret says Otis lies. Opal says Mehran lies. Orhan says Ozzy tells the truth. Shaunda says Crista lies. Muriel says Natasha tells the truth. Millicent says Mario lies. Jamey says exactly two of Phoebe, Margaret and Otis tell the truth. Vina says Sal lies. Obadiah says Mavis lies. Oran says Miller lies. Kandi tells the truth. Crista says exactly two of Antwan, Mario and Millicent tells the truth. Antwan says exactly two of Rashida, Mackenzie and Miranda tells the truth. Gwenn says Fletcher, Mercedes and Oran all tell the truth. Ozzy says Orhan tells the truth. Natasha says Muriel tells the truth. Miranda says Mackenzie lies. Mavis says Obadiah lies. Fletcher says Jamey tells the truth. Maybelle says Sanket tells the truth. Octavius says Murphy lies. Mackenzie says Miranda lies. Shenna says only one of Shaunda, Maybelle and Ottoline lies. Phoebe says only one of Jim, Oda and Mina lies. Otis says Margaret lies. Do Antwan, Miranda, and Shenna tell the truth? Your answer should be a list of three words separated by commas. Each of them must be either yes (if the person tells the truth), no (if the person lies), or unknown (if the person's truth cannot be determined). For example, <unknown, no, yes> or <yes, yes, yes>.\"}","display_format":"text","language":"","answer_status":"published","assets":[],"source_url":"https://github.com/google-deepmind/bbeh","history":"initial import","indexing_mode":"index","subproblems":[],"grids":[]}