{"kind":"task","effective_mode":"full","benchmark":{"kind":"benchmark","effective_mode":"full","slug":"bbeh","formal_name":"BIG-Bench Extra Hard (BBEH)","introduction":"BIG-Bench Extra Hard evaluates language-model reasoning through challenging task categories. It offers full and mini distributions; this import contains the full distribution: 4,520 questions across all 23 task categories.","introduction_ja":"","introduction_en":"","category":"Category not supplied","task_count":null,"acquisition_status":"Acquisition status not supplied","official_url":"https://github.com/google-deepmind/bbeh","indexing_mode":"index","profile":{"resources":[],"task_format":"","scoring":"","metric":"","size":"","answer_access":"","license":"","citation":"","maintainer":"","released":"","why_hard":"","related":[],"history":"","construction":"","limitations":"","saturation":"","scores":[]}},"task_id":"8e448a29-b561-5bdf-86df-ebd859575a85","task_key":"bbeh~5fweb~5fof~5flies--full--8e448a29-b561-5bdf-86df-ebd859575a85","task_revision_id":"1","upstream_id":"","short_description":"In this question, assume each person either always tells the truth or always…","config":"bbeh_web_of_lies","split":"full","body":"{\"input\":\"In this question, assume each person either always tells the truth or always lies. Megan says Oriana tells the truth. Tamika says either exactly two of Lorine, Millie and Bahar tell the truth or none of them. Ollie says Marsha tells the truth. Mary says Michael lies. Maverick says Octavius tells the truth. Octavius says Maverick tells the truth. Gwenn says only one of Sal, Melody and Olivia lies. Nicole says Oleg tells the truth. Jaymie says Tamika lies. Lorine says only one of Vina, Orin and Monica lies. Ofelia says Mickey tells the truth. Octavio says Ophelia tells the truth. Mustafa says Miguel tells the truth. Mona says Oriana lies. Shenna says only one of Delfina, Mustafa and Nishanth lies. Alexis says Leda tells the truth. Delfina says Jaymie tells the truth. Ophelia says Octavio tells the truth. Antwan tells the truth. Helene says either exactly two of Shenna, Odin and Matthew tell the truth or none of them. Olivia says Maximilian lies. Sal says exactly one of Vernell, Michael and Mary tells the truth.Ryan says Christie tells the truth. Monica says Marina lies. Odin says Matthew tells the truth. Orin says Marina tells the truth. Nishanth says Miguel lies. Vernell says either exactly two of Delbert, Nicole and Oleg tell the truth or none of them. Vina says either all three of Alexis, Octavio and Ophelia tell the truth or only one of them. Leda says either all three of Antwan, Marsha and Ollie tell the truth or only one of them. Melody says Maximilian tells the truth. Delbert says either all three of Helene, Ofelia and Mickey lie, or two of them tell the truth. Shalonda says Crista, Megan and Mona all tell the truth. Michael says Mary lies. Christie says either all three of Shalonda, Octavius and Maverick lie, or two of them tell the truth. Mickey says Ofelia tells the truth. Matthew says Odin tells the truth. Bahar says Millie tells the truth. Marsha says Ollie tells the truth. Millie says Bahar tells the truth. Crista says Gwenn tells the truth. Oleg says Nicole tells the truth. Do Vernell, Mona, and Ryan tell the truth? Your answer should be a list of three words separated by commas. Each of them must be either yes (if the person tells the truth), no (if the person lies), or unknown (if the person's truth cannot be determined). For example, <unknown, no, yes> or <yes, yes, yes>.\"}","display_format":"text","language":"","answer_status":"published","assets":[],"source_url":"https://github.com/google-deepmind/bbeh","history":"initial import","indexing_mode":"index","subproblems":[],"grids":[]}