{"kind":"task","effective_mode":"full","benchmark":{"kind":"benchmark","effective_mode":"full","slug":"bbeh","formal_name":"BIG-Bench Extra Hard (BBEH)","introduction":"BIG-Bench Extra Hard evaluates language-model reasoning through challenging task categories. It offers full and mini distributions; this import contains the full distribution: 4,520 questions across all 23 task categories.","introduction_ja":"","introduction_en":"","category":"Category not supplied","task_count":null,"acquisition_status":"Acquisition status not supplied","official_url":"https://github.com/google-deepmind/bbeh","indexing_mode":"index","profile":{"resources":[],"task_format":"","scoring":"","metric":"","size":"","answer_access":"","license":"","citation":"","maintainer":"","released":"","why_hard":"","related":[],"history":"","construction":"","limitations":"","saturation":"","scores":[]}},"task_id":"11660c54-282f-5901-a61f-9220ea3128a6","task_key":"bbeh~5fweb~5fof~5flies--full--11660c54-282f-5901-a61f-9220ea3128a6","task_revision_id":"1","upstream_id":"","short_description":"In this question, assume each person either always tells the truth or always…","config":"bbeh_web_of_lies","split":"full","body":"{\"input\":\"In this question, assume each person either always tells the truth or always lies. Mustafa says Oakleigh lies. Maverick says Oakley lies. Oleg says Mickey lies. Madeleine says Melissa lies. Matthew says Milton tells the truth. Bernita says exactly two of Jaymie, Ozzy and Oda tell the truth. Sal tells the truth. Marshall says Mitchell lies. Kristian says either exactly two of Lorine, Matthew and Milton tell the truth or none of them. Otto says Ocean tells the truth. Ottoline says Mercedes tells the truth. Fletcher says exactly one of Vernell, Mona and Oona tells the truth.Jamey says only one of Vina, Ophelia and Mandy tells the truth. Ofelia says Oswin lies. Alejandro says either exactly two of Sal, Oriana and Michelle tell the truth or none of them. Oprah says Mindy tells the truth. Melissa says Madeleine lies. Micah says Mehran lies. Delfina says either exactly one of Gwenn, Ottoline and Mercedes tells the truth, or all three of them. Mackenzie says Malcolm tells the truth. Inga says either all three of Rashida, Mindy and Oprah lie, or two of them tell the truth. Oden says Oswin tells the truth. Delbert says only one of Alejandro, Morris and Uri tells the truth. Oona says Mona lies. Ozzy says Oda lies. Mandy says Mariah lies. Michelle says Oriana tells the truth. Uri says Kate lies. Orla says Oakleigh tells the truth. Crista says exactly two of Ryan, Ollie and Miguel tell the truth. Mehran says Micah lies. Morris says Kate tells the truth. Mona says Oona lies. Gwenn says exactly one of Inga, Melissa and Madeleine tell the truth. Vina says either exactly two of Crista, Owyn and Max tell the truth or none of them. Orlando says Ocean lies. Milton says Matthew tells the truth. Mitchell says Marshall lies. Shenna says exactly two of Fidel, Mickey and Oleg tell the truth. Owyn says Max tells the truth. Fidel says Shalonda tells the truth. Sherrie says Delbert tells the truth. Ryan says only one of Amberly, Otto and Orlando lies. Orson says Natasha lies. Vernell says Christie lies. Mercedes says Ottoline tells the truth. Max says Owyn tells the truth. Oriana says Michelle tells the truth. Oda says Ozzy lies. Jaymie says exactly two of Sherrie, Micah and Mehran tell the truth. Natasha says Orson lies. Oakley says Maverick lies. Miguel says Ollie lies. Lorine says exactly one of Delfina, Oakley and Maverick tells the truth.Ollie says Miguel lies. Ozzie says Malcolm lies. Mickey says Oleg lies. Amberly says only one of Shenna, Orla and Mustafa lies. Mindy says Oprah tells the truth. Ophelia says Mariah tells the truth. Christie says only one of Sima, Mackenzie and Ozzie tells the truth. Sima says Bernita, Oden and Ofelia all tell the truth. Shalonda says exactly two of Fletcher, Marshall and Mitchell tell the truth. Rashida says exactly one of Jamey, Natasha and Orson tell the truth. Do Delfina, Ophelia, and Kristian tell the truth? Your answer should be a list of three words separated by commas. Each of them must be either yes (if the person tells the truth), no (if the person lies), or unknown (if the person's truth cannot be determined). For example, <unknown, no, yes> or <yes, yes, yes>.\"}","display_format":"text","language":"","answer_status":"published","assets":[],"source_url":"https://github.com/google-deepmind/bbeh","history":"initial import","indexing_mode":"index","subproblems":[],"grids":[]}