{"kind":"task","effective_mode":"full","benchmark":{"kind":"benchmark","effective_mode":"full","slug":"bbeh","formal_name":"BIG-Bench Extra Hard (BBEH)","introduction":"BIG-Bench Extra Hard evaluates language-model reasoning through challenging task categories. It offers full and mini distributions; this import contains the full distribution: 4,520 questions across all 23 task categories.","introduction_ja":"","introduction_en":"","category":"Category not supplied","task_count":null,"acquisition_status":"Acquisition status not supplied","official_url":"https://github.com/google-deepmind/bbeh","indexing_mode":"index","profile":{"resources":[],"task_format":"","scoring":"","metric":"","size":"","answer_access":"","license":"","citation":"","maintainer":"","released":"","why_hard":"","related":[],"history":"","construction":"","limitations":"","saturation":"","scores":[]}},"task_id":"1eae172f-12af-59d2-a0ef-dffa12666f2f","task_key":"bbeh~5fweb~5fof~5flies--full--1eae172f-12af-59d2-a0ef-dffa12666f2f","task_revision_id":"1","upstream_id":"","short_description":"In this question, assume each person either always tells the truth or always…","config":"bbeh_web_of_lies","split":"full","body":"{\"input\":\"In this question, assume each person either always tells the truth or always lies. Kate says Mandy tells the truth. Mateo says Mina lies. Antwan says only one of Kandi, Melinda and Octavian tells the truth. Uri says Odell lies. Crista says either all three of Vernell, Kate and Mandy tell the truth or only one of them. Oden says Marcus lies. Mary says Muhammad tells the truth. Helene lies. Orhan says Hritik lies. Osborne says Octavio tells the truth. Tamika says Phoebe, Melissa and Mehran all tell the truth. Octavian says Matteo lies. Phoebe says exactly two of Antwan, Mateo and Mina tell the truth. Mandy says Kate tells the truth. Melissa says Morgan tells the truth. Meadow says Hritik tells the truth. Gwenn says Crista tells the truth. Oakley says Octavio lies. Vernell says Shalonda lies. Elanor says Gwenn lies. Orla says Odell tells the truth. Kandi says only one of Dallas, Mary and Melanie lies. Jaymie says exactly one of Christie, Mohamed and Disha tell the truth. Dallas says only one of Amberly, Meadow and Orhan lies. Melinda says Matteo tells the truth. Otto says Osmond lies. Mina says Mateo lies. Melanie says Muhammad lies. Osmond says Otto lies. Disha says Mohamed lies. Christie says Yoland tells the truth. Odysseus says Marcus tells the truth. Amberly says only one of Ka, Orla and Uri lies. Mohamed says Disha lies. Mehran says Morgan lies. Shalonda says exactly two of Helene, Otto and Osmond tells the truth. Ka says only one of Jaymie, Osborne and Oakley lies. Yoland says only one of Elanor, Odysseus and Oden tells the truth. Do Dallas, Oden, and Tamika tell the truth? Your answer should be a list of three words separated by commas. Each of them must be either yes (if the person tells the truth), no (if the person lies), or unknown (if the person's truth cannot be determined). For example, <unknown, no, yes> or <yes, yes, yes>.\"}","display_format":"text","language":"","answer_status":"published","assets":[],"source_url":"https://github.com/google-deepmind/bbeh","history":"initial import","indexing_mode":"index","subproblems":[],"grids":[]}