{"kind":"task","effective_mode":"full","benchmark":{"kind":"benchmark","effective_mode":"full","slug":"bbeh","formal_name":"BIG-Bench Extra Hard (BBEH)","introduction":"BIG-Bench Extra Hard evaluates language-model reasoning through challenging task categories. It offers full and mini distributions; this import contains the full distribution: 4,520 questions across all 23 task categories.","introduction_ja":"","introduction_en":"","category":"Category not supplied","task_count":null,"acquisition_status":"Acquisition status not supplied","official_url":"https://github.com/google-deepmind/bbeh","indexing_mode":"index","profile":{"resources":[],"task_format":"","scoring":"","metric":"","size":"","answer_access":"","license":"","citation":"","maintainer":"","released":"","why_hard":"","related":[],"history":"","construction":"","limitations":"","saturation":"","scores":[]}},"task_id":"e9f04e1f-dd88-5d6b-afe7-2d4206f65ca7","task_key":"bbeh~5fweb~5fof~5flies--full--e9f04e1f-dd88-5d6b-afe7-2d4206f65ca7","task_revision_id":"1","upstream_id":"","short_description":"In this question, assume each person either always tells the truth or always…","config":"bbeh_web_of_lies","split":"full","body":"{\"input\":\"In this question, assume each person either always tells the truth or always lies. The person at the hotel says the person at the barbershop tells the truth. Tala is at the hotel. The person at the city hall says the person at the farm tells the truth. Anika is at the train station. The person at the planetarium says the person at the airport tells the truth. The person at the botanical garden tells the truth. The person at the train station says the person at the city hall lies. Nadia is at the airport. Quan is at the botanical garden. Priya is at the planetarium. Liam is at the farm. The person at the planetarium says the person at the city hall lies. Nia is at the city hall. The person at the city hall says the person at the botanical garden lies. Jake is at the barbershop. Olivia is at the vineyard. The person at the barbershop says the person at the planetarium tells the truth. Maya is at the library. The person at the planetarium says the person at the library tells the truth. The person at the vineyard says the person at the city hall lies. Does the person at the hotel tell the truth? Does the person at the barbershop tell the truth? Does the person at the planetarium tell the truth? Your answer should be a list of three words, yes or no (for example, yes, no, yes).\"}","display_format":"text","language":"","answer_status":"published","assets":[],"source_url":"https://github.com/google-deepmind/bbeh","history":"initial import","indexing_mode":"index","subproblems":[],"grids":[]}