{"kind":"task","effective_mode":"full","benchmark":{"kind":"benchmark","effective_mode":"full","slug":"bbeh","formal_name":"BIG-Bench Extra Hard (BBEH)","introduction":"BIG-Bench Extra Hard evaluates language-model reasoning through challenging task categories. It offers full and mini distributions; this import contains the full distribution: 4,520 questions across all 23 task categories.","introduction_ja":"","introduction_en":"","category":"Category not supplied","task_count":null,"acquisition_status":"Acquisition status not supplied","official_url":"https://github.com/google-deepmind/bbeh","indexing_mode":"index","profile":{"resources":[],"task_format":"","scoring":"","metric":"","size":"","answer_access":"","license":"","citation":"","maintainer":"","released":"","why_hard":"","related":[],"history":"","construction":"","limitations":"","saturation":"","scores":[]}},"task_id":"dcd1e9f9-5c42-530a-8c5a-baa14fb9b64c","task_key":"bbeh~5fweb~5fof~5flies--full--dcd1e9f9-5c42-530a-8c5a-baa14fb9b64c","task_revision_id":"1","upstream_id":"","short_description":"In this question, assume each person either always tells the truth or always…","config":"bbeh_web_of_lies","split":"full","body":"{\"input\":\"In this question, assume each person either always tells the truth or always lies. Liam is at the botanical garden. The person at the hotel says the person at the airport lies. The person at the barbershop says the person at the botanical garden tells the truth. Soren is at the restaurant. The person at the zoo says the person at the art gallery tells the truth. The person at the airport says the person at the skate park lies. The person at the aquarium tells the truth. The person at the restaurant tells the truth. Ayaan is at the barbershop. Anika is at the art gallery. Kehinde is at the museum. Charlie is at the school. The person at the observatory says the person at the zoo lies. The person at the botanical garden says the person at the airport tells the truth. The person at the zoo tells the truth. Zara is at the aquarium. The person at the airport says the person at the observatory tells the truth. Devika is at the airport. The person at the museum tells the truth. Ethan is at the campground. Luna is at the skate park. The person at the campground says the person at the zoo tells the truth. Theo is at the observatory. Farid is at the zoo. Grace is at the hotel. The person at the school tells the truth. Does the person at the botanical garden tell the truth? Does the person at the airport tell the truth? Does the person at the observatory tell the truth? Your answer should be a list of three words, yes or no (for example, yes, no, yes).\"}","display_format":"text","language":"","answer_status":"published","assets":[],"source_url":"https://github.com/google-deepmind/bbeh","history":"initial import","indexing_mode":"index","subproblems":[],"grids":[]}