{"kind":"task","effective_mode":"full","benchmark":{"kind":"benchmark","effective_mode":"full","slug":"bbeh","formal_name":"BIG-Bench Extra Hard (BBEH)","introduction":"BIG-Bench Extra Hard evaluates language-model reasoning through challenging task categories. It offers full and mini distributions; this import contains the full distribution: 4,520 questions across all 23 task categories.","introduction_ja":"","introduction_en":"","category":"Category not supplied","task_count":null,"acquisition_status":"Acquisition status not supplied","official_url":"https://github.com/google-deepmind/bbeh","indexing_mode":"index","profile":{"resources":[],"task_format":"","scoring":"","metric":"","size":"","answer_access":"","license":"","citation":"","maintainer":"","released":"","why_hard":"","related":[],"history":"","construction":"","limitations":"","saturation":"","scores":[]}},"task_id":"9820ac53-f932-560a-bd58-3483e20674df","task_key":"bbeh~5fweb~5fof~5flies--full--9820ac53-f932-560a-bd58-3483e20674df","task_revision_id":"1","upstream_id":"","short_description":"In this question, assume each person either always tells the truth or always…","config":"bbeh_web_of_lies","split":"full","body":"{\"input\":\"In this question, assume each person either always tells the truth or always lies. The person at the gym says the person at the library lies. The person at the barbershop thinks their friend is lying. Nia is at the gym. The person at the art gallery tells the truth. The person at the barbershop says the person at the library lies. Jaxon is at the barbershop. Devika is at the beach. The person at the ice skating rink tells the truth. The person at the skate park thinks their neighbor is telling the truth. Anika is at the campground. The person at the skate park says the person at the campground tells the truth. The person at the park says the person at the barbershop lies. Tala is at the ice skating rink. Liam is at the park. Quan is at the library. The person at the campground says the person at the barbershop lies. Hiroshi is at the skate park. The person at the beach says the person at the barbershop lies. Ayaan is at the art gallery. The person at the library says the person at the ice skating rink lies. The person at the barbershop thinks their neighbor is telling the truth. Does the person at the skate park tell the truth? Does the person at the campground tell the truth? Does the person at the barbershop tell the truth? Your answer should be a list of three words, yes or no (for example, yes, no, yes).\"}","display_format":"text","language":"","answer_status":"published","assets":[],"source_url":"https://github.com/google-deepmind/bbeh","history":"initial import","indexing_mode":"index","subproblems":[],"grids":[]}