{"kind":"task","effective_mode":"full","benchmark":{"kind":"benchmark","effective_mode":"full","slug":"bbeh","formal_name":"BIG-Bench Extra Hard (BBEH)","introduction":"BIG-Bench Extra Hard evaluates language-model reasoning through challenging task categories. It offers full and mini distributions; this import contains the full distribution: 4,520 questions across all 23 task categories.","introduction_ja":"","introduction_en":"","category":"Category not supplied","task_count":null,"acquisition_status":"Acquisition status not supplied","official_url":"https://github.com/google-deepmind/bbeh","indexing_mode":"index","profile":{"resources":[],"task_format":"","scoring":"","metric":"","size":"","answer_access":"","license":"","citation":"","maintainer":"","released":"","why_hard":"","related":[],"history":"","construction":"","limitations":"","saturation":"","scores":[]}},"task_id":"0ef0664b-f1cf-54a9-bc73-76c89b62ea45","task_key":"bbeh~5fweb~5fof~5flies--full--0ef0664b-f1cf-54a9-bc73-76c89b62ea45","task_revision_id":"1","upstream_id":"","short_description":"In this question, assume each person either always tells the truth or always…","config":"bbeh_web_of_lies","split":"full","body":"{\"input\":\"In this question, assume each person either always tells the truth or always lies. The person at the art gallery says the person at the bookstore tells the truth. Zara is at the barbershop. Quan is at the city hall. The person at the city hall tells the truth. Tala is at the bookstore. Beatriz is at the restaurant. The person at the barbershop says the person at the planetarium lies. The person at the zoo says the person at the art gallery tells the truth. The person at the art gallery thinks their friend is lying. The person at the bookstore tells the truth. Farid is at the ice skating rink. Ethan is at the zoo. Mateo is at the hotel. The person at the restaurant lies. The person at the ice skating rink lies. The person at the planetarium says the person at the zoo lies. Luna is at the art gallery. The person at the hotel lies. Emily is at the planetarium. Does the person at the barbershop tell the truth? Does the person at the planetarium tell the truth? Does the person at the zoo tell the truth? Your answer should be a list of three words, yes or no (for example, yes, no, yes).\"}","display_format":"text","language":"","answer_status":"published","assets":[],"source_url":"https://github.com/google-deepmind/bbeh","history":"initial import","indexing_mode":"index","subproblems":[],"grids":[]}