{"kind":"task","effective_mode":"full","benchmark":{"kind":"benchmark","effective_mode":"full","slug":"bbeh","formal_name":"BIG-Bench Extra Hard (BBEH)","introduction":"BIG-Bench Extra Hard evaluates language-model reasoning through challenging task categories. It offers full and mini distributions; this import contains the full distribution: 4,520 questions across all 23 task categories.","introduction_ja":"","introduction_en":"","category":"Category not supplied","task_count":null,"acquisition_status":"Acquisition status not supplied","official_url":"https://github.com/google-deepmind/bbeh","indexing_mode":"index","profile":{"resources":[],"task_format":"","scoring":"","metric":"","size":"","answer_access":"","license":"","citation":"","maintainer":"","released":"","why_hard":"","related":[],"history":"","construction":"","limitations":"","saturation":"","scores":[]}},"task_id":"6539d839-a489-5b04-b6fc-4b34ae855162","task_key":"bbeh~5fweb~5fof~5flies--full--6539d839-a489-5b04-b6fc-4b34ae855162","task_revision_id":"1","upstream_id":"","short_description":"In this question, assume each person either always tells the truth or always…","config":"bbeh_web_of_lies","split":"full","body":"{\"input\":\"In this question, assume each person either always tells the truth or always lies. The person at the barbershop lies. Isabella is at the observatory. The person at the gym says the person at the barbershop tells the truth. The person at the train station thinks their neighbor is telling the truth. The person at the hotel says the person at the train station lies. Olivia is at the gym. The person at the train station says the person at the observatory tells the truth. The person at the observatory saw a firetruck. Nadia is at the beach. The person at the observatory says the person at the city hall tells the truth. The person at the observatory says the person at the beach lies. Liam is at the city hall. Max is at the train station. The person at the school says the person at the observatory tells the truth. The person at the city hall says the person at the barbershop lies. Zara is at the school. Devika is at the hotel. Ryan is at the barbershop. Does the person at the train station tell the truth? Does the person at the observatory tell the truth? Does the person at the city hall tell the truth? Your answer should be a list of three words, yes or no (for example, yes, no, yes).\"}","display_format":"text","language":"","answer_status":"published","assets":[],"source_url":"https://github.com/google-deepmind/bbeh","history":"initial import","indexing_mode":"index","subproblems":[],"grids":[]}