{"kind":"task","effective_mode":"full","benchmark":{"kind":"benchmark","effective_mode":"full","slug":"bbeh","formal_name":"BIG-Bench Extra Hard (BBEH)","introduction":"BIG-Bench Extra Hard evaluates language-model reasoning through challenging task categories. It offers full and mini distributions; this import contains the full distribution: 4,520 questions across all 23 task categories.","introduction_ja":"","introduction_en":"","category":"Category not supplied","task_count":null,"acquisition_status":"Acquisition status not supplied","official_url":"https://github.com/google-deepmind/bbeh","indexing_mode":"index","profile":{"resources":[],"task_format":"","scoring":"","metric":"","size":"","answer_access":"","license":"","citation":"","maintainer":"","released":"","why_hard":"","related":[],"history":"","construction":"","limitations":"","saturation":"","scores":[]}},"task_id":"311751bb-cf3c-554c-b469-692e01dc2c2e","task_key":"bbeh~5fweb~5fof~5flies--full--311751bb-cf3c-554c-b469-692e01dc2c2e","task_revision_id":"1","upstream_id":"","short_description":"In this question, assume each person either always tells the truth or always…","config":"bbeh_web_of_lies","split":"full","body":"{\"input\":\"In this question, assume each person either always tells the truth or always lies. The person at the hotel says the person at the amusement park tells the truth. Ryan is at the botanical garden. The person at the art gallery says the person at the beach tells the truth. The person at the theater says the person at the hotel lies. The person at the hotel says the person at the botanical garden tells the truth. The person at the campground says the person at the city hall lies. Kehinde is at the gym. Maya is at the aquarium. Isabella is at the campground. The person at the city hall says the person at the theater tells the truth. Jaxon is at the cafe. Mateo is at the ice skating rink. The person at the ice skating rink says the person at the gym lies. Max is at the hotel. The person at the gym says the person at the vineyard tells the truth. The person at the gym says the person at the aquarium lies. Luna is at the beach. Hiroshi is at the art gallery. The person at the gym says the person at the campground lies. The person at the cafe lies. Liam is at the amusement park. Olivia is at the theater. The person at the beach tells the truth. Charlie is at the city hall. The person at the botanical garden says the person at the art gallery tells the truth. Nia is at the vineyard. Does the person at the ice skating rink tell the truth? Does the person at the gym tell the truth? Does the person at the campground tell the truth? Your answer should be a list of three words, yes or no (for example, yes, no, yes).\"}","display_format":"text","language":"","answer_status":"published","assets":[],"source_url":"https://github.com/google-deepmind/bbeh","history":"initial import","indexing_mode":"index","subproblems":[],"grids":[]}