{"kind":"task","effective_mode":"full","benchmark":{"kind":"benchmark","effective_mode":"full","slug":"bbeh","formal_name":"BIG-Bench Extra Hard (BBEH)","introduction":"BIG-Bench Extra Hard evaluates language-model reasoning through challenging task categories. It offers full and mini distributions; this import contains the full distribution: 4,520 questions across all 23 task categories.","introduction_ja":"","introduction_en":"","category":"Category not supplied","task_count":null,"acquisition_status":"Acquisition status not supplied","official_url":"https://github.com/google-deepmind/bbeh","indexing_mode":"index","profile":{"resources":[],"task_format":"","scoring":"","metric":"","size":"","answer_access":"","license":"","citation":"","maintainer":"","released":"","why_hard":"","related":[],"history":"","construction":"","limitations":"","saturation":"","scores":[]}},"task_id":"b43a0230-c7b5-50bc-aa0d-7488b8fec271","task_key":"bbeh~5fweb~5fof~5flies--full--b43a0230-c7b5-50bc-aa0d-7488b8fec271","task_revision_id":"1","upstream_id":"","short_description":"In this question, assume each person either always tells the truth or always…","config":"bbeh_web_of_lies","split":"full","body":"{\"input\":\"In this question, assume each person either always tells the truth or always lies. Nadia is at the restaurant. Tala is at the planetarium. The person at the restaurant says the person at the vineyard tells the truth. Elowen is at the vineyard. The person at the zoo says the person at the library lies. Kehinde is at the campground. Zara is at the library. Mateo is at the aquarium. The person at the botanical garden says the person at the bowling alley lies. The person at the planetarium says the person at the gym lies. The person at the campground lies. The person at the aquarium says the person at the planetarium lies. The person at the botanical garden says the person at the restaurant tells the truth. Luna is at the zoo. The person at the library tells the truth. Isabella is at the bowling alley. The person at the planetarium says the person at the library lies. Soren is at the gym. The person at the beach says the person at the planetarium tells the truth. The person at the city hall says the person at the vineyard lies. Hiroshi is at the city hall. Priya is at the botanical garden. Nia is at the beach. The person at the vineyard says the person at the planetarium tells the truth. Does the person at the botanical garden tell the truth? Does the person at the restaurant tell the truth? Does the person at the vineyard tell the truth? Your answer should be a list of three words, yes or no (for example, yes, no, yes).\"}","display_format":"text","language":"","answer_status":"published","assets":[],"source_url":"https://github.com/google-deepmind/bbeh","history":"initial import","indexing_mode":"index","subproblems":[],"grids":[]}