{"kind":"task","effective_mode":"full","benchmark":{"kind":"benchmark","effective_mode":"full","slug":"bbeh","formal_name":"BIG-Bench Extra Hard (BBEH)","introduction":"BIG-Bench Extra Hard evaluates language-model reasoning through challenging task categories. It offers full and mini distributions; this import contains the full distribution: 4,520 questions across all 23 task categories.","introduction_ja":"","introduction_en":"","category":"Category not supplied","task_count":null,"acquisition_status":"Acquisition status not supplied","official_url":"https://github.com/google-deepmind/bbeh","indexing_mode":"index","profile":{"resources":[],"task_format":"","scoring":"","metric":"","size":"","answer_access":"","license":"","citation":"","maintainer":"","released":"","why_hard":"","related":[],"history":"","construction":"","limitations":"","saturation":"","scores":[]}},"task_id":"cabd0372-7503-5d97-9170-8971932018ae","task_key":"bbeh~5fweb~5fof~5flies--full--cabd0372-7503-5d97-9170-8971932018ae","task_revision_id":"1","upstream_id":"","short_description":"In this question, assume each person either always tells the truth or always…","config":"bbeh_web_of_lies","split":"full","body":"{\"input\":\"In this question, assume each person either always tells the truth or always lies. The person at the ice skating rink says the person at the city hall tells the truth. Devika is at the library. The person at the bookstore lies. Olivia is at the bookstore. The person at the ice skating rink says the person at the park tells the truth. Ayaan is at the park. The person at the aquarium says the person at the vineyard tells the truth. Maya is at the art gallery. The person at the art gallery says the person at the ice skating rink tells the truth. The person at the movie theater says the person at the museum lies. The person at the movie theater says the person at the vineyard lies. The person at the library says the person at the planetarium tells the truth. The person at the ice skating rink says the person at the shopping mall lies. Grace is at the movie theater. Jaxon is at the vineyard. The person at the shopping mall says the person at the library lies. Isabella is at the aquarium. Elowen is at the shopping mall. Liam is at the ice skating rink. Farid is at the museum. Emily is at the planetarium. Priya is at the city hall. The person at the vineyard lies. Soren is at the school. The person at the planetarium says the person at the movie theater tells the truth. The person at the school says the person at the library tells the truth. Does the person at the art gallery tell the truth? Does the person at the ice skating rink tell the truth? Does the person at the shopping mall tell the truth? Your answer should be a list of three words, yes or no (for example, yes, no, yes).\"}","display_format":"text","language":"","answer_status":"published","assets":[],"source_url":"https://github.com/google-deepmind/bbeh","history":"initial import","indexing_mode":"index","subproblems":[],"grids":[]}