{"kind":"task","effective_mode":"full","benchmark":{"kind":"benchmark","effective_mode":"full","slug":"bbeh","formal_name":"BIG-Bench Extra Hard (BBEH)","introduction":"BIG-Bench Extra Hard evaluates language-model reasoning through challenging task categories. It offers full and mini distributions; this import contains the full distribution: 4,520 questions across all 23 task categories.","introduction_ja":"","introduction_en":"","category":"Category not supplied","task_count":null,"acquisition_status":"Acquisition status not supplied","official_url":"https://github.com/google-deepmind/bbeh","indexing_mode":"index","profile":{"resources":[],"task_format":"","scoring":"","metric":"","size":"","answer_access":"","license":"","citation":"","maintainer":"","released":"","why_hard":"","related":[],"history":"","construction":"","limitations":"","saturation":"","scores":[]}},"task_id":"89035192-1afd-5459-966f-706190f71563","task_key":"bbeh~5fweb~5fof~5flies--full--89035192-1afd-5459-966f-706190f71563","task_revision_id":"1","upstream_id":"","short_description":"In this question, assume each person either always tells the truth or always…","config":"bbeh_web_of_lies","split":"full","body":"{\"input\":\"In this question, assume each person either always tells the truth or always lies. Emily is at the skate park. Priya is at the farm. Soren is at the train station. The person at the farm says the person at the beach lies. The person at the museum lies. Hiroshi is at the observatory. The person at the zoo saw a firetruck. The person at the shopping mall tells the truth. Devika is at the zoo. The person at the theater says the person at the train station lies. Zara is at the city hall. Beatriz is at the cafe. Nia is at the museum. Isabella is at the shopping mall. The person at the observatory says the person at the farm lies. Ethan is at the beach. The person at the cafe says the person at the train station lies. The person at the zoo lies. The person at the train station says the person at the zoo tells the truth. Farid is at the restaurant. The person at the restaurant says the person at the zoo tells the truth. The person at the city hall tells the truth. The person at the skate park says the person at the farm tells the truth. Tala is at the theater. The person at the farm says the person at the train station tells the truth. Does the person at the farm tell the truth? Does the person at the train station tell the truth? Does the person at the zoo tell the truth? Your answer should be a list of three words, yes or no (for example, yes, no, yes).\"}","display_format":"text","language":"","answer_status":"published","assets":[],"source_url":"https://github.com/google-deepmind/bbeh","history":"initial import","indexing_mode":"index","subproblems":[],"grids":[]}