{"kind":"task","effective_mode":"full","benchmark":{"kind":"benchmark","effective_mode":"full","slug":"mmlu-pro","formal_name":"MMLU-Pro","introduction":"MMLU-Pro rebuilds MMLU with ten answer options instead of four and removes items that no longer separate models. Questions span fourteen academic and professional subjects.","introduction_ja":"","introduction_en":"","category":"Category not supplied","task_count":null,"acquisition_status":"Acquisition status not supplied","official_url":"https://huggingface.co/datasets/TIGER-Lab/MMLU-Pro","indexing_mode":"index","profile":{"resources":[],"task_format":"","scoring":"","metric":"","size":"","answer_access":"","license":"","citation":"","maintainer":"","released":"","why_hard":"","related":[],"history":"","construction":"","limitations":"","saturation":"","scores":[]}},"task_id":"7f0777cb-700d-5af7-b544-2de82acd6865","task_key":"default--test--1457","task_revision_id":"1","upstream_id":"1457","short_description":"A grocery store is sued by a customer for damages for injuries allegedly…","config":"default","split":"test","body":"{\"category\":\"law\",\"options\":\"['inadmissible, due to the possibility of bias.', 'admissible, as a first-hand witness account of the event.', 'inadmissible, as it lacks direct connection to the case.', \\\"admissible, as evidence of the store's negligence.\\\", 'inadmissible, on the grounds of public policy considerations.', 'inadmissible, as it is hearsay evidence.', 'admissible as a tacit admission of a party opponent.', 'inadmissible, on the grounds of irrelevancy.', 'admissible as a commonsense impression for which a lay opinion is entirely proper.']\",\"question\":\"A grocery store is sued by a customer for damages for injuries allegedly suffered to the face and forearms, which resulted from his collision with the store's automatic doors. The customer contended that at the time of the mishap, the doors were programmed to swing outward, taking would-be customers entering the store, by surprise. At trial, the customer testified to both the nature of his injuries and the events leading up to their occurrence. The grocery store denied that the doors to its store were in any way unsafe or that the customer's injuries were caused by his encounter with the doors. The customer's sister testified at trial that she accompanied her brother to the store the day of the accident and witnessed the outward swinging doors, which struck him. She further alleged that when she returned to the store the following week, the doors had been re-programmed to swing inward. The trial judge should rule the sister's testimony\",\"src\":\"ori_mmlu-professional_law\"}","display_format":"text","language":"","answer_status":"published","assets":[],"source_url":"https://huggingface.co/datasets/TIGER-Lab/MMLU-Pro","history":"initial import","indexing_mode":"index","subproblems":[],"grids":[]}