{"kind":"task","effective_mode":"full","benchmark":{"kind":"benchmark","effective_mode":"full","slug":"mmlu-pro","formal_name":"MMLU-Pro","introduction":"MMLU-Pro rebuilds MMLU with ten answer options instead of four and removes items that no longer separate models. Questions span fourteen academic and professional subjects.","introduction_ja":"","introduction_en":"","category":"Category not supplied","task_count":null,"acquisition_status":"Acquisition status not supplied","official_url":"https://huggingface.co/datasets/TIGER-Lab/MMLU-Pro","indexing_mode":"index","profile":{"resources":[],"task_format":"","scoring":"","metric":"","size":"","answer_access":"","license":"","citation":"","maintainer":"","released":"","why_hard":"","related":[],"history":"","construction":"","limitations":"","saturation":"","scores":[]}},"task_id":"57289e5d-f026-51c0-9764-b045c5000a7c","task_key":"default--test--1637","task_revision_id":"1","upstream_id":"1637","short_description":"A seat in a movie theater collapsed, causing the customer sitting in that seat…","config":"default","split":"test","body":"{\"category\":\"law\",\"options\":\"['not recover, because the customer assumed the risk by choosing to sit in the seat.', 'not recover, unless the customer had informed the theater about the faulty seat before the accident.', 'recover, only if the theater had received previous complaints about the seat.', 'recover, if the theater failed to put up a warning sign about the faulty seat.', 'recover, only if the theater had prior knowledge of the dangerous condition of the seat.', \\\"recover, only if the customer can prove the theater's negligence in maintaining the seat.\\\", 'not recover, unless the theater failed to make a reasonable inspection of the seats for the safety of its customers.', 'not recover, because the customer did not suffer any permanent injury.', 'not recover, because the theater was under no obligation to inspect the premises to discover unknown dangers.']\",\"question\":\"A seat in a movie theater collapsed, causing the customer sitting in that seat to fall to the ground and suffer a back injury. In a personal injury action by the customer against the theater, the customer will most likely\",\"src\":\"ori_mmlu-professional_law\"}","display_format":"text","language":"","answer_status":"published","assets":[],"source_url":"https://huggingface.co/datasets/TIGER-Lab/MMLU-Pro","history":"initial import","indexing_mode":"index","subproblems":[],"grids":[]}