{"kind":"task","effective_mode":"full","benchmark":{"kind":"benchmark","effective_mode":"full","slug":"mmlu-pro","formal_name":"MMLU-Pro","introduction":"MMLU-Pro rebuilds MMLU with ten answer options instead of four and removes items that no longer separate models. Questions span fourteen academic and professional subjects.","introduction_ja":"","introduction_en":"","category":"Category not supplied","task_count":null,"acquisition_status":"Acquisition status not supplied","official_url":"https://huggingface.co/datasets/TIGER-Lab/MMLU-Pro","indexing_mode":"index","profile":{"resources":[],"task_format":"","scoring":"","metric":"","size":"","answer_access":"","license":"","citation":"","maintainer":"","released":"","why_hard":"","related":[],"history":"","construction":"","limitations":"","saturation":"","scores":[]}},"task_id":"c71bb187-d5e6-5d3c-a61d-b9a16f1a2ef4","task_key":"default--test--1670","task_revision_id":"1","upstream_id":"1670","short_description":"A 17-year-old boy was jostling with his 12-year-old brother. A pedestrian…","config":"default","split":"test","body":"{\"category\":\"law\",\"options\":\"['prevail, because the pedestrian reasonably believed that the boy was about to injure his brother.', 'prevail, based upon the doctrine of transferred intent.', 'not prevail, because the pedestrian assumed the risk.', 'not prevail, unless the boy was negligent.']\",\"question\":\"A 17-year-old boy was jostling with his 12-year-old brother. A pedestrian witnessed the boy shoving his brother and mistakenly believed that they were fighting. Acting as a Good Samaritan, the pedestrian approached the boys and tried to intercede. The boy stumbled and accidentally fell into the pedestrian, knocking him to the ground. The pedestrian broke his arm as a result of the fall. If the pedestrian sues the boy to recover damages for his injury, he should\",\"src\":\"ori_mmlu-professional_law\"}","display_format":"text","language":"","answer_status":"published","assets":[],"source_url":"https://huggingface.co/datasets/TIGER-Lab/MMLU-Pro","history":"initial import","indexing_mode":"index","subproblems":[],"grids":[]}