{"kind":"task","effective_mode":"full","benchmark":{"kind":"benchmark","effective_mode":"full","slug":"mmlu-pro","formal_name":"MMLU-Pro","introduction":"MMLU-Pro rebuilds MMLU with ten answer options instead of four and removes items that no longer separate models. Questions span fourteen academic and professional subjects.","introduction_ja":"","introduction_en":"","category":"Category not supplied","task_count":null,"acquisition_status":"Acquisition status not supplied","official_url":"https://huggingface.co/datasets/TIGER-Lab/MMLU-Pro","indexing_mode":"index","profile":{"resources":[],"task_format":"","scoring":"","metric":"","size":"","answer_access":"","license":"","citation":"","maintainer":"","released":"","why_hard":"","related":[],"history":"","construction":"","limitations":"","saturation":"","scores":[]}},"task_id":"3de4dac1-60f1-59da-b580-c07e9c906699","task_key":"default--test--1593","task_revision_id":"1","upstream_id":"1593","short_description":"A defendant was driving his new sports car at a high rate of speed on a busy…","config":"default","split":"test","body":"{\"category\":\"law\",\"options\":\"['attempted murder.', 'attempted manslaughter.', 'assault with the intent to cause serious bodily injury.', 'battery.']\",\"question\":\"A defendant was driving his new sports car at a high rate of speed on a busy city street in the middle of the day. A pedestrian was struck by the defendant's car as she was crossing the street and was seriously injured. A criminal complaint was filed against the defendant and, at trial, he testified that he did not intend to injure anyone, but admitted that he was indifferent as to the consequences of driving recklessly. Based on the defendant's actions and testimony, he should be found guilty of\",\"src\":\"ori_mmlu-professional_law\"}","display_format":"text","language":"","answer_status":"published","assets":[],"source_url":"https://huggingface.co/datasets/TIGER-Lab/MMLU-Pro","history":"initial import","indexing_mode":"index","subproblems":[],"grids":[]}