{"kind":"task","effective_mode":"full","benchmark":{"kind":"benchmark","effective_mode":"full","slug":"mmlu-pro","formal_name":"MMLU-Pro","introduction":"MMLU-Pro rebuilds MMLU with ten answer options instead of four and removes items that no longer separate models. Questions span fourteen academic and professional subjects.","introduction_ja":"","introduction_en":"","category":"Category not supplied","task_count":null,"acquisition_status":"Acquisition status not supplied","official_url":"https://huggingface.co/datasets/TIGER-Lab/MMLU-Pro","indexing_mode":"index","profile":{"resources":[],"task_format":"","scoring":"","metric":"","size":"","answer_access":"","license":"","citation":"","maintainer":"","released":"","why_hard":"","related":[],"history":"","construction":"","limitations":"","saturation":"","scores":[]}},"task_id":"1c927f7a-627f-5fb0-86de-84bf6a0503ef","task_key":"default--test--1835","task_revision_id":"1","upstream_id":"1835","short_description":"A defendant had an argument with his neighbor. As they were quarreling, the…","config":"default","split":"test","body":"{\"category\":\"law\",\"options\":\"['Negligent homicide.', 'Involuntary manslaughter.', 'Murder.', 'Aggravated assault.', 'Voluntary manslaughter.', 'Reckless endangerment.', 'Assault with a deadly weapon.', 'Battery.', 'Felony murder.', 'Attempted murder.']\",\"question\":\"A defendant had an argument with his neighbor. As they were quarreling, the defendant pulled out his penknife intending only to frighten the neighbor. The defendant accidentally slightly nicked the neighbor's arm with the knife. Unknown to the defendant, his neighbor was a hemophiliac who then died from the cut. What is the most serious crime that the defendant can be convicted of?\",\"src\":\"ori_mmlu-professional_law\"}","display_format":"text","language":"","answer_status":"published","assets":[],"source_url":"https://huggingface.co/datasets/TIGER-Lab/MMLU-Pro","history":"initial import","indexing_mode":"index","subproblems":[],"grids":[]}