{"kind":"task","effective_mode":"full","benchmark":{"kind":"benchmark","effective_mode":"full","slug":"mmlu-pro","formal_name":"MMLU-Pro","introduction":"MMLU-Pro rebuilds MMLU with ten answer options instead of four and removes items that no longer separate models. Questions span fourteen academic and professional subjects.","introduction_ja":"","introduction_en":"","category":"Category not supplied","task_count":null,"acquisition_status":"Acquisition status not supplied","official_url":"https://huggingface.co/datasets/TIGER-Lab/MMLU-Pro","indexing_mode":"index","profile":{"resources":[],"task_format":"","scoring":"","metric":"","size":"","answer_access":"","license":"","citation":"","maintainer":"","released":"","why_hard":"","related":[],"history":"","construction":"","limitations":"","saturation":"","scores":[]}},"task_id":"a6314316-77c1-54a4-b72f-175983ec4696","task_key":"default--test--5344","task_revision_id":"1","upstream_id":"5344","short_description":"A spouse died on December 31, year 1. The couple had no dependents. What should…","config":"default","split":"test","body":"{\"category\":\"other\",\"options\":\"['Nonresident alien.', 'Qualifying widow(er) with dependent child.', 'Single.', 'Married filing separately.', 'Married filing jointly.', 'Widow(er) with dependent child.', 'Head of household.', 'Jointly with deceased spouse.', 'Qualifying widow(er).', 'Dependent of another taxpayer.']\",\"question\":\"A spouse died on December 31, year 1. The couple had no dependents. What should be the filing status of the surviving spouse in year 2?\",\"src\":\"ori_mmlu-professional_accounting\"}","display_format":"text","language":"","answer_status":"published","assets":[],"source_url":"https://huggingface.co/datasets/TIGER-Lab/MMLU-Pro","history":"initial import","indexing_mode":"index","subproblems":[],"grids":[]}