{"kind":"task","effective_mode":"full","benchmark":{"kind":"benchmark","effective_mode":"full","slug":"mmlu-pro","formal_name":"MMLU-Pro","introduction":"MMLU-Pro rebuilds MMLU with ten answer options instead of four and removes items that no longer separate models. Questions span fourteen academic and professional subjects.","introduction_ja":"","introduction_en":"","category":"Category not supplied","task_count":null,"acquisition_status":"Acquisition status not supplied","official_url":"https://huggingface.co/datasets/TIGER-Lab/MMLU-Pro","indexing_mode":"index","profile":{"resources":[],"task_format":"","scoring":"","metric":"","size":"","answer_access":"","license":"","citation":"","maintainer":"","released":"","why_hard":"","related":[],"history":"","construction":"","limitations":"","saturation":"","scores":[]}},"task_id":"ed879c08-3e50-5ce8-a38f-38b07a453c37","task_key":"default--test--1579","task_revision_id":"1","upstream_id":"1579","short_description":"A woman sued her former employer in state court, asserting age and sex…","config":"default","split":"test","body":"{\"category\":\"law\",\"options\":\"[\\\"Retain the case, because the employer's attorney did not provide sufficient reasons for remand.\\\", 'Remand the entire case.', \\\"Retain the case to avoid the risk of bias and impropriety in having it proceed before a judge who has shown clear hostility toward the woman's attorney.\\\", \\\"Retain the case, because the woman's attorney has a valid concern about possible bias in state court.\\\", 'Retain the case, as federal courts have exclusive jurisdiction over federal claims.', \\\"Retain the case, as the employer's attorney has not objected to the removal.\\\", 'Retain the case, because it was timely removed and the woman alleges federal claims.', 'Remand the state claims but keep the federal claims.']\",\"question\":\"A woman sued her former employer in state court, asserting age and sex discrimination claims under both state and federal law. The woman's attorney had recently been embarrassed in court by the judge to whom the case was assigned. Wishing to avoid difficulties with the judge, the woman's attorney promptly removed the case to federal court on the basis of federal-question jurisdiction. The employer's attorney has timely moved to remand. How is the federal court likely to proceed?\",\"src\":\"ori_mmlu-professional_law\"}","display_format":"text","language":"","answer_status":"published","assets":[],"source_url":"https://huggingface.co/datasets/TIGER-Lab/MMLU-Pro","history":"initial import","indexing_mode":"index","subproblems":[],"grids":[]}