{"kind":"task","effective_mode":"full","benchmark":{"kind":"benchmark","effective_mode":"full","slug":"mmlu-pro","formal_name":"MMLU-Pro","introduction":"MMLU-Pro rebuilds MMLU with ten answer options instead of four and removes items that no longer separate models. Questions span fourteen academic and professional subjects.","introduction_ja":"","introduction_en":"","category":"Category not supplied","task_count":null,"acquisition_status":"Acquisition status not supplied","official_url":"https://huggingface.co/datasets/TIGER-Lab/MMLU-Pro","indexing_mode":"index","profile":{"resources":[],"task_format":"","scoring":"","metric":"","size":"","answer_access":"","license":"","citation":"","maintainer":"","released":"","why_hard":"","related":[],"history":"","construction":"","limitations":"","saturation":"","scores":[]}},"task_id":"44ef9877-0717-53fa-8365-ea27f84f009c","task_key":"default--test--1983","task_revision_id":"1","upstream_id":"1983","short_description":"A woman attended a private university. One afternoon she was approached by a…","config":"default","split":"test","body":"{\"category\":\"law\",\"options\":\"['dismiss the action as moot.', 'dismiss the action because the university has agreed to stop the harassment.', 'dismiss the case because the woman did not file a complaint with the university first.', 'dismiss the action since the issues are no longer ripe.', 'dismiss the action because there is no case or controversy.', \\\"hear the case because the woman's career could be affected.\\\", 'dismiss the action because the woman agreed to pose for the pictures.', 'dismiss the case because it involves a private institution.', 'hear the case.', \\\"hear the case because the woman's rights have been violated.\\\"]\",\"question\":\"A woman attended a private university. One afternoon she was approached by a photographer for a magazine, who asked her if she would be interested in posing nude for an upcoming issue. The woman, who was an aspiring model, agreed and posed for a number of nude pictures, which subsequently appeared in the magazine. Afterward, administrators and professors at the university began to harass her for what they considered to be imprudent behavior. During class, her instructors frequently called her \\\"Playmate\\\" and \\\"Stripper. \\\" Consequently, the woman brought suit in federal court against the university, alleging harassment, and seeking an injunction and damages. After this action was instituted, the university signed a stipulation agreeing not to harass her in the future. The federal court should now\",\"src\":\"ori_mmlu-professional_law\"}","display_format":"text","language":"","answer_status":"published","assets":[],"source_url":"https://huggingface.co/datasets/TIGER-Lab/MMLU-Pro","history":"initial import","indexing_mode":"index","subproblems":[],"grids":[]}