{"kind":"task","effective_mode":"full","benchmark":{"kind":"benchmark","effective_mode":"full","slug":"mmlu-pro","formal_name":"MMLU-Pro","introduction":"MMLU-Pro rebuilds MMLU with ten answer options instead of four and removes items that no longer separate models. Questions span fourteen academic and professional subjects.","introduction_ja":"","introduction_en":"","category":"Category not supplied","task_count":null,"acquisition_status":"Acquisition status not supplied","official_url":"https://huggingface.co/datasets/TIGER-Lab/MMLU-Pro","indexing_mode":"index","profile":{"resources":[],"task_format":"","scoring":"","metric":"","size":"","answer_access":"","license":"","citation":"","maintainer":"","released":"","why_hard":"","related":[],"history":"","construction":"","limitations":"","saturation":"","scores":[]}},"task_id":"2f6a6d44-b025-594d-b836-653ffb849772","task_key":"default--test--1373","task_revision_id":"1","upstream_id":"1373","short_description":"Which of the following quotations best describes the central thesis of…","config":"default","split":"test","body":"{\"category\":\"law\",\"options\":\"[\\\"Men are unable to comprehend their differences from women.'\\\", \\\"Men and women have different conceptions of the feminist project.'\\\", \\\"Women look to context, whereas men appeal to neutral, abstract notions of justice.'\\\", \\\"'Women and men have identical perspectives on justice.'\\\", \\\"'Feminism is about eradicating differences between men and women.'\\\", \\\"'Difference feminism argues that men and women are fundamentally the same.'\\\", \\\"'Men and women differ only in physical attributes, not in their thought processes.'\\\", \\\"'Men are more inclined towards justice, while women tend to focus on fairness.'\\\", \\\"There are fundamental differences between individual women.'\\\", \\\"'Women are more focused on equality, while men prioritize freedom.'\\\"]\",\"question\":\"Which of the following quotations best describes the central thesis of difference feminism? \",\"src\":\"ori_mmlu-jurisprudence\"}","display_format":"text","language":"","answer_status":"published","assets":[],"source_url":"https://huggingface.co/datasets/TIGER-Lab/MMLU-Pro","history":"initial import","indexing_mode":"index","subproblems":[],"grids":[]}