{"kind":"task","effective_mode":"full","benchmark":{"kind":"benchmark","effective_mode":"full","slug":"mmlu-pro","formal_name":"MMLU-Pro","introduction":"MMLU-Pro rebuilds MMLU with ten answer options instead of four and removes items that no longer separate models. Questions span fourteen academic and professional subjects.","introduction_ja":"","introduction_en":"","category":"Category not supplied","task_count":null,"acquisition_status":"Acquisition status not supplied","official_url":"https://huggingface.co/datasets/TIGER-Lab/MMLU-Pro","indexing_mode":"index","profile":{"resources":[],"task_format":"","scoring":"","metric":"","size":"","answer_access":"","license":"","citation":"","maintainer":"","released":"","why_hard":"","related":[],"history":"","construction":"","limitations":"","saturation":"","scores":[]}},"task_id":"0698b88f-3004-52f0-9ab5-c9f8386fe86a","task_key":"default--test--10873","task_revision_id":"1","upstream_id":"10873","short_description":"Kant's Universal Law formulation does not refer to an agent's wants; rather, it…","config":"default","split":"test","body":"{\"category\":\"philosophy\",\"options\":\"['a kind of consistency test.', 'an instruction for determining the legality of an action.', \\\"a way to measure the value of an action's consequences.\\\", 'a criterion for social acceptability.', 'a protocol for interpersonal communication.', 'a method to determine personal desires.', \\\"a tool for assessing an individual's moral character.\\\", 'a set of guidelines for achieving happiness.', 'an ethical dilemma.', 'a system for evaluating societal norms.']\",\"question\":\" Kant's Universal Law formulation does not refer to an agent's wants; rather, it represents\",\"src\":\"ori_mmlu-moral_disputes\"}","display_format":"text","language":"","answer_status":"published","assets":[],"source_url":"https://huggingface.co/datasets/TIGER-Lab/MMLU-Pro","history":"initial import","indexing_mode":"index","subproblems":[],"grids":[]}