{"kind":"task","effective_mode":"full","benchmark":{"kind":"benchmark","effective_mode":"full","slug":"mmlu-pro","formal_name":"MMLU-Pro","introduction":"MMLU-Pro rebuilds MMLU with ten answer options instead of four and removes items that no longer separate models. Questions span fourteen academic and professional subjects.","introduction_ja":"","introduction_en":"","category":"Category not supplied","task_count":null,"acquisition_status":"Acquisition status not supplied","official_url":"https://huggingface.co/datasets/TIGER-Lab/MMLU-Pro","indexing_mode":"index","profile":{"resources":[],"task_format":"","scoring":"","metric":"","size":"","answer_access":"","license":"","citation":"","maintainer":"","released":"","why_hard":"","related":[],"history":"","construction":"","limitations":"","saturation":"","scores":[]}},"task_id":"5e492719-f230-583d-a7bc-0c9573ade522","task_key":"default--test--10904","task_revision_id":"1","upstream_id":"10904","short_description":"According to Mill, the value of a particular pleasure depends on","config":"default","split":"test","body":"{\"category\":\"philosophy\",\"options\":\"['its rarity or frequency.', 'societal norms and values.', \\\"the individual's personal preference.\\\", 'the amount of effort required to obtain it.', 'the potential pain that might accompany it.', 'its quantity alone.', 'its quality alone.', 'neither its quantity nor its quality.', 'both its quantity and its quality.', 'the duration of the pleasure.']\",\"question\":\" According to Mill, the value of a particular pleasure depends on\",\"src\":\"ori_mmlu-moral_disputes\"}","display_format":"text","language":"","answer_status":"published","assets":[],"source_url":"https://huggingface.co/datasets/TIGER-Lab/MMLU-Pro","history":"initial import","indexing_mode":"index","subproblems":[],"grids":[]}