{"kind":"task","effective_mode":"full","benchmark":{"kind":"benchmark","effective_mode":"full","slug":"mmlu-pro","formal_name":"MMLU-Pro","introduction":"MMLU-Pro rebuilds MMLU with ten answer options instead of four and removes items that no longer separate models. Questions span fourteen academic and professional subjects.","introduction_ja":"","introduction_en":"","category":"Category not supplied","task_count":null,"acquisition_status":"Acquisition status not supplied","official_url":"https://huggingface.co/datasets/TIGER-Lab/MMLU-Pro","indexing_mode":"index","profile":{"resources":[],"task_format":"","scoring":"","metric":"","size":"","answer_access":"","license":"","citation":"","maintainer":"","released":"","why_hard":"","related":[],"history":"","construction":"","limitations":"","saturation":"","scores":[]}},"task_id":"78fe8f59-5814-59f8-9ddf-214c4996ae29","task_key":"default--test--10827","task_revision_id":"1","upstream_id":"10827","short_description":"According to Singer, international corporations that deal with corrupt dictators…","config":"default","split":"test","body":"{\"category\":\"philosophy\",\"options\":\"[\\\"accomplices in the dictator's crimes.\\\", 'people who knowingly buy stolen goods.', 'thieves who steal from the poor.', 'merchants who sell harmful products.', 'war profiteers exploiting conflict.', 'prostitutes who knowingly spread disease.', 'bystanders who ignore injustice.', 'politicians who accept bribes.', 'all of the above', 'murderers of those who witness crimes.']\",\"question\":\" According to Singer, international corporations that deal with corrupt dictators are akin to\",\"src\":\"ori_mmlu-moral_disputes\"}","display_format":"text","language":"","answer_status":"published","assets":[],"source_url":"https://huggingface.co/datasets/TIGER-Lab/MMLU-Pro","history":"initial import","indexing_mode":"index","subproblems":[],"grids":[]}