{"kind":"task","effective_mode":"full","benchmark":{"kind":"benchmark","effective_mode":"full","slug":"mmlu-pro","formal_name":"MMLU-Pro","introduction":"MMLU-Pro rebuilds MMLU with ten answer options instead of four and removes items that no longer separate models. Questions span fourteen academic and professional subjects.","introduction_ja":"","introduction_en":"","category":"Category not supplied","task_count":null,"acquisition_status":"Acquisition status not supplied","official_url":"https://huggingface.co/datasets/TIGER-Lab/MMLU-Pro","indexing_mode":"index","profile":{"resources":[],"task_format":"","scoring":"","metric":"","size":"","answer_access":"","license":"","citation":"","maintainer":"","released":"","why_hard":"","related":[],"history":"","construction":"","limitations":"","saturation":"","scores":[]}},"task_id":"262d6d49-7689-549f-bc6d-42ed21593d51","task_key":"default--test--11252","task_revision_id":"1","upstream_id":"11252","short_description":"When it comes to defining good, Moore claims that many past philosophers are…","config":"default","split":"test","body":"{\"category\":\"philosophy\",\"options\":\"['the false dilemma fallacy.', 'the appeal to authority fallacy.', 'the straw man fallacy.', 'the naturalistic fallacy.', 'the ad hominem fallacy.', 'begging the question.', 'the slippery slope fallacy.', 'the red herring fallacy.', 'circular reasoning.', 'the hasty generalization fallacy.']\",\"question\":\"When it comes to defining good, Moore claims that many past philosophers are guilty of:\",\"src\":\"ori_mmlu-philosophy\"}","display_format":"text","language":"","answer_status":"published","assets":[],"source_url":"https://huggingface.co/datasets/TIGER-Lab/MMLU-Pro","history":"initial import","indexing_mode":"index","subproblems":[],"grids":[]}