{"kind":"task","effective_mode":"full","benchmark":{"kind":"benchmark","effective_mode":"full","slug":"mmlu-pro","formal_name":"MMLU-Pro","introduction":"MMLU-Pro rebuilds MMLU with ten answer options instead of four and removes items that no longer separate models. Questions span fourteen academic and professional subjects.","introduction_ja":"","introduction_en":"","category":"Category not supplied","task_count":null,"acquisition_status":"Acquisition status not supplied","official_url":"https://huggingface.co/datasets/TIGER-Lab/MMLU-Pro","indexing_mode":"index","profile":{"resources":[],"task_format":"","scoring":"","metric":"","size":"","answer_access":"","license":"","citation":"","maintainer":"","released":"","why_hard":"","related":[],"history":"","construction":"","limitations":"","saturation":"","scores":[]}},"task_id":"c696d822-8849-51fc-ad07-e7ff05ea0003","task_key":"default--test--11281","task_revision_id":"1","upstream_id":"11281","short_description":"Of the two versions of the principle that Singer considers:","config":"default","split":"test","body":"{\"category\":\"philosophy\",\"options\":\"['he considers the stronger version to be correct.', 'he considers the weaker version to be correct.', 'he proposes a third, completely different principle.', 'he argues that the weaker version is stronger.', 'he claims that they are equivalent.', 'he supports both the stronger and the weaker versions.', 'he suggests that the principles are contradictory.', 'he does not distinguish between the two principles.', 'he states that both principles are incorrect.', 'he ends up rejecting them both.']\",\"question\":\"Of the two versions of the principle that Singer considers:\",\"src\":\"ori_mmlu-philosophy\"}","display_format":"text","language":"","answer_status":"published","assets":[],"source_url":"https://huggingface.co/datasets/TIGER-Lab/MMLU-Pro","history":"initial import","indexing_mode":"index","subproblems":[],"grids":[]}