{"kind":"task","effective_mode":"full","benchmark":{"kind":"benchmark","effective_mode":"full","slug":"mmlu-pro","formal_name":"MMLU-Pro","introduction":"MMLU-Pro rebuilds MMLU with ten answer options instead of four and removes items that no longer separate models. Questions span fourteen academic and professional subjects.","introduction_ja":"","introduction_en":"","category":"Category not supplied","task_count":null,"acquisition_status":"Acquisition status not supplied","official_url":"https://huggingface.co/datasets/TIGER-Lab/MMLU-Pro","indexing_mode":"index","profile":{"resources":[],"task_format":"","scoring":"","metric":"","size":"","answer_access":"","license":"","citation":"","maintainer":"","released":"","why_hard":"","related":[],"history":"","construction":"","limitations":"","saturation":"","scores":[]}},"task_id":"811cc520-0efe-593d-bf3e-5bc784fecc6b","task_key":"default--test--11073","task_revision_id":"1","upstream_id":"11073","short_description":"Anscombe criticizes Butler on the grounds that:","config":"default","split":"test","body":"{\"category\":\"philosophy\",\"options\":\"['the notion of \\\"freedom\\\" is overly simplistic.', 'the concept of “pleasure” is unclear.', 'the notion of “legislating for oneself” is absurd.', 'the idea of \\\"moral truth\\\" is a contradiction.', \\\"we can't really distinguish between good and evil.\\\", 'our consciences are fallible.', 'we really can derive an “ought” from an “is.”', 'the idea of moral obligations is incoherent.', 'the concept of \\\"identity\\\" is too subjective.', 'the concept of \\\"self\\\" is ill-defined.']\",\"question\":\"Anscombe criticizes Butler on the grounds that:\",\"src\":\"ori_mmlu-philosophy\"}","display_format":"text","language":"","answer_status":"published","assets":[],"source_url":"https://huggingface.co/datasets/TIGER-Lab/MMLU-Pro","history":"initial import","indexing_mode":"index","subproblems":[],"grids":[]}