{"kind":"task","effective_mode":"full","benchmark":{"kind":"benchmark","effective_mode":"full","slug":"mmlu-pro","formal_name":"MMLU-Pro","introduction":"MMLU-Pro rebuilds MMLU with ten answer options instead of four and removes items that no longer separate models. Questions span fourteen academic and professional subjects.","introduction_ja":"","introduction_en":"","category":"Category not supplied","task_count":null,"acquisition_status":"Acquisition status not supplied","official_url":"https://huggingface.co/datasets/TIGER-Lab/MMLU-Pro","indexing_mode":"index","profile":{"resources":[],"task_format":"","scoring":"","metric":"","size":"","answer_access":"","license":"","citation":"","maintainer":"","released":"","why_hard":"","related":[],"history":"","construction":"","limitations":"","saturation":"","scores":[]}},"task_id":"4fb6b6e1-8172-5a59-b9a4-fb452289e165","task_key":"default--test--11115","task_revision_id":"1","upstream_id":"11115","short_description":"According to Socrates, the value or quality of one's life depends on…","config":"default","split":"test","body":"{\"category\":\"philosophy\",\"options\":\"['understanding existence philosophically leads to personal satisfaction.', \\\"one's life lacks virtue if they do not contribute to society.\\\", 'ignorance is a virtue in itself, as it leads to humility.', 'philosophical understanding is not necessary as long as one is happy.', 'acting virtuously means acting in way that is informed about what one is doing and why.', 'virtue is irrelevant to the quality of life.', 'someone who does not understand existence philosophically could never do anything right.', 'not only is virtue knowledge but also the unexamined life is not worth living.', \\\"the value of life is determined by one's personal wealth and status.\\\", 'to have the power or ability to do anything at all requires that we know what we are doing.']\",\"question\":\"According to Socrates, the value or quality of one's life depends on understanding the principles of, or basic rationale for human existence.  Without such knowledge (he suggests) life lacks virtue, because:\",\"src\":\"ori_mmlu-philosophy\"}","display_format":"text","language":"","answer_status":"published","assets":[],"source_url":"https://huggingface.co/datasets/TIGER-Lab/MMLU-Pro","history":"initial import","indexing_mode":"index","subproblems":[],"grids":[]}