{"kind":"task","effective_mode":"full","benchmark":{"kind":"benchmark","effective_mode":"full","slug":"mmlu-pro","formal_name":"MMLU-Pro","introduction":"MMLU-Pro rebuilds MMLU with ten answer options instead of four and removes items that no longer separate models. Questions span fourteen academic and professional subjects.","introduction_ja":"","introduction_en":"","category":"Category not supplied","task_count":null,"acquisition_status":"Acquisition status not supplied","official_url":"https://huggingface.co/datasets/TIGER-Lab/MMLU-Pro","indexing_mode":"index","profile":{"resources":[],"task_format":"","scoring":"","metric":"","size":"","answer_access":"","license":"","citation":"","maintainer":"","released":"","why_hard":"","related":[],"history":"","construction":"","limitations":"","saturation":"","scores":[]}},"task_id":"09275613-cb05-55c2-ad3d-5b8d51dba544","task_key":"default--test--11030","task_revision_id":"1","upstream_id":"11030","short_description":"For Socrates, the belief that \"virtue is knowledge\" is related to his claim that…","config":"default","split":"test","body":"{\"category\":\"philosophy\",\"options\":\"['the unexamined life is one in which we live day to day without asking questions about who we are and why we are here in the first place.', 'the Delphic oracle identified Socrates as the wisest person on earth because he claimed to know nothing.', 'by questioning traditional beliefs, we learn to recognize how some answers seem to be more satisfactory than others.', 'the only way to be a good or worthwhile person is to know how human beings should behave based on universal norms or values.']\",\"question\":\" For Socrates, the belief that \\\"virtue is knowledge\\\" is related to his claim that \\\"the unexamined life is not worth living,\\\" because he believes that:\",\"src\":\"ori_mmlu-philosophy\"}","display_format":"text","language":"","answer_status":"published","assets":[],"source_url":"https://huggingface.co/datasets/TIGER-Lab/MMLU-Pro","history":"initial import","indexing_mode":"index","subproblems":[],"grids":[]}