{"kind":"task","effective_mode":"full","benchmark":{"kind":"benchmark","effective_mode":"full","slug":"mmlu-pro","formal_name":"MMLU-Pro","introduction":"MMLU-Pro rebuilds MMLU with ten answer options instead of four and removes items that no longer separate models. Questions span fourteen academic and professional subjects.","introduction_ja":"","introduction_en":"","category":"Category not supplied","task_count":null,"acquisition_status":"Acquisition status not supplied","official_url":"https://huggingface.co/datasets/TIGER-Lab/MMLU-Pro","indexing_mode":"index","profile":{"resources":[],"task_format":"","scoring":"","metric":"","size":"","answer_access":"","license":"","citation":"","maintainer":"","released":"","why_hard":"","related":[],"history":"","construction":"","limitations":"","saturation":"","scores":[]}},"task_id":"94730645-8cbf-5c03-8e75-66ee445108ac","task_key":"default--test--10979","task_revision_id":"1","upstream_id":"10979","short_description":"In Aristotle’s terminology, incontinence is when:","config":"default","split":"test","body":"{\"category\":\"philosophy\",\"options\":\"[\\\"one knows that one's feelings are inappropriate, but acts on them anyway.\\\", 'one knows that one’s feelings are inappropriate, and does not act on them.', \\\"one acts on one's feelings, regardless of their appropriateness.\\\", 'one does not know that one’s actions are wrong.', 'one knows that one’s actions are wrong, but does them anyway.', \\\"one does not act despite knowing that one's actions are right.\\\", 'one does the right action, but for the wrong reason.', \\\"one does not know that one's actions are right.\\\", \\\"one does the right action, but does not understand why it's right.\\\"]\",\"question\":\"In Aristotle’s terminology, incontinence is when:\",\"src\":\"ori_mmlu-philosophy\"}","display_format":"text","language":"","answer_status":"published","assets":[],"source_url":"https://huggingface.co/datasets/TIGER-Lab/MMLU-Pro","history":"initial import","indexing_mode":"index","subproblems":[],"grids":[]}