{"kind":"task","effective_mode":"full","benchmark":{"kind":"benchmark","effective_mode":"full","slug":"mmlu-pro","formal_name":"MMLU-Pro","introduction":"MMLU-Pro rebuilds MMLU with ten answer options instead of four and removes items that no longer separate models. Questions span fourteen academic and professional subjects.","introduction_ja":"","introduction_en":"","category":"Category not supplied","task_count":null,"acquisition_status":"Acquisition status not supplied","official_url":"https://huggingface.co/datasets/TIGER-Lab/MMLU-Pro","indexing_mode":"index","profile":{"resources":[],"task_format":"","scoring":"","metric":"","size":"","answer_access":"","license":"","citation":"","maintainer":"","released":"","why_hard":"","related":[],"history":"","construction":"","limitations":"","saturation":"","scores":[]}},"task_id":"8ff64504-fa1a-5c3c-97bd-1bdd211a7880","task_key":"default--test--11280","task_revision_id":"1","upstream_id":"11280","short_description":"Mill claims that in deciding how to act:","config":"default","split":"test","body":"{\"category\":\"philosophy\",\"options\":\"['we should consult tradition.', 'we should always follow our personal instincts.', 'we should depend on rules of thumb derived from the principle of utility.', 'we should always act in a way that aligns with our personal values, regardless of the consequences.', 'we should always do what is legally required, regardless of the consequences.', 'we should always act in a way that benefits the most people, regardless of the consequences.', 'we should always do what our peers or society expects of us.', 'we should always calculate the expected consequences of our action.', 'we should always do what makes us happy in the moment.', 'we should consult scripture.']\",\"question\":\"Mill claims that in deciding how to act:\",\"src\":\"ori_mmlu-philosophy\"}","display_format":"text","language":"","answer_status":"published","assets":[],"source_url":"https://huggingface.co/datasets/TIGER-Lab/MMLU-Pro","history":"initial import","indexing_mode":"index","subproblems":[],"grids":[]}