{"kind":"task","effective_mode":"full","benchmark":{"kind":"benchmark","effective_mode":"full","slug":"mmlu-pro","formal_name":"MMLU-Pro","introduction":"MMLU-Pro rebuilds MMLU with ten answer options instead of four and removes items that no longer separate models. Questions span fourteen academic and professional subjects.","introduction_ja":"","introduction_en":"","category":"Category not supplied","task_count":null,"acquisition_status":"Acquisition status not supplied","official_url":"https://huggingface.co/datasets/TIGER-Lab/MMLU-Pro","indexing_mode":"index","profile":{"resources":[],"task_format":"","scoring":"","metric":"","size":"","answer_access":"","license":"","citation":"","maintainer":"","released":"","why_hard":"","related":[],"history":"","construction":"","limitations":"","saturation":"","scores":[]}},"task_id":"fccd2d14-2d57-5571-8bad-f197b6c2eb20","task_key":"default--test--10942","task_revision_id":"1","upstream_id":"10942","short_description":"Anscombe claims that on Sidgwick’s view, the badness of an action must be…","config":"default","split":"test","body":"{\"category\":\"philosophy\",\"options\":\"['whether it upholds social justice.', 'whether it violates divine law.', 'whether it violates any duties.', 'its potential for causing harm.', \\\"the actor's intentions.\\\", 'its expected consequences.', 'its actual consequences.', 'its potential benefits.', 'its alignment with personal values.', 'the cultural norms of the time.']\",\"question\":\"Anscombe claims that on Sidgwick’s view, the badness of an action must be estimated in light of:\",\"src\":\"ori_mmlu-philosophy\"}","display_format":"text","language":"","answer_status":"published","assets":[],"source_url":"https://huggingface.co/datasets/TIGER-Lab/MMLU-Pro","history":"initial import","indexing_mode":"index","subproblems":[],"grids":[]}