{"kind":"task","effective_mode":"full","benchmark":{"kind":"benchmark","effective_mode":"full","slug":"mmlu-pro","formal_name":"MMLU-Pro","introduction":"MMLU-Pro rebuilds MMLU with ten answer options instead of four and removes items that no longer separate models. Questions span fourteen academic and professional subjects.","introduction_ja":"","introduction_en":"","category":"Category not supplied","task_count":null,"acquisition_status":"Acquisition status not supplied","official_url":"https://huggingface.co/datasets/TIGER-Lab/MMLU-Pro","indexing_mode":"index","profile":{"resources":[],"task_format":"","scoring":"","metric":"","size":"","answer_access":"","license":"","citation":"","maintainer":"","released":"","why_hard":"","related":[],"history":"","construction":"","limitations":"","saturation":"","scores":[]}},"task_id":"3114e537-32e4-5be0-a9e6-181ffa88daa9","task_key":"default--test--11040","task_revision_id":"1","upstream_id":"11040","short_description":"Ashford thinks that the right test of whether or not legal and economic…","config":"default","split":"test","body":"{\"category\":\"philosophy\",\"options\":\"['they are accepted by the majority of people.', 'they provide equal opportunities for all.', 'they help develop human capabilities.', 'they recognize, enforce, and implement duties not to violate human rights.', 'they maximize the amount of happiness in the world.', 'they are universalizable.', 'they minimize the amount of suffering in the world.', 'they prioritize the needs of the marginalized.', 'they uphold the principles of democratic governance.', 'they ensure a fair distribution of wealth.']\",\"question\":\" Ashford thinks that the right test of whether or not legal and economic structures are minimally just is to see whether or not\",\"src\":\"ori_mmlu-moral_disputes\"}","display_format":"text","language":"","answer_status":"published","assets":[],"source_url":"https://huggingface.co/datasets/TIGER-Lab/MMLU-Pro","history":"initial import","indexing_mode":"index","subproblems":[],"grids":[]}