{"kind":"task","effective_mode":"full","benchmark":{"kind":"benchmark","effective_mode":"full","slug":"mmlu-pro","formal_name":"MMLU-Pro","introduction":"MMLU-Pro rebuilds MMLU with ten answer options instead of four and removes items that no longer separate models. Questions span fourteen academic and professional subjects.","introduction_ja":"","introduction_en":"","category":"Category not supplied","task_count":null,"acquisition_status":"Acquisition status not supplied","official_url":"https://huggingface.co/datasets/TIGER-Lab/MMLU-Pro","indexing_mode":"index","profile":{"resources":[],"task_format":"","scoring":"","metric":"","size":"","answer_access":"","license":"","citation":"","maintainer":"","released":"","why_hard":"","related":[],"history":"","construction":"","limitations":"","saturation":"","scores":[]}},"task_id":"f73ad387-d854-56c7-9b84-865e5cb9ee94","task_key":"default--test--10867","task_revision_id":"1","upstream_id":"10867","short_description":"Which of the following is not mentioned by Macedo as a duty that societies have…","config":"default","split":"test","body":"{\"category\":\"philosophy\",\"options\":\"['duties of fair dealing (e.g., nonexploitation)', 'cosmopolitan egalitarian duties (e.g., ensuring that everyone has equal wealth)', 'duties of rectification and redress (e.g., redressing those who were oppressed)', 'humanitarian duties (e.g., preventing systematic violations of human rights)']\",\"question\":\" Which of the following is not mentioned by Macedo as a duty that societies have toward nonmembers?\",\"src\":\"ori_mmlu-moral_disputes\"}","display_format":"text","language":"","answer_status":"published","assets":[],"source_url":"https://huggingface.co/datasets/TIGER-Lab/MMLU-Pro","history":"initial import","indexing_mode":"index","subproblems":[],"grids":[]}