{"kind":"task","effective_mode":"full","benchmark":{"kind":"benchmark","effective_mode":"full","slug":"mmlu-pro","formal_name":"MMLU-Pro","introduction":"MMLU-Pro rebuilds MMLU with ten answer options instead of four and removes items that no longer separate models. Questions span fourteen academic and professional subjects.","introduction_ja":"","introduction_en":"","category":"Category not supplied","task_count":null,"acquisition_status":"Acquisition status not supplied","official_url":"https://huggingface.co/datasets/TIGER-Lab/MMLU-Pro","indexing_mode":"index","profile":{"resources":[],"task_format":"","scoring":"","metric":"","size":"","answer_access":"","license":"","citation":"","maintainer":"","released":"","why_hard":"","related":[],"history":"","construction":"","limitations":"","saturation":"","scores":[]}},"task_id":"0f186387-da80-54c4-b80b-20ad6439b5f8","task_key":"default--test--11254","task_revision_id":"1","upstream_id":"11254","short_description":"In Butler’s view, acting virtuously consists of acting in accordance with:","config":"default","split":"test","body":"{\"category\":\"philosophy\",\"options\":\"['all of our inward principles.', 'our highest inward principle.', 'the principles dictated by law.', 'the commands of God.', 'the principles of our elders.', 'the principles of society.', 'our weakest inward principle.', 'our strongest inward principle.', 'the principles of our peers.']\",\"question\":\"In Butler’s view, acting virtuously consists of acting in accordance with:\",\"src\":\"ori_mmlu-philosophy\"}","display_format":"text","language":"","answer_status":"published","assets":[],"source_url":"https://huggingface.co/datasets/TIGER-Lab/MMLU-Pro","history":"initial import","indexing_mode":"index","subproblems":[],"grids":[]}