{"kind":"task","effective_mode":"full","benchmark":{"kind":"benchmark","effective_mode":"full","slug":"mmlu-pro","formal_name":"MMLU-Pro","introduction":"MMLU-Pro rebuilds MMLU with ten answer options instead of four and removes items that no longer separate models. Questions span fourteen academic and professional subjects.","introduction_ja":"","introduction_en":"","category":"Category not supplied","task_count":null,"acquisition_status":"Acquisition status not supplied","official_url":"https://huggingface.co/datasets/TIGER-Lab/MMLU-Pro","indexing_mode":"index","profile":{"resources":[],"task_format":"","scoring":"","metric":"","size":"","answer_access":"","license":"","citation":"","maintainer":"","released":"","why_hard":"","related":[],"history":"","construction":"","limitations":"","saturation":"","scores":[]}},"task_id":"044ecb08-7e00-533c-a101-ee72c1ea3e5e","task_key":"default--test--7919","task_revision_id":"1","upstream_id":"7919","short_description":"Mr. Kohlberg owns a flower shop. At the beginning of the day, he had 152 roses.…","config":"default","split":"test","body":"{\"category\":\"math\",\"options\":\"['14', '48', '36', '16', '8', '12', '56', '7', '20', '28']\",\"question\":\"Mr. Kohlberg owns a flower shop. At the beginning of the day, he had 152 roses. Mr. Kohlberg sold 96 of the roses and then wanted to separate the rest of the roses equally among 8 vases. What will be the total number of roses in each vase?\",\"src\":\"ori_mmlu-elementary_mathematics\"}","display_format":"text","language":"","answer_status":"published","assets":[],"source_url":"https://huggingface.co/datasets/TIGER-Lab/MMLU-Pro","history":"initial import","indexing_mode":"index","subproblems":[],"grids":[]}