{"kind":"task","effective_mode":"full","benchmark":{"kind":"benchmark","effective_mode":"full","slug":"mmlu-pro","formal_name":"MMLU-Pro","introduction":"MMLU-Pro rebuilds MMLU with ten answer options instead of four and removes items that no longer separate models. Questions span fourteen academic and professional subjects.","introduction_ja":"","introduction_en":"","category":"Category not supplied","task_count":null,"acquisition_status":"Acquisition status not supplied","official_url":"https://huggingface.co/datasets/TIGER-Lab/MMLU-Pro","indexing_mode":"index","profile":{"resources":[],"task_format":"","scoring":"","metric":"","size":"","answer_access":"","license":"","citation":"","maintainer":"","released":"","why_hard":"","related":[],"history":"","construction":"","limitations":"","saturation":"","scores":[]}},"task_id":"f1babd0a-3cb6-5408-8e2f-8009f1b63dff","task_key":"default--test--5843","task_revision_id":"1","upstream_id":"5843","short_description":"Which of the following techniques effectively measures improvements in product…","config":"default","split":"test","body":"{\"category\":\"other\",\"options\":\"['Inspection of in-process goods.', 'Monitoring the number of customer complaints.', 'Measuring the time taken to manufacture each product.', 'Comparing sales data over time.', 'Counting the number of products produced.', 'Checking the quantity of raw materials used.', 'Tracking warranty expenses over time.', 'Evaluating employee performance reviews.', 'Tracking the number of products reworked.', 'Recording the number of products returned over time.']\",\"question\":\"Which of the following techniques effectively measures improvements in product quality as a result of internal failure costs?\",\"src\":\"ori_mmlu-professional_accounting\"}","display_format":"text","language":"","answer_status":"published","assets":[],"source_url":"https://huggingface.co/datasets/TIGER-Lab/MMLU-Pro","history":"initial import","indexing_mode":"index","subproblems":[],"grids":[]}