{"kind":"task","effective_mode":"full","benchmark":{"kind":"benchmark","effective_mode":"full","slug":"mmlu-pro","formal_name":"MMLU-Pro","introduction":"MMLU-Pro rebuilds MMLU with ten answer options instead of four and removes items that no longer separate models. Questions span fourteen academic and professional subjects.","introduction_ja":"","introduction_en":"","category":"Category not supplied","task_count":null,"acquisition_status":"Acquisition status not supplied","official_url":"https://huggingface.co/datasets/TIGER-Lab/MMLU-Pro","indexing_mode":"index","profile":{"resources":[],"task_format":"","scoring":"","metric":"","size":"","answer_access":"","license":"","citation":"","maintainer":"","released":"","why_hard":"","related":[],"history":"","construction":"","limitations":"","saturation":"","scores":[]}},"task_id":"3140f34f-f717-50f7-a1c2-89c23cfb3536","task_key":"default--test--7370","task_revision_id":"1","upstream_id":"7370","short_description":"The total utility from sardines is maximized when they are purchased until","config":"default","split":"test","body":"{\"category\":\"economics\",\"options\":\"['total cost equals total benefit', 'marginal benefit equals marginal cost', 'consumer surplus is zero', 'distributive efficiency is achieved', 'marginal utility is zero', 'consumer demand is at its highest point', 'price is at its lowest point', 'production cost is at its minimum', 'marginal utility equals marginal cost', 'marginal cost is zero']\",\"question\":\"The total utility from sardines is maximized when they are purchased until\",\"src\":\"ori_mmlu-high_school_microeconomics\"}","display_format":"text","language":"","answer_status":"published","assets":[],"source_url":"https://huggingface.co/datasets/TIGER-Lab/MMLU-Pro","history":"initial import","indexing_mode":"index","subproblems":[],"grids":[]}