{"kind":"task","effective_mode":"full","benchmark":{"kind":"benchmark","effective_mode":"full","slug":"mmlu-pro","formal_name":"MMLU-Pro","introduction":"MMLU-Pro rebuilds MMLU with ten answer options instead of four and removes items that no longer separate models. Questions span fourteen academic and professional subjects.","introduction_ja":"","introduction_en":"","category":"Category not supplied","task_count":null,"acquisition_status":"Acquisition status not supplied","official_url":"https://huggingface.co/datasets/TIGER-Lab/MMLU-Pro","indexing_mode":"index","profile":{"resources":[],"task_format":"","scoring":"","metric":"","size":"","answer_access":"","license":"","citation":"","maintainer":"","released":"","why_hard":"","related":[],"history":"","construction":"","limitations":"","saturation":"","scores":[]}},"task_id":"374972af-243f-5ee2-b209-4b01df7a9ad4","task_key":"default--test--7112","task_revision_id":"1","upstream_id":"7112","short_description":"A monopoly with a straight, downward-sloping demand curve has a marginal revenue…","config":"default","split":"test","body":"{\"category\":\"economics\",\"options\":\"['initially upward sloping and then downward sloping', 'the same as the demand curve', 'in the shape of a U', 'the inverse of the demand curve', 'parallel to the demand curve', 'upward sloping', 'halfway between the demand curve and the vertical axis', 'initially downward sloping and then upward sloping', 'horizontal', 'perpendicular to the demand curve']\",\"question\":\"A monopoly with a straight, downward-sloping demand curve has a marginal revenue curve that is\",\"src\":\"ori_mmlu-high_school_microeconomics\"}","display_format":"text","language":"","answer_status":"published","assets":[],"source_url":"https://huggingface.co/datasets/TIGER-Lab/MMLU-Pro","history":"initial import","indexing_mode":"index","subproblems":[],"grids":[]}