{"kind":"task","effective_mode":"full","benchmark":{"kind":"benchmark","effective_mode":"full","slug":"mmlu-pro","formal_name":"MMLU-Pro","introduction":"MMLU-Pro rebuilds MMLU with ten answer options instead of four and removes items that no longer separate models. Questions span fourteen academic and professional subjects.","introduction_ja":"","introduction_en":"","category":"Category not supplied","task_count":null,"acquisition_status":"Acquisition status not supplied","official_url":"https://huggingface.co/datasets/TIGER-Lab/MMLU-Pro","indexing_mode":"index","profile":{"resources":[],"task_format":"","scoring":"","metric":"","size":"","answer_access":"","license":"","citation":"","maintainer":"","released":"","why_hard":"","related":[],"history":"","construction":"","limitations":"","saturation":"","scores":[]}},"task_id":"56c8e8b6-153e-5fca-9183-b0e30d33f653","task_key":"default--test--7446","task_revision_id":"1","upstream_id":"7446","short_description":"If there were a leverage effect in practice, what would be the shape of the news…","config":"default","split":"test","body":"{\"category\":\"economics\",\"options\":\"['It would rise more quickly for negative disturbances than for positive ones of the same magnitude', 'It would be symmetrical about zero', 'It would rise less quickly for negative disturbances than for positive ones of the same magnitude', 'It would be zero for all positive disturbances']\",\"question\":\"If there were a leverage effect in practice, what would be the shape of the news impact curve for as model that accounted for that leverage?\",\"src\":\"ori_mmlu-econometrics\"}","display_format":"text","language":"","answer_status":"published","assets":[],"source_url":"https://huggingface.co/datasets/TIGER-Lab/MMLU-Pro","history":"initial import","indexing_mode":"index","subproblems":[],"grids":[]}