{"kind":"task","effective_mode":"full","benchmark":{"kind":"benchmark","effective_mode":"full","slug":"mmlu-pro","formal_name":"MMLU-Pro","introduction":"MMLU-Pro rebuilds MMLU with ten answer options instead of four and removes items that no longer separate models. Questions span fourteen academic and professional subjects.","introduction_ja":"","introduction_en":"","category":"Category not supplied","task_count":null,"acquisition_status":"Acquisition status not supplied","official_url":"https://huggingface.co/datasets/TIGER-Lab/MMLU-Pro","indexing_mode":"index","profile":{"resources":[],"task_format":"","scoring":"","metric":"","size":"","answer_access":"","license":"","citation":"","maintainer":"","released":"","why_hard":"","related":[],"history":"","construction":"","limitations":"","saturation":"","scores":[]}},"task_id":"f7b66f83-8dfe-54c4-92ee-67e28e452d24","task_key":"default--test--10022","task_revision_id":"1","upstream_id":"10022","short_description":"A magnetic south pole of 15 units strength, when placed 10 cm away from another…","config":"default","split":"test","body":"{\"category\":\"physics\",\"options\":\"['South pole, 1500 poles', 'North pole, 1000 poles', 'South pole, 1000 poles', 'South pole, 2000 poles', 'North pole, 2500 poles', 'North pole, 2000 poles', 'North pole, 1500 poles', 'South pole, 2500 poles', 'North pole, 500 poles', 'South pole, 3000 poles']\",\"question\":\"A magnetic south pole of 15 units strength, when placed 10 cm away from another pole in air, experiences a repulsion of 300 dynes. What is the nature and strength of the second pole?\",\"src\":\"stemez-Physics\"}","display_format":"text","language":"","answer_status":"published","assets":[],"source_url":"https://huggingface.co/datasets/TIGER-Lab/MMLU-Pro","history":"initial import","indexing_mode":"index","subproblems":[],"grids":[]}