{"kind":"task","effective_mode":"full","benchmark":{"kind":"benchmark","effective_mode":"full","slug":"mmlu-pro","formal_name":"MMLU-Pro","introduction":"MMLU-Pro rebuilds MMLU with ten answer options instead of four and removes items that no longer separate models. Questions span fourteen academic and professional subjects.","introduction_ja":"","introduction_en":"","category":"Category not supplied","task_count":null,"acquisition_status":"Acquisition status not supplied","official_url":"https://huggingface.co/datasets/TIGER-Lab/MMLU-Pro","indexing_mode":"index","profile":{"resources":[],"task_format":"","scoring":"","metric":"","size":"","answer_access":"","license":"","citation":"","maintainer":"","released":"","why_hard":"","related":[],"history":"","construction":"","limitations":"","saturation":"","scores":[]}},"task_id":"582603a2-7440-52b6-b1d8-7a285e608364","task_key":"default--test--9934","task_revision_id":"1","upstream_id":"9934","short_description":"An alarm whose frequency is 400 Hz is dropped out of a third-floor window. The…","config":"default","split":"test","body":"{\"category\":\"physics\",\"options\":\"['appears higher than 400 Hz and the frequency decreases as it falls', 'appears higher than 400 Hz and the frequency alternates as it falls', 'appears higher than 400 Hz and the frequency increase as it falls', 'appears higher than 400 Hz but the frequency remains constant as it falls', 'appears lower than 400 Hz and the frequency increases as it falls', 'appears lower than 400 Hz but the frequency remains constant as it falls', 'appears lower than 400 Hz and the frequency decreases as it falls', 'remains at 400 Hz as it falls', 'appears lower than 400 Hz and the frequency alternates as it falls', 'appears to fluctuate above and below 400 Hz as it falls']\",\"question\":\"An alarm whose frequency is 400 Hz is dropped out of a third-floor window. The student who drops it measures the frequency with a very sensitive oscilloscope. The measured frequency\",\"src\":\"ori_mmlu-high_school_physics\"}","display_format":"text","language":"","answer_status":"published","assets":[],"source_url":"https://huggingface.co/datasets/TIGER-Lab/MMLU-Pro","history":"initial import","indexing_mode":"index","subproblems":[],"grids":[]}