{"kind":"task","effective_mode":"full","benchmark":{"kind":"benchmark","effective_mode":"full","slug":"mmlu-pro","formal_name":"MMLU-Pro","introduction":"MMLU-Pro rebuilds MMLU with ten answer options instead of four and removes items that no longer separate models. Questions span fourteen academic and professional subjects.","introduction_ja":"","introduction_en":"","category":"Category not supplied","task_count":null,"acquisition_status":"Acquisition status not supplied","official_url":"https://huggingface.co/datasets/TIGER-Lab/MMLU-Pro","indexing_mode":"index","profile":{"resources":[],"task_format":"","scoring":"","metric":"","size":"","answer_access":"","license":"","citation":"","maintainer":"","released":"","why_hard":"","related":[],"history":"","construction":"","limitations":"","saturation":"","scores":[]}},"task_id":"f8e972cf-7db2-5b40-b5bd-44dc5f2a6648","task_key":"default--test--9995","task_revision_id":"1","upstream_id":"9995","short_description":"The quantum efficiency of a photon detector is 0.1. If 100 photons are sent into…","config":"default","split":"test","body":"{\"category\":\"physics\",\"options\":\"['an average of 10 times, with an rms deviation of about 5', 'an average of 10 times, with an rms deviation of about 3', 'an average of 10 times, with an rms deviation of about 0.1', 'an average of 10 times, with an rms deviation of about 1', 'an average of 10 times, with an rms deviation of about 6', 'an average of 5 times, with an rms deviation of about 2', 'an average of 20 times, with an rms deviation of about 3', 'an average of 10 times, with an rms deviation of about 4', 'an average of 15 times, with an rms deviation of about 4', 'an average of 10 times, with an rms deviation of about 2']\",\"question\":\"The quantum efficiency of a photon detector is 0.1. If 100 photons are sent into the detector, one after the other, the detector will detect photons\",\"src\":\"ori_mmlu-college_physics\"}","display_format":"text","language":"","answer_status":"published","assets":[],"source_url":"https://huggingface.co/datasets/TIGER-Lab/MMLU-Pro","history":"initial import","indexing_mode":"index","subproblems":[],"grids":[]}