{"kind":"task","effective_mode":"full","benchmark":{"kind":"benchmark","effective_mode":"full","slug":"mmlu-pro","formal_name":"MMLU-Pro","introduction":"MMLU-Pro rebuilds MMLU with ten answer options instead of four and removes items that no longer separate models. Questions span fourteen academic and professional subjects.","introduction_ja":"","introduction_en":"","category":"Category not supplied","task_count":null,"acquisition_status":"Acquisition status not supplied","official_url":"https://huggingface.co/datasets/TIGER-Lab/MMLU-Pro","indexing_mode":"index","profile":{"resources":[],"task_format":"","scoring":"","metric":"","size":"","answer_access":"","license":"","citation":"","maintainer":"","released":"","why_hard":"","related":[],"history":"","construction":"","limitations":"","saturation":"","scores":[]}},"task_id":"efcdb6bf-c6c1-5a65-b2c4-baedb3a2c8a2","task_key":"default--test--2755","task_revision_id":"1","upstream_id":"2755","short_description":"The correction for attenuation formula is used to measure the impact of…","config":"default","split":"test","body":"{\"category\":\"psychology\",\"options\":\"['the number of test takers on the test’s validity.', 'the number of test takers on the test’s reliability.', \\\"the test's difficulty on its validity.\\\", 'a test’s validity on its difficulty.', 'a test’s reliability on its difficulty.', 'a test’s validity on its reliability.', \\\"the test's difficulty on its reliability.\\\", 'the number of test items on the test’s validity.', 'a test’s reliability on its validity.', 'the number of test items on the test’s reliability.']\",\"question\":\"The correction for attenuation formula is used to measure the impact of increasing:\",\"src\":\"ori_mmlu-professional_psychology\"}","display_format":"text","language":"","answer_status":"published","assets":[],"source_url":"https://huggingface.co/datasets/TIGER-Lab/MMLU-Pro","history":"initial import","indexing_mode":"index","subproblems":[],"grids":[]}