{"kind":"task","effective_mode":"full","benchmark":{"kind":"benchmark","effective_mode":"full","slug":"mmlu-pro","formal_name":"MMLU-Pro","introduction":"MMLU-Pro rebuilds MMLU with ten answer options instead of four and removes items that no longer separate models. Questions span fourteen academic and professional subjects.","introduction_ja":"","introduction_en":"","category":"Category not supplied","task_count":null,"acquisition_status":"Acquisition status not supplied","official_url":"https://huggingface.co/datasets/TIGER-Lab/MMLU-Pro","indexing_mode":"index","profile":{"resources":[],"task_format":"","scoring":"","metric":"","size":"","answer_access":"","license":"","citation":"","maintainer":"","released":"","why_hard":"","related":[],"history":"","construction":"","limitations":"","saturation":"","scores":[]}},"task_id":"109080de-fa73-5314-96ae-1e5265dc18b3","task_key":"default--test--8006","task_revision_id":"1","upstream_id":"8006","short_description":"In one study half of a class were instructed to watch exactly 1 hour of…","config":"default","split":"test","body":"{\"category\":\"math\",\"options\":\"['The first study was an experiment without a control group, while the second was an observational study.', 'Both studies were observational studies.', 'The first study was a controlled experiment, while the second was an observational study.', 'The first study was an observational study, while the second was an experiment without a control group.', 'The first study was an experiment with a control group, while the second was an observational study.', 'Both studies were controlled experiments.', 'Both studies were experiments without control groups.', 'The first study was an observational study, the second was neither an observational study nor a controlled experiment.', 'The first study was a controlled experiment, while the second was neither an observational study nor a controlled experiment.', 'The first study was an observational study, while the second was a controlled experiment.']\",\"question\":\"In one study half of a class were instructed to watch exactly 1 hour of television per day, the other half were told to watch 5 hours per day, and then their class grades were compared. In a second study students in a class responded to a questionnaire asking about their television usage and their class grades.\",\"src\":\"ori_mmlu-high_school_statistics\"}","display_format":"text","language":"","answer_status":"published","assets":[],"source_url":"https://huggingface.co/datasets/TIGER-Lab/MMLU-Pro","history":"initial import","indexing_mode":"index","subproblems":[],"grids":[]}