{"kind":"task","effective_mode":"full","benchmark":{"kind":"benchmark","effective_mode":"full","slug":"mmlu-pro","formal_name":"MMLU-Pro","introduction":"MMLU-Pro rebuilds MMLU with ten answer options instead of four and removes items that no longer separate models. Questions span fourteen academic and professional subjects.","introduction_ja":"","introduction_en":"","category":"Category not supplied","task_count":null,"acquisition_status":"Acquisition status not supplied","official_url":"https://huggingface.co/datasets/TIGER-Lab/MMLU-Pro","indexing_mode":"index","profile":{"resources":[],"task_format":"","scoring":"","metric":"","size":"","answer_access":"","license":"","citation":"","maintainer":"","released":"","why_hard":"","related":[],"history":"","construction":"","limitations":"","saturation":"","scores":[]}},"task_id":"e56df16c-a742-5809-90a5-0d0b2d378ea6","task_key":"default--test--3117","task_revision_id":"1","upstream_id":"3117","short_description":"A large island is devastated by a volcanic eruption. Most of the horses die…","config":"default","split":"test","body":"{\"category\":\"biology\",\"options\":\"['a higher mean weight compared with members of the original population', 'the same mean weight as members of the original population', 'a lower mean weight compared with their parents', 'the same mean weight as their parents', 'a lower mean weight compared with members of the original population', 'a greater range of weights compared to the original population', 'a weight distribution skewed towards lighter weights']\",\"question\":\"A large island is devastated by a volcanic eruption. Most of the horses die except for the heaviest males and heaviest females of the group. They survive, reproduce, and perpetuate the population. Since weight is highly heritable and the distribution of weights approximates a binomial distribution, the offspring of the next generation would be expected to have\",\"src\":\"ori_mmlu-high_school_biology\"}","display_format":"text","language":"","answer_status":"published","assets":[],"source_url":"https://huggingface.co/datasets/TIGER-Lab/MMLU-Pro","history":"initial import","indexing_mode":"index","subproblems":[],"grids":[]}