{"kind":"task","effective_mode":"full","benchmark":{"kind":"benchmark","effective_mode":"full","slug":"mmlu-pro","formal_name":"MMLU-Pro","introduction":"MMLU-Pro rebuilds MMLU with ten answer options instead of four and removes items that no longer separate models. Questions span fourteen academic and professional subjects.","introduction_ja":"","introduction_en":"","category":"Category not supplied","task_count":null,"acquisition_status":"Acquisition status not supplied","official_url":"https://huggingface.co/datasets/TIGER-Lab/MMLU-Pro","indexing_mode":"index","profile":{"resources":[],"task_format":"","scoring":"","metric":"","size":"","answer_access":"","license":"","citation":"","maintainer":"","released":"","why_hard":"","related":[],"history":"","construction":"","limitations":"","saturation":"","scores":[]}},"task_id":"02e6777c-6bd4-527b-8694-8794dc75582c","task_key":"default--test--4722","task_revision_id":"1","upstream_id":"4722","short_description":"This question refers to the following information.","config":"default","split":"test","body":"{\"category\":\"history\",\"options\":\"['a process that breaks down ordinary matter', 'essentially a philosophical notion', 'a term that has no physical basis', 'a concept beyond human understanding', 'merely a property of a certain kind of matter', 'a result of carbonic acid, water, and nitrogenous compounds', 'a supernatural phenomenon', 'subject to continual death and renewal', 'a complex chemical reaction', 'a force that works through matter']\",\"question\":\"This question refers to the following information.\\nIn order to make the title of this discourse generally intelligible, I have translated the term \\\"Protoplasm,\\\" which is the scientific name of the substance of which I am about to speak, by the words \\\"the physical basis of life.\\\" I suppose that, to many, the idea that there is such a thing as a physical basis, or matter, of life may be novel—so widely spread is the conception of life as something which works through matter. … Thus the matter of life, so far as we know it (and we have no right to speculate on any other), breaks up, in consequence of that continual death which is the condition of its manifesting vitality, into carbonic acid, water, and nitrogenous compounds, which certainly possess no properties but those of ordinary matter.\\nThomas Henry Huxley, \\\"The Physical Basis of Life,\\\" 1868\\nFrom the passage, one may infer that Huxley argued that \\\"life\\\" was\",\"src\":\"ori_mmlu-high_school_european_history\"}","display_format":"text","language":"","answer_status":"published","assets":[],"source_url":"https://huggingface.co/datasets/TIGER-Lab/MMLU-Pro","history":"initial import","indexing_mode":"index","subproblems":[],"grids":[]}