{"kind":"task","effective_mode":"full","benchmark":{"kind":"benchmark","effective_mode":"full","slug":"mmlu-pro","formal_name":"MMLU-Pro","introduction":"MMLU-Pro rebuilds MMLU with ten answer options instead of four and removes items that no longer separate models. Questions span fourteen academic and professional subjects.","introduction_ja":"","introduction_en":"","category":"Category not supplied","task_count":null,"acquisition_status":"Acquisition status not supplied","official_url":"https://huggingface.co/datasets/TIGER-Lab/MMLU-Pro","indexing_mode":"index","profile":{"resources":[],"task_format":"","scoring":"","metric":"","size":"","answer_access":"","license":"","citation":"","maintainer":"","released":"","why_hard":"","related":[],"history":"","construction":"","limitations":"","saturation":"","scores":[]}},"task_id":"d6c3ed74-9d35-54a2-8844-1ea57b973dbb","task_key":"default--test--10807","task_revision_id":"1","upstream_id":"10807","short_description":"Berkeley insists that heat and cold are _____.","config":"default","split":"test","body":"{\"category\":\"philosophy\",\"options\":\"['elements of nature that do not exist independently', 'only things existing apart from our minds', 'only sensations existing in our minds', 'physical objects', 'manifestations of our subconscious', 'only sensations existing outside our minds', 'not real unless they are observed', 'illusions', 'types of energy', 'purely subjective and not measurable']\",\"question\":\"Berkeley insists that heat and cold are _____.\",\"src\":\"ori_mmlu-philosophy\"}","display_format":"text","language":"","answer_status":"published","assets":[],"source_url":"https://huggingface.co/datasets/TIGER-Lab/MMLU-Pro","history":"initial import","indexing_mode":"index","subproblems":[],"grids":[]}