{"kind":"task","effective_mode":"full","benchmark":{"kind":"benchmark","effective_mode":"full","slug":"mmlu-pro","formal_name":"MMLU-Pro","introduction":"MMLU-Pro rebuilds MMLU with ten answer options instead of four and removes items that no longer separate models. Questions span fourteen academic and professional subjects.","introduction_ja":"","introduction_en":"","category":"Category not supplied","task_count":null,"acquisition_status":"Acquisition status not supplied","official_url":"https://huggingface.co/datasets/TIGER-Lab/MMLU-Pro","indexing_mode":"index","profile":{"resources":[],"task_format":"","scoring":"","metric":"","size":"","answer_access":"","license":"","citation":"","maintainer":"","released":"","why_hard":"","related":[],"history":"","construction":"","limitations":"","saturation":"","scores":[]}},"task_id":"1644ac32-f16f-58de-9f9c-2124fe5ee947","task_key":"default--test--10784","task_revision_id":"1","upstream_id":"10784","short_description":"According to Reiman, van den Haag's arguments in favor of the death penalty","config":"default","split":"test","body":"{\"category\":\"philosophy\",\"options\":\"['are not based on logical reasoning.', 'are flawed due to lack of empirical evidence.', 'contradict the principles of human rights.', 'follow common sense.', 'are based on biased principles.', 'should be accepted.', 'prove that torture is morally impermissible.', 'are based on the concept of retribution.', 'promote the idea of capital punishment.', 'are incompatible with the right to retaliation.']\",\"question\":\" According to Reiman, van den Haag's arguments in favor of the death penalty\",\"src\":\"ori_mmlu-moral_disputes\"}","display_format":"text","language":"","answer_status":"published","assets":[],"source_url":"https://huggingface.co/datasets/TIGER-Lab/MMLU-Pro","history":"initial import","indexing_mode":"index","subproblems":[],"grids":[]}