{"kind":"task","effective_mode":"full","benchmark":{"kind":"benchmark","effective_mode":"full","slug":"mmlu-pro","formal_name":"MMLU-Pro","introduction":"MMLU-Pro rebuilds MMLU with ten answer options instead of four and removes items that no longer separate models. Questions span fourteen academic and professional subjects.","introduction_ja":"","introduction_en":"","category":"Category not supplied","task_count":null,"acquisition_status":"Acquisition status not supplied","official_url":"https://huggingface.co/datasets/TIGER-Lab/MMLU-Pro","indexing_mode":"index","profile":{"resources":[],"task_format":"","scoring":"","metric":"","size":"","answer_access":"","license":"","citation":"","maintainer":"","released":"","why_hard":"","related":[],"history":"","construction":"","limitations":"","saturation":"","scores":[]}},"task_id":"300d2663-5527-5463-81c3-38fffbf9a2f3","task_key":"default--test--10918","task_revision_id":"1","upstream_id":"10918","short_description":"Which of the following is not one of the potentially negative consequences of…","config":"default","split":"test","body":"{\"category\":\"philosophy\",\"options\":\"['the execution of innocent persons', 'the diminishing of state moral authority', 'the creation of martyrs', 'the inequality of application across social classes', 'the incitement effect', 'the emotional trauma for the executioners', 'the promotion of a culture of violence', 'great financial cost', 'the prevention of future murders']\",\"question\":\" Which of the following is not one of the potentially negative consequences of imposing the death penalty?\",\"src\":\"ori_mmlu-moral_disputes\"}","display_format":"text","language":"","answer_status":"published","assets":[],"source_url":"https://huggingface.co/datasets/TIGER-Lab/MMLU-Pro","history":"initial import","indexing_mode":"index","subproblems":[],"grids":[]}