{"kind":"task","effective_mode":"full","benchmark":{"kind":"benchmark","effective_mode":"full","slug":"mmlu-pro","formal_name":"MMLU-Pro","introduction":"MMLU-Pro rebuilds MMLU with ten answer options instead of four and removes items that no longer separate models. Questions span fourteen academic and professional subjects.","introduction_ja":"","introduction_en":"","category":"Category not supplied","task_count":null,"acquisition_status":"Acquisition status not supplied","official_url":"https://huggingface.co/datasets/TIGER-Lab/MMLU-Pro","indexing_mode":"index","profile":{"resources":[],"task_format":"","scoring":"","metric":"","size":"","answer_access":"","license":"","citation":"","maintainer":"","released":"","why_hard":"","related":[],"history":"","construction":"","limitations":"","saturation":"","scores":[]}},"task_id":"9dca4a10-a544-5497-bcfe-0e9e81628f0d","task_key":"default--test--11047","task_revision_id":"1","upstream_id":"11047","short_description":"Marcia Baron argues that the artificiality of ticking bomb cases is problematic…","config":"default","split":"test","body":"{\"category\":\"philosophy\",\"options\":\"['the artificial assumptions are, in fact, empirical claims that are central to the debate.', 'it leads us to believe that torture is always permissible.', 'all artificial thought experiments are philosophically useless.']\",\"question\":\" Marcia Baron argues that the artificiality of ticking bomb cases is problematic because\",\"src\":\"ori_mmlu-moral_disputes\"}","display_format":"text","language":"","answer_status":"published","assets":[],"source_url":"https://huggingface.co/datasets/TIGER-Lab/MMLU-Pro","history":"initial import","indexing_mode":"index","subproblems":[],"grids":[]}