{"kind":"task","effective_mode":"full","benchmark":{"kind":"benchmark","effective_mode":"full","slug":"mmlu-pro","formal_name":"MMLU-Pro","introduction":"MMLU-Pro rebuilds MMLU with ten answer options instead of four and removes items that no longer separate models. Questions span fourteen academic and professional subjects.","introduction_ja":"","introduction_en":"","category":"Category not supplied","task_count":null,"acquisition_status":"Acquisition status not supplied","official_url":"https://huggingface.co/datasets/TIGER-Lab/MMLU-Pro","indexing_mode":"index","profile":{"resources":[],"task_format":"","scoring":"","metric":"","size":"","answer_access":"","license":"","citation":"","maintainer":"","released":"","why_hard":"","related":[],"history":"","construction":"","limitations":"","saturation":"","scores":[]}},"task_id":"602c5c11-26ac-5c6a-918d-14b70e025c6a","task_key":"default--test--11093","task_revision_id":"1","upstream_id":"11093","short_description":"In his discussion of discrimination in war, Valls suggests that","config":"default","split":"test","body":"{\"category\":\"philosophy\",\"options\":\"['the concept of combatants and noncombatants is outdated and irrelevant.', '\\\"terrorism\\\" should be defined as violence against noncombatants.', 'there is no difference between combatants and noncombatants.', 'the difference between combatants and noncombatants is largely ambiguous.', 'the difference between combatants and noncombatants is categorical and clear.', '\\\"terrorism\\\" should be defined as indiscriminate violence.', 'there is a fine line between combatants and noncombatants.', '\\\"terrorism\\\" should be defined as violence excluding war.', 'the difference between combatants and noncombatants is subjective and varies with perspective.', 'the difference between combatants and noncombatants is a matter of degree.']\",\"question\":\" In his discussion of discrimination in war, Valls suggests that\",\"src\":\"ori_mmlu-moral_disputes\"}","display_format":"text","language":"","answer_status":"published","assets":[],"source_url":"https://huggingface.co/datasets/TIGER-Lab/MMLU-Pro","history":"initial import","indexing_mode":"index","subproblems":[],"grids":[]}