{"kind":"task","effective_mode":"full","benchmark":{"kind":"benchmark","effective_mode":"full","slug":"mmlu-pro","formal_name":"MMLU-Pro","introduction":"MMLU-Pro rebuilds MMLU with ten answer options instead of four and removes items that no longer separate models. Questions span fourteen academic and professional subjects.","introduction_ja":"","introduction_en":"","category":"Category not supplied","task_count":null,"acquisition_status":"Acquisition status not supplied","official_url":"https://huggingface.co/datasets/TIGER-Lab/MMLU-Pro","indexing_mode":"index","profile":{"resources":[],"task_format":"","scoring":"","metric":"","size":"","answer_access":"","license":"","citation":"","maintainer":"","released":"","why_hard":"","related":[],"history":"","construction":"","limitations":"","saturation":"","scores":[]}},"task_id":"fb2deb43-2fa1-50f3-a47a-72440558e74c","task_key":"default--test--10988","task_revision_id":"1","upstream_id":"10988","short_description":"Identify the antecedent of the following conditional proposition: When the…","config":"default","split":"test","body":"{\"category\":\"philosophy\",\"options\":\"['The board of trustees prevents it.', 'The university raises tuition.', 'The board of trustees approves it.', \\\"The governor and the board of trustees don't prevent it.\\\", 'The governor approves of it.', \\\"The board of trustees doesn't prevent it.\\\", 'The governor and the board of trustees prevent it.', 'The governor disapproves of it.', \\\"The university doesn't raise tuition.\\\", 'The university lowers tuition.']\",\"question\":\"Identify the antecedent of the following conditional proposition: When the university raises tuition, then either the governor approves of it or the board of trustees doesn't prevent it.\",\"src\":\"ori_mmlu-formal_logic\"}","display_format":"text","language":"","answer_status":"published","assets":[],"source_url":"https://huggingface.co/datasets/TIGER-Lab/MMLU-Pro","history":"initial import","indexing_mode":"index","subproblems":[],"grids":[]}