{"kind":"task","effective_mode":"full","benchmark":{"kind":"benchmark","effective_mode":"full","slug":"mmlu-pro","formal_name":"MMLU-Pro","introduction":"MMLU-Pro rebuilds MMLU with ten answer options instead of four and removes items that no longer separate models. Questions span fourteen academic and professional subjects.","introduction_ja":"","introduction_en":"","category":"Category not supplied","task_count":null,"acquisition_status":"Acquisition status not supplied","official_url":"https://huggingface.co/datasets/TIGER-Lab/MMLU-Pro","indexing_mode":"index","profile":{"resources":[],"task_format":"","scoring":"","metric":"","size":"","answer_access":"","license":"","citation":"","maintainer":"","released":"","why_hard":"","related":[],"history":"","construction":"","limitations":"","saturation":"","scores":[]}},"task_id":"590a57e8-6a82-579f-a719-4342dfce328d","task_key":"default--test--11159","task_revision_id":"1","upstream_id":"11159","short_description":"Singer claims that famine relief organizations direct our aid to distant…","config":"default","split":"test","body":"{\"category\":\"philosophy\",\"options\":\"['with a focus on quantity over quality, leading to possible misallocation.', 'very inefficiently, and are therefore not worth donating to.', 'with a high level of bureaucracy and corruption, therefore not recommended.', 'more effectively than we could get it to our close neighbors.', 'extremely efficiently, making them the only viable option for aid distribution.', 'very effectively, but only in certain regions.', 'moderately efficiently, although there may be better alternatives.', 'almost as effectively as we could get it to our close neighbors.', 'effectively, but at a high operational cost.']\",\"question\":\"Singer claims that famine relief organizations direct our aid to distant refugees:\",\"src\":\"ori_mmlu-philosophy\"}","display_format":"text","language":"","answer_status":"published","assets":[],"source_url":"https://huggingface.co/datasets/TIGER-Lab/MMLU-Pro","history":"initial import","indexing_mode":"index","subproblems":[],"grids":[]}