{"kind":"task","effective_mode":"full","benchmark":{"kind":"benchmark","effective_mode":"full","slug":"mmlu-pro","formal_name":"MMLU-Pro","introduction":"MMLU-Pro rebuilds MMLU with ten answer options instead of four and removes items that no longer separate models. Questions span fourteen academic and professional subjects.","introduction_ja":"","introduction_en":"","category":"Category not supplied","task_count":null,"acquisition_status":"Acquisition status not supplied","official_url":"https://huggingface.co/datasets/TIGER-Lab/MMLU-Pro","indexing_mode":"index","profile":{"resources":[],"task_format":"","scoring":"","metric":"","size":"","answer_access":"","license":"","citation":"","maintainer":"","released":"","why_hard":"","related":[],"history":"","construction":"","limitations":"","saturation":"","scores":[]}},"task_id":"05a48ee7-8982-586d-8d32-c522e484f708","task_key":"default--test--11091","task_revision_id":"1","upstream_id":"11091","short_description":"Suppose two athletes, A and B, are equally skilled. Suppose also that A…","config":"default","split":"test","body":"{\"category\":\"philosophy\",\"options\":\"['A is a better athlete.', 'B is a better athlete.', \\\"B doesn't count as an athlete because training precludes athleticism.\\\", 'A and B are equally good athletes.']\",\"question\":\" Suppose two athletes, A and B, are equally skilled. Suppose also that A naturally inherited certain traits from her father that made playing the sport easier for her, and thus she didn't have to train as hard as B. According to the ethic of willfulness,\",\"src\":\"ori_mmlu-moral_disputes\"}","display_format":"text","language":"","answer_status":"published","assets":[],"source_url":"https://huggingface.co/datasets/TIGER-Lab/MMLU-Pro","history":"initial import","indexing_mode":"index","subproblems":[],"grids":[]}