{"kind":"task","effective_mode":"full","benchmark":{"kind":"benchmark","effective_mode":"full","slug":"mmlu-pro","formal_name":"MMLU-Pro","introduction":"MMLU-Pro rebuilds MMLU with ten answer options instead of four and removes items that no longer separate models. Questions span fourteen academic and professional subjects.","introduction_ja":"","introduction_en":"","category":"Category not supplied","task_count":null,"acquisition_status":"Acquisition status not supplied","official_url":"https://huggingface.co/datasets/TIGER-Lab/MMLU-Pro","indexing_mode":"index","profile":{"resources":[],"task_format":"","scoring":"","metric":"","size":"","answer_access":"","license":"","citation":"","maintainer":"","released":"","why_hard":"","related":[],"history":"","construction":"","limitations":"","saturation":"","scores":[]}},"task_id":"d500e535-f33a-50be-a061-353ddd4cc511","task_key":"default--test--10839","task_revision_id":"1","upstream_id":"10839","short_description":"In a faulty analogy, the things being compared are...","config":"default","split":"test","body":"{\"category\":\"philosophy\",\"options\":\"['Completely unrelated in any way', 'Similar in some ways, different in others', 'Similar in appearance, but not in function', 'Sufficiently similar', 'Not sufficiently similar in relevant ways', 'Different in appearance, but similar in function', 'Completely identical in all ways', 'Completely dissimilar in all ways', 'Complete opposites']\",\"question\":\"In a faulty analogy, the things being compared are...\",\"src\":\"ori_mmlu-logical_fallacies\"}","display_format":"text","language":"","answer_status":"published","assets":[],"source_url":"https://huggingface.co/datasets/TIGER-Lab/MMLU-Pro","history":"initial import","indexing_mode":"index","subproblems":[],"grids":[]}