{"kind":"task","effective_mode":"full","benchmark":{"kind":"benchmark","effective_mode":"full","slug":"mmlu-pro","formal_name":"MMLU-Pro","introduction":"MMLU-Pro rebuilds MMLU with ten answer options instead of four and removes items that no longer separate models. Questions span fourteen academic and professional subjects.","introduction_ja":"","introduction_en":"","category":"Category not supplied","task_count":null,"acquisition_status":"Acquisition status not supplied","official_url":"https://huggingface.co/datasets/TIGER-Lab/MMLU-Pro","indexing_mode":"index","profile":{"resources":[],"task_format":"","scoring":"","metric":"","size":"","answer_access":"","license":"","citation":"","maintainer":"","released":"","why_hard":"","related":[],"history":"","construction":"","limitations":"","saturation":"","scores":[]}},"task_id":"31019373-ee8e-5463-b7d9-91e0538624b0","task_key":"default--test--11173","task_revision_id":"1","upstream_id":"11173","short_description":"At the end of the course, Professor Jones told us that we were the best class he…","config":"default","split":"test","body":"{\"category\":\"philosophy\",\"options\":\"['Appeal to Gallery (Provicialism)', 'Fallacy of Composition', 'Red Herring', 'Appeal to Authority', 'Straw Man', 'Fallacy of Division', 'Slippery Slope', 'Appeal to Ignorance', 'False Dilemma', 'Questionable Cause']\",\"question\":\"At the end of the course, Professor Jones told us that we were the best class he had ever had. That's why I'm sure that the 0.0 the appeared on my grade report is a mistake. Surely, if we were as good a class as Professor Jones said we were, none of us deserved to fail.\",\"src\":\"ori_mmlu-logical_fallacies\"}","display_format":"text","language":"","answer_status":"published","assets":[],"source_url":"https://huggingface.co/datasets/TIGER-Lab/MMLU-Pro","history":"initial import","indexing_mode":"index","subproblems":[],"grids":[]}