{"kind":"task","effective_mode":"full","benchmark":{"kind":"benchmark","effective_mode":"full","slug":"mmlu-pro","formal_name":"MMLU-Pro","introduction":"MMLU-Pro rebuilds MMLU with ten answer options instead of four and removes items that no longer separate models. Questions span fourteen academic and professional subjects.","introduction_ja":"","introduction_en":"","category":"Category not supplied","task_count":null,"acquisition_status":"Acquisition status not supplied","official_url":"https://huggingface.co/datasets/TIGER-Lab/MMLU-Pro","indexing_mode":"index","profile":{"resources":[],"task_format":"","scoring":"","metric":"","size":"","answer_access":"","license":"","citation":"","maintainer":"","released":"","why_hard":"","related":[],"history":"","construction":"","limitations":"","saturation":"","scores":[]}},"task_id":"965700c1-a426-57cf-b676-209a925a8c7e","task_key":"default--test--11013","task_revision_id":"1","upstream_id":"11013","short_description":"\" _Ad lazarum_ \" is","config":"default","split":"test","body":"{\"category\":\"philosophy\",\"options\":\"['a specific type of false sign that argues there is necessarily an inverse relationship between cost and quality, so something is necessarily a better value because it costs less', 'arguing that what is true of an entire object must also be true of every individual part of that object', 'a specific type of false analogy that reasons what is true about nature must be true about humans', 'drawing a conclusion without enough evidence to support that conclusion']\",\"question\":\"\\\" _Ad lazarum_ \\\" is\",\"src\":\"ori_mmlu-logical_fallacies\"}","display_format":"text","language":"","answer_status":"published","assets":[],"source_url":"https://huggingface.co/datasets/TIGER-Lab/MMLU-Pro","history":"initial import","indexing_mode":"index","subproblems":[],"grids":[]}