{"kind":"task","effective_mode":"full","benchmark":{"kind":"benchmark","effective_mode":"full","slug":"mmlu-pro","formal_name":"MMLU-Pro","introduction":"MMLU-Pro rebuilds MMLU with ten answer options instead of four and removes items that no longer separate models. Questions span fourteen academic and professional subjects.","introduction_ja":"","introduction_en":"","category":"Category not supplied","task_count":null,"acquisition_status":"Acquisition status not supplied","official_url":"https://huggingface.co/datasets/TIGER-Lab/MMLU-Pro","indexing_mode":"index","profile":{"resources":[],"task_format":"","scoring":"","metric":"","size":"","answer_access":"","license":"","citation":"","maintainer":"","released":"","why_hard":"","related":[],"history":"","construction":"","limitations":"","saturation":"","scores":[]}},"task_id":"248302d3-ee57-5e26-acc4-8b1b8a9c37fe","task_key":"default--test--2531","task_revision_id":"1","upstream_id":"2531","short_description":"Complete the following example using deductive reasoning: All students who…","config":"default","split":"test","body":"{\"category\":\"psychology\",\"options\":\"['Jerome does not study calculus', 'Jerome studies Latin', 'Jerome studies either calculus or Latin, but not both', 'Jerome studies calculus', 'Jerome studies calculus but not Latin', 'Jerome studies either calculus or Latin', 'Jerome studies both calculus and Latin', 'Jerome studies neither calculus nor Latin', 'Jerome does not study Latin', 'Jerome studies both calculus and Latin, but only if he chooses to']\",\"question\":\"Complete the following example using deductive reasoning: All students who attend the Peddie School study Latin. Some students who study Latin also study calculus. Jerome is a student at the Peddie School. It can therefore be determined that\",\"src\":\"ori_mmlu-high_school_psychology\"}","display_format":"text","language":"","answer_status":"published","assets":[],"source_url":"https://huggingface.co/datasets/TIGER-Lab/MMLU-Pro","history":"initial import","indexing_mode":"index","subproblems":[],"grids":[]}