{"kind":"task","effective_mode":"full","benchmark":{"kind":"benchmark","effective_mode":"full","slug":"mmlu-pro","formal_name":"MMLU-Pro","introduction":"MMLU-Pro rebuilds MMLU with ten answer options instead of four and removes items that no longer separate models. Questions span fourteen academic and professional subjects.","introduction_ja":"","introduction_en":"","category":"Category not supplied","task_count":null,"acquisition_status":"Acquisition status not supplied","official_url":"https://huggingface.co/datasets/TIGER-Lab/MMLU-Pro","indexing_mode":"index","profile":{"resources":[],"task_format":"","scoring":"","metric":"","size":"","answer_access":"","license":"","citation":"","maintainer":"","released":"","why_hard":"","related":[],"history":"","construction":"","limitations":"","saturation":"","scores":[]}},"task_id":"9e169ef7-7fad-5a84-ac0f-ca6232040596","task_key":"default--test--11048","task_revision_id":"1","upstream_id":"11048","short_description":"Which of the following is not an example Aquinas gives of a self-evident…","config":"default","split":"test","body":"{\"category\":\"philosophy\",\"options\":\"['\\\"Man is a rational being.\\\"', '\\\"All bachelors are unmarried.\\\"', '\\\"Night follows day.\\\"', '\\\"The sum of the angles of a triangle is 180 degrees.\\\"', '\\\"The same thing cannot be affirmed and denied at the same time.\\\"', '\\\"The light of Thy countenance, O Lord, is signed upon us.\\\"', '\\\"A square has four sides.\\\"', '\\\"Every whole is greater than its part.\\\"', '\\\"Things equal to one and the same are equal to one another.\\\"', '\\\"An effect depends on its cause.\\\"']\",\"question\":\" Which of the following is not an example Aquinas gives of a self-evident proposition?\",\"src\":\"ori_mmlu-moral_disputes\"}","display_format":"text","language":"","answer_status":"published","assets":[],"source_url":"https://huggingface.co/datasets/TIGER-Lab/MMLU-Pro","history":"initial import","indexing_mode":"index","subproblems":[],"grids":[]}