{"kind":"task","effective_mode":"full","benchmark":{"kind":"benchmark","effective_mode":"full","slug":"mmlu-pro","formal_name":"MMLU-Pro","introduction":"MMLU-Pro rebuilds MMLU with ten answer options instead of four and removes items that no longer separate models. Questions span fourteen academic and professional subjects.","introduction_ja":"","introduction_en":"","category":"Category not supplied","task_count":null,"acquisition_status":"Acquisition status not supplied","official_url":"https://huggingface.co/datasets/TIGER-Lab/MMLU-Pro","indexing_mode":"index","profile":{"resources":[],"task_format":"","scoring":"","metric":"","size":"","answer_access":"","license":"","citation":"","maintainer":"","released":"","why_hard":"","related":[],"history":"","construction":"","limitations":"","saturation":"","scores":[]}},"task_id":"2c87993e-4a6b-5b31-bdd3-b57c38dbc3d4","task_key":"default--test--2625","task_revision_id":"1","upstream_id":"2625","short_description":"A graduate student designed and conducted a research project for her…","config":"default","split":"test","body":"{\"category\":\"psychology\",\"options\":\"['authorship should be determined by the dean of the school.', 'the student and advisor should decide authorship through a coin flip.', 'the student should file a complaint against the advisor with the Ethics Committee.', 'the student should abandon the project to avoid conflict.', 'it is up to the student to determine authorship credit.', 'the advisor should be listed as the only author.', 'the student should be listed as the first author.', 'it is up to the advisor to determine authorship credit.', 'both the student and advisor should be listed as co-first authors.', 'the advisor should be listed as the first author.']\",\"question\":\"A graduate student designed and conducted a research project for her dissertation and subsequently wrote a journal article describing the study and its results. The original idea for the study was derived from the work of the student’s faculty advisor who also provided the facilities for the student's research. The faculty advisor wants to be listed as first author on the article when it is published. In this situation:\",\"src\":\"ori_mmlu-professional_psychology\"}","display_format":"text","language":"","answer_status":"published","assets":[],"source_url":"https://huggingface.co/datasets/TIGER-Lab/MMLU-Pro","history":"initial import","indexing_mode":"index","subproblems":[],"grids":[]}