{"kind":"task","effective_mode":"full","benchmark":{"kind":"benchmark","effective_mode":"full","slug":"mmlu-pro","formal_name":"MMLU-Pro","introduction":"MMLU-Pro rebuilds MMLU with ten answer options instead of four and removes items that no longer separate models. Questions span fourteen academic and professional subjects.","introduction_ja":"","introduction_en":"","category":"Category not supplied","task_count":null,"acquisition_status":"Acquisition status not supplied","official_url":"https://huggingface.co/datasets/TIGER-Lab/MMLU-Pro","indexing_mode":"index","profile":{"resources":[],"task_format":"","scoring":"","metric":"","size":"","answer_access":"","license":"","citation":"","maintainer":"","released":"","why_hard":"","related":[],"history":"","construction":"","limitations":"","saturation":"","scores":[]}},"task_id":"f936408f-640f-5489-b749-472bba3b2af5","task_key":"default--test--2515","task_revision_id":"1","upstream_id":"2515","short_description":"A psychologist is subpoenaed to testify at a deposition about a current therapy…","config":"default","split":"test","body":"{\"category\":\"psychology\",\"options\":\"[\\\"must present a detailed analysis of the client's psychological state.\\\", 'must appear at the deposition as requested.', 'must testify about the client as requested.', \\\"must provide a written statement about the client's mental health.\\\", 'can choose not to attend the deposition.', \\\"must reveal all details about the client's therapy sessions.\\\", 'is required to discontinue therapy with the client.', 'can refuse to testify on grounds of client confidentiality.', 'can expect to receive a court order within 60 days.', 'must provide the requested documents.']\",\"question\":\"A psychologist is subpoenaed to testify at a deposition about a current therapy client. This means that the psychologist:\",\"src\":\"ori_mmlu-professional_psychology\"}","display_format":"text","language":"","answer_status":"published","assets":[],"source_url":"https://huggingface.co/datasets/TIGER-Lab/MMLU-Pro","history":"initial import","indexing_mode":"index","subproblems":[],"grids":[]}