{"kind":"task","effective_mode":"full","benchmark":{"kind":"benchmark","effective_mode":"full","slug":"mmlu-pro","formal_name":"MMLU-Pro","introduction":"MMLU-Pro rebuilds MMLU with ten answer options instead of four and removes items that no longer separate models. Questions span fourteen academic and professional subjects.","introduction_ja":"","introduction_en":"","category":"Category not supplied","task_count":null,"acquisition_status":"Acquisition status not supplied","official_url":"https://huggingface.co/datasets/TIGER-Lab/MMLU-Pro","indexing_mode":"index","profile":{"resources":[],"task_format":"","scoring":"","metric":"","size":"","answer_access":"","license":"","citation":"","maintainer":"","released":"","why_hard":"","related":[],"history":"","construction":"","limitations":"","saturation":"","scores":[]}},"task_id":"d3ec1601-1e79-5e0b-bdb5-05e76c5a4118","task_key":"default--test--2267","task_revision_id":"1","upstream_id":"2267","short_description":"A psychiatrist sends a signed patient release form to a psychologist requesting…","config":"default","split":"test","body":"{\"category\":\"psychology\",\"options\":\"['ignore the request as it is outside their responsibility', 'request a new patient release form from the psychiatrist', 'send the report with a covering letter indicating thar the report may now be obsolete', 'send an updated report instead of the original one', 'write a letter to the psychiatrist refusing to send the report because too much rime bas elapsed', 'send the report but withhold any sensitive information', 'refuse the request as it violates patient confidentiality', 'send the report as requested', 'consult with a legal team before sending the report', 'contact the patient directly prior to sending the report']\",\"question\":\"A psychiatrist sends a signed patient release form to a psychologist requesting a copy of a report on a patient who was assessed 7 years earlier. The psychologist should\",\"src\":\"ori_mmlu-professional_psychology\"}","display_format":"text","language":"","answer_status":"published","assets":[],"source_url":"https://huggingface.co/datasets/TIGER-Lab/MMLU-Pro","history":"initial import","indexing_mode":"index","subproblems":[],"grids":[]}