{"kind":"task","effective_mode":"full","benchmark":{"kind":"benchmark","effective_mode":"full","slug":"mmlu-pro","formal_name":"MMLU-Pro","introduction":"MMLU-Pro rebuilds MMLU with ten answer options instead of four and removes items that no longer separate models. Questions span fourteen academic and professional subjects.","introduction_ja":"","introduction_en":"","category":"Category not supplied","task_count":null,"acquisition_status":"Acquisition status not supplied","official_url":"https://huggingface.co/datasets/TIGER-Lab/MMLU-Pro","indexing_mode":"index","profile":{"resources":[],"task_format":"","scoring":"","metric":"","size":"","answer_access":"","license":"","citation":"","maintainer":"","released":"","why_hard":"","related":[],"history":"","construction":"","limitations":"","saturation":"","scores":[]}},"task_id":"1e49547e-f0f3-53e6-bafc-f0a4e15fe2a6","task_key":"default--test--2703","task_revision_id":"1","upstream_id":"2703","short_description":"If a psychologist acts as both a fact witness for the plaintiff and an expert…","config":"default","split":"test","body":"{\"category\":\"psychology\",\"options\":\"['unethically unless she obtains a waiver from the plaintiff.', 'ethically as long as she did not have a prior relationship with the plaintiff.', 'unethically unless she has a prior relationship with the defendant.', 'ethically as long as she maintains neutrality throughout the trial.', 'ethically only if she has a prior relationship with the defendant.', 'unethically by accepting dual roles.', 'unethically because she has a conflict of interest.', 'ethically as long as she clarifies her roles with all parties.', 'ethically as long as she does not share confidential information.', 'ethically as long as she obtains a waiver from the court.']\",\"question\":\"If a psychologist acts as both a fact witness for the plaintiff and an expert witness for the court in a criminal trial, she has acted:\",\"src\":\"ori_mmlu-professional_psychology\"}","display_format":"text","language":"","answer_status":"published","assets":[],"source_url":"https://huggingface.co/datasets/TIGER-Lab/MMLU-Pro","history":"initial import","indexing_mode":"index","subproblems":[],"grids":[]}