{"kind":"task","effective_mode":"full","benchmark":{"kind":"benchmark","effective_mode":"full","slug":"mmlu-pro","formal_name":"MMLU-Pro","introduction":"MMLU-Pro rebuilds MMLU with ten answer options instead of four and removes items that no longer separate models. Questions span fourteen academic and professional subjects.","introduction_ja":"","introduction_en":"","category":"Category not supplied","task_count":null,"acquisition_status":"Acquisition status not supplied","official_url":"https://huggingface.co/datasets/TIGER-Lab/MMLU-Pro","indexing_mode":"index","profile":{"resources":[],"task_format":"","scoring":"","metric":"","size":"","answer_access":"","license":"","citation":"","maintainer":"","released":"","why_hard":"","related":[],"history":"","construction":"","limitations":"","saturation":"","scores":[]}},"task_id":"dfce7afa-950d-5d5f-93d8-b5c70b0a2e83","task_key":"default--test--1401","task_revision_id":"1","upstream_id":"1401","short_description":"A relevant fact in a plaintiff's suit is the magnitude of an explosion at an oil…","config":"default","split":"test","body":"{\"category\":\"law\",\"options\":\"['admissible, because the son is available as a witness.', \\\"admissible, if the son's testimony corroborates the plaintiff's statement.\\\", 'inadmissible, because the statement can be considered secondhand information.', 'admissible, because it gives context to the magnitude of the explosion.', 'inadmissible under the excited utterance exception, because it can be a product of reflection and deliberation.', 'admissible, because it provides relevant details about the incident.', 'admissible, whether or not the son is available as a witness.', \\\"inadmissible, because the plaintiff's statement is based on the son's account.\\\", 'inadmissible as hearsay not within any recognized exception.', \\\"inadmissible, unless the son's account is presented as evidence.\\\"]\",\"question\":\"A relevant fact in a plaintiff's suit is the magnitude of an explosion at an oil refinery one evening. At trial, the plaintiff was asked on direct examination if he remembered the explosion. He replied, \\\"I recall my son running into the home and screaming that the car windows just shattered. \\\" The son was available to testify at trial. The defendant's attorney makes a motion to strike the plaintiff's answer. The trial judge should rule the plaintiff's testimony\",\"src\":\"ori_mmlu-professional_law\"}","display_format":"text","language":"","answer_status":"published","assets":[],"source_url":"https://huggingface.co/datasets/TIGER-Lab/MMLU-Pro","history":"initial import","indexing_mode":"index","subproblems":[],"grids":[]}