{"kind":"task","effective_mode":"full","benchmark":{"kind":"benchmark","effective_mode":"full","slug":"mmlu-pro","formal_name":"MMLU-Pro","introduction":"MMLU-Pro rebuilds MMLU with ten answer options instead of four and removes items that no longer separate models. Questions span fourteen academic and professional subjects.","introduction_ja":"","introduction_en":"","category":"Category not supplied","task_count":null,"acquisition_status":"Acquisition status not supplied","official_url":"https://huggingface.co/datasets/TIGER-Lab/MMLU-Pro","indexing_mode":"index","profile":{"resources":[],"task_format":"","scoring":"","metric":"","size":"","answer_access":"","license":"","citation":"","maintainer":"","released":"","why_hard":"","related":[],"history":"","construction":"","limitations":"","saturation":"","scores":[]}},"task_id":"49dc957e-73bf-5c87-bc32-e9a3b12f95a2","task_key":"default--test--1357","task_revision_id":"1","upstream_id":"1357","short_description":"A large man with red hair robbed a liquor store. Thereafter, a defendant was…","config":"default","split":"test","body":"{\"category\":\"law\",\"options\":\"['inadmissible, because the corrections officer was not present at the scene of the crime.', 'inadmissible, because it is irrelevant to the case.', 'inadmissible, because it is opinion testimony.', 'admissible as a character evidence.', 'admissible as a prior identification.', \\\"admissible, as it is a part of the arresting officer's routine report.\\\", 'admissible, as it is a part of the chain of evidence in the case.', 'inadmissible as hearsay not within any recognized exception.', \\\"admissible, for the limited purpose of clarifying the discrepancy in the witnesses' testimony.\\\", 'inadmissible, as it is prejudicial against the defendant.']\",\"question\":\"A large man with red hair robbed a liquor store. Thereafter, a defendant was arrested and charged with the armed robbery. At the defendant's trial, several eyewitnesses testified that they had seen a large redheaded man pull out a gun and rob the owner of the liquor store. The defendant appeared at trial with a shaven head. The prosecution calls a corrections officer to testify that the defendant had red hair when he was first brought to jail. The defendant's counsel objects. The trial judge should rule the correction officer's testimony\",\"src\":\"ori_mmlu-professional_law\"}","display_format":"text","language":"","answer_status":"published","assets":[],"source_url":"https://huggingface.co/datasets/TIGER-Lab/MMLU-Pro","history":"initial import","indexing_mode":"index","subproblems":[],"grids":[]}