{"kind":"task","effective_mode":"full","benchmark":{"kind":"benchmark","effective_mode":"full","slug":"mmlu-pro","formal_name":"MMLU-Pro","introduction":"MMLU-Pro rebuilds MMLU with ten answer options instead of four and removes items that no longer separate models. Questions span fourteen academic and professional subjects.","introduction_ja":"","introduction_en":"","category":"Category not supplied","task_count":null,"acquisition_status":"Acquisition status not supplied","official_url":"https://huggingface.co/datasets/TIGER-Lab/MMLU-Pro","indexing_mode":"index","profile":{"resources":[],"task_format":"","scoring":"","metric":"","size":"","answer_access":"","license":"","citation":"","maintainer":"","released":"","why_hard":"","related":[],"history":"","construction":"","limitations":"","saturation":"","scores":[]}},"task_id":"6a7d5c4a-9489-53c8-9fb9-562f79665f17","task_key":"default--test--1840","task_revision_id":"1","upstream_id":"1840","short_description":"A man is on trial for securities fraud. He takes the stand to testify in his own…","config":"default","split":"test","body":"{\"category\":\"law\",\"options\":\"['The defendant may introduce evidence to show good moral character.', 'The defendant may not introduce evidence to show honesty because character evidence is inadmissible.', 'The defendant may introduce evidence of his past good deeds.', 'The defendant may introduce evidence to show honesty.', 'The defendant may introduce evidence to show truthfulness.', 'The defendant may not introduce evidence of truthfulness unless it is inquired into on cross- examination.', 'The defendant may not introduce evidence of good character unless the prosecution first attacks his character.', 'The defendant may not introduce evidence to show honesty unless it is inquired into on cross- examination.', 'The defendant may introduce evidence to show dishonesty.']\",\"question\":\"A man is on trial for securities fraud. He takes the stand to testify in his own behalf. On direct examination, the man's attorney proffers evidence of his good character. Upon objection by the prosecution, which of the following statements is correct?\",\"src\":\"ori_mmlu-professional_law\"}","display_format":"text","language":"","answer_status":"published","assets":[],"source_url":"https://huggingface.co/datasets/TIGER-Lab/MMLU-Pro","history":"initial import","indexing_mode":"index","subproblems":[],"grids":[]}