{"kind":"task","effective_mode":"full","benchmark":{"kind":"benchmark","effective_mode":"full","slug":"mmlu-pro","formal_name":"MMLU-Pro","introduction":"MMLU-Pro rebuilds MMLU with ten answer options instead of four and removes items that no longer separate models. Questions span fourteen academic and professional subjects.","introduction_ja":"","introduction_en":"","category":"Category not supplied","task_count":null,"acquisition_status":"Acquisition status not supplied","official_url":"https://huggingface.co/datasets/TIGER-Lab/MMLU-Pro","indexing_mode":"index","profile":{"resources":[],"task_format":"","scoring":"","metric":"","size":"","answer_access":"","license":"","citation":"","maintainer":"","released":"","why_hard":"","related":[],"history":"","construction":"","limitations":"","saturation":"","scores":[]}},"task_id":"c9258cee-f1ac-5aaf-a0fb-05e9bf7abe5f","task_key":"default--test--1548","task_revision_id":"1","upstream_id":"1548","short_description":"Two weeks before an election, a local newspaper published an article implying…","config":"default","split":"test","body":"{\"category\":\"law\",\"options\":\"['excluded, as it can potentially bias the jury.', 'excluded, because it is hearsay evidence.', \\\"admitted, as it supports the newspaper's defense of truth.\\\", 'excluded, because it is improper character evidence.', \\\"admitted as relevant evidence of the candidate's character as a thief.\\\", 'excluded, because the act was committed too long ago to be relevant.', \\\"admitted as probative evidence of the candidate's character for veracity.\\\", \\\"admitted, as it is relevant to the candidate's credibility.\\\", 'admitted, but only if the witness can provide conclusive proof of the larceny.', 'excluded, because bad acts may not be proved by extrinsic evidence.']\",\"question\":\"Two weeks before an election, a local newspaper published an article implying that one of the mayoral candidates was a thief. The candidate subsequently lost the election and sued the newspaper for defamation. The newspaper defended on the grounds of truth. At trial, the candidate took the stand and testified that he was not a thief. Thereafter, the newspaper called a witness to testify that three years ago the candidate committed a larceny while employed at his former job. Upon objection by the candidate's attorney, the witness's testimony should be\",\"src\":\"ori_mmlu-professional_law\"}","display_format":"text","language":"","answer_status":"published","assets":[],"source_url":"https://huggingface.co/datasets/TIGER-Lab/MMLU-Pro","history":"initial import","indexing_mode":"index","subproblems":[],"grids":[]}