{"kind":"task","effective_mode":"full","benchmark":{"kind":"benchmark","effective_mode":"full","slug":"mmlu-pro","formal_name":"MMLU-Pro","introduction":"MMLU-Pro rebuilds MMLU with ten answer options instead of four and removes items that no longer separate models. Questions span fourteen academic and professional subjects.","introduction_ja":"","introduction_en":"","category":"Category not supplied","task_count":null,"acquisition_status":"Acquisition status not supplied","official_url":"https://huggingface.co/datasets/TIGER-Lab/MMLU-Pro","indexing_mode":"index","profile":{"resources":[],"task_format":"","scoring":"","metric":"","size":"","answer_access":"","license":"","citation":"","maintainer":"","released":"","why_hard":"","related":[],"history":"","construction":"","limitations":"","saturation":"","scores":[]}},"task_id":"d14e617e-5c73-5879-a63c-1074771173e6","task_key":"default--test--882","task_revision_id":"1","upstream_id":"882","short_description":"A man was charged with tax fraud. He wished to offer evidence, on his behalf, of…","config":"default","split":"test","body":"{\"category\":\"law\",\"options\":\"[\\\"The defendant's honesty during his political campaign has no bearing on his honesty in financial matters.\\\", 'The evidence is biased as it is coming from the defendant himself.', \\\"The man's honesty in one situation doesn't necessarily apply to all situations.\\\", 'Character evidence is not admissible in such cases.', \\\"One incident doesn't prove character.\\\", \\\"The evidence could sway the jury's opinion unfairly.\\\", 'This is an impermissible way to prove character.', 'The tax fraud case and the political campaign are two separate incidents.', 'The incident is too far in the past to be relevant.']\",\"question\":\"A man was charged with tax fraud. He wished to offer evidence, on his behalf, of his honesty while running for mayor. During his political campaign, he refused to lie about a psychiatric problem in his past, and his honesty cost him the election. The man was quoted as saying, \\\"If I lied, I'd be mayor today, but that's no way to win an election. \\\"The BEST argument of the objector to such evidence in the tax fraud case would be which of the following?\",\"src\":\"ori_mmlu-professional_law\"}","display_format":"text","language":"","answer_status":"published","assets":[],"source_url":"https://huggingface.co/datasets/TIGER-Lab/MMLU-Pro","history":"initial import","indexing_mode":"index","subproblems":[],"grids":[]}