{"kind":"task","effective_mode":"full","benchmark":{"kind":"benchmark","effective_mode":"full","slug":"mmlu-pro","formal_name":"MMLU-Pro","introduction":"MMLU-Pro rebuilds MMLU with ten answer options instead of four and removes items that no longer separate models. Questions span fourteen academic and professional subjects.","introduction_ja":"","introduction_en":"","category":"Category not supplied","task_count":null,"acquisition_status":"Acquisition status not supplied","official_url":"https://huggingface.co/datasets/TIGER-Lab/MMLU-Pro","indexing_mode":"index","profile":{"resources":[],"task_format":"","scoring":"","metric":"","size":"","answer_access":"","license":"","citation":"","maintainer":"","released":"","why_hard":"","related":[],"history":"","construction":"","limitations":"","saturation":"","scores":[]}},"task_id":"52033693-4618-5edb-a1d0-f66eb3df2670","task_key":"default--test--1972","task_revision_id":"1","upstream_id":"1972","short_description":"A man was prosecuted for assault and battery after he admitted striking a victim…","config":"default","split":"test","body":"{\"category\":\"law\",\"options\":\"[\\\"admissible, because it could sway the jury's opinion in favor of the man.\\\", \\\"inadmissible, because the neighbor's opinion is not considered reliable evidence.\\\", \\\"admissible, because it is relevant to show the improbability of the man's having committed an unprovoked attack.\\\", 'inadmissible, because it is merely the opinion of a character witness.', \\\"admissible, because it could help establish the man's reputation.\\\", \\\"admissible, because it is relevant to support the man's credibility.\\\", \\\"inadmissible, because it is not directed toward a pertinent trait of the man's character.\\\", \\\"admissible, because it shows the man's peaceful nature.\\\", \\\"inadmissible, because it does not provide direct evidence of the man's actions during the incident.\\\"]\",\"question\":\"A man was prosecuted for assault and battery after he admitted striking a victim with a pool cue during a barroom argument. The man claimed that he acted in self-defense after he was attacked by the victim, who was drunk and belligerent. As his first defense witness, the man calls his neighbor to testify that the man is a good neighbor. The neighbor's testimony is\",\"src\":\"ori_mmlu-professional_law\"}","display_format":"text","language":"","answer_status":"published","assets":[],"source_url":"https://huggingface.co/datasets/TIGER-Lab/MMLU-Pro","history":"initial import","indexing_mode":"index","subproblems":[],"grids":[]}