{"kind":"task","effective_mode":"full","benchmark":{"kind":"benchmark","effective_mode":"full","slug":"mmlu-pro","formal_name":"MMLU-Pro","introduction":"MMLU-Pro rebuilds MMLU with ten answer options instead of four and removes items that no longer separate models. Questions span fourteen academic and professional subjects.","introduction_ja":"","introduction_en":"","category":"Category not supplied","task_count":null,"acquisition_status":"Acquisition status not supplied","official_url":"https://huggingface.co/datasets/TIGER-Lab/MMLU-Pro","indexing_mode":"index","profile":{"resources":[],"task_format":"","scoring":"","metric":"","size":"","answer_access":"","license":"","citation":"","maintainer":"","released":"","why_hard":"","related":[],"history":"","construction":"","limitations":"","saturation":"","scores":[]}},"task_id":"8afe023b-e128-5f38-9806-e97a622eef70","task_key":"default--test--1550","task_revision_id":"1","upstream_id":"1550","short_description":"A national association brought suit in federal court against a city. The suit…","config":"default","split":"test","body":"{\"category\":\"law\",\"options\":\"['the privilege law of the state where the national association is located.', 'the privilege law of the state where the city is located.', 'either federal privilege law or the privilege law of the forum state as determined by the court.', 'city’s privilege law.', 'either federal privilege law or the privilege law of the city as determined by the court.', 'either federal privilege law or the privilege law of the forum state, whichever the court determines is more likely to admit the evidence.', 'federal privilege law.', 'federal privilege law or international privilege law, as determined by the court.', 'the privilege law of the forum state.', 'international privilege law.']\",\"question\":\"A national association brought suit in federal court against a city. The suit charged the city with violating several federal statutes. During the trial, the association's attorney called the city attorney as an adverse witness. The city attorney asserted the attorney-client privilege and objected that the matters communicated to him were intended to be confidential. The issues raised by the objections of the city attorney should be resolved under\",\"src\":\"ori_mmlu-professional_law\"}","display_format":"text","language":"","answer_status":"published","assets":[],"source_url":"https://huggingface.co/datasets/TIGER-Lab/MMLU-Pro","history":"initial import","indexing_mode":"index","subproblems":[],"grids":[]}