{"kind":"task","effective_mode":"full","benchmark":{"kind":"benchmark","effective_mode":"full","slug":"mmlu-pro","formal_name":"MMLU-Pro","introduction":"MMLU-Pro rebuilds MMLU with ten answer options instead of four and removes items that no longer separate models. Questions span fourteen academic and professional subjects.","introduction_ja":"","introduction_en":"","category":"Category not supplied","task_count":null,"acquisition_status":"Acquisition status not supplied","official_url":"https://huggingface.co/datasets/TIGER-Lab/MMLU-Pro","indexing_mode":"index","profile":{"resources":[],"task_format":"","scoring":"","metric":"","size":"","answer_access":"","license":"","citation":"","maintainer":"","released":"","why_hard":"","related":[],"history":"","construction":"","limitations":"","saturation":"","scores":[]}},"task_id":"e466e33d-e950-56bd-b8aa-7e8f33cce694","task_key":"default--test--1763","task_revision_id":"1","upstream_id":"1763","short_description":"When a reservation is considered as invalid under the law of treaties?","config":"default","split":"test","body":"{\"category\":\"law\",\"options\":\"['A reservation is invalid when the majority of the State parties objects to it', 'A reservation is invalid only when an international tribunal says so', 'A reservation is invalid only when is incompatible with a peremptory norm of international law (jus cogens)', 'A reservation is invalid when it is incompatible with the object and purpose of the treaty']\",\"question\":\"When a reservation is considered as invalid under the law of treaties?\",\"src\":\"ori_mmlu-international_law\"}","display_format":"text","language":"","answer_status":"published","assets":[],"source_url":"https://huggingface.co/datasets/TIGER-Lab/MMLU-Pro","history":"initial import","indexing_mode":"index","subproblems":[],"grids":[]}