{"kind":"task","effective_mode":"full","benchmark":{"kind":"benchmark","effective_mode":"full","slug":"mmlu-pro","formal_name":"MMLU-Pro","introduction":"MMLU-Pro rebuilds MMLU with ten answer options instead of four and removes items that no longer separate models. Questions span fourteen academic and professional subjects.","introduction_ja":"","introduction_en":"","category":"Category not supplied","task_count":null,"acquisition_status":"Acquisition status not supplied","official_url":"https://huggingface.co/datasets/TIGER-Lab/MMLU-Pro","indexing_mode":"index","profile":{"resources":[],"task_format":"","scoring":"","metric":"","size":"","answer_access":"","license":"","citation":"","maintainer":"","released":"","why_hard":"","related":[],"history":"","construction":"","limitations":"","saturation":"","scores":[]}},"task_id":"0059d65c-8b9a-581f-b1fa-2fbf1bea0e97","task_key":"default--test--10620","task_revision_id":"1","upstream_id":"10620","short_description":"The access matrix approach to protection has the difficulty that","config":"default","split":"test","body":"{\"category\":\"computer science\",\"options\":\"['it requires an extensive amount of computational power', 'it is not capable of expressing complex protection requirements', 'the matrix, if stored directly, is large and can be clumsy to manage', 'the matrix cannot be modified once it is created', 'it does not allow for the sharing of resources between processes', 'there is no way to express who has rights to change the access matrix itself', 'it does not provide sufficient protection against external threats', 'there is no way to express the hierarchy of access rights within the matrix', 'deciding whether a process has access to a resource is undecidable', 'the matrix, if not stored directly, can cause system failure']\",\"question\":\"The access matrix approach to protection has the difficulty that\",\"src\":\"ori_mmlu-college_computer_science\"}","display_format":"text","language":"","answer_status":"published","assets":[],"source_url":"https://huggingface.co/datasets/TIGER-Lab/MMLU-Pro","history":"initial import","indexing_mode":"index","subproblems":[],"grids":[]}