{"kind":"task","effective_mode":"full","benchmark":{"kind":"benchmark","effective_mode":"full","slug":"mmlu-pro","formal_name":"MMLU-Pro","introduction":"MMLU-Pro rebuilds MMLU with ten answer options instead of four and removes items that no longer separate models. Questions span fourteen academic and professional subjects.","introduction_ja":"","introduction_en":"","category":"Category not supplied","task_count":null,"acquisition_status":"Acquisition status not supplied","official_url":"https://huggingface.co/datasets/TIGER-Lab/MMLU-Pro","indexing_mode":"index","profile":{"resources":[],"task_format":"","scoring":"","metric":"","size":"","answer_access":"","license":"","citation":"","maintainer":"","released":"","why_hard":"","related":[],"history":"","construction":"","limitations":"","saturation":"","scores":[]}},"task_id":"d4e14d1f-d219-5303-87aa-978ded48282d","task_key":"default--test--10383","task_revision_id":"1","upstream_id":"10383","short_description":"In multiprogrammed systems it is advantageous if some programs such as editors…","config":"default","split":"test","body":"{\"category\":\"computer science\",\"options\":\"['I and III only', 'II and III are true but not necessary for sharing a program', 'None of the statements must be true', 'II and III only', 'I only', 'I, II, and III all must be true', 'I and II only', 'II only', 'The program is non-recursive and reentrant', 'III only']\",\"question\":\"In multiprogrammed systems it is advantageous if some programs such as editors and compilers can be shared by several users. Which of the following must be true of multiprogrammed systems in order that a single copy of a program can be shared by several users?\\nI. The program is a macro.\\nII. The program is recursive.\\nIII. The program is reentrant.\",\"src\":\"ori_mmlu-college_computer_science\"}","display_format":"text","language":"","answer_status":"published","assets":[],"source_url":"https://huggingface.co/datasets/TIGER-Lab/MMLU-Pro","history":"initial import","indexing_mode":"index","subproblems":[],"grids":[]}