{"kind":"task","effective_mode":"full","benchmark":{"kind":"benchmark","effective_mode":"full","slug":"mmlu-pro","formal_name":"MMLU-Pro","introduction":"MMLU-Pro rebuilds MMLU with ten answer options instead of four and removes items that no longer separate models. Questions span fourteen academic and professional subjects.","introduction_ja":"","introduction_en":"","category":"Category not supplied","task_count":null,"acquisition_status":"Acquisition status not supplied","official_url":"https://huggingface.co/datasets/TIGER-Lab/MMLU-Pro","indexing_mode":"index","profile":{"resources":[],"task_format":"","scoring":"","metric":"","size":"","answer_access":"","license":"","citation":"","maintainer":"","released":"","why_hard":"","related":[],"history":"","construction":"","limitations":"","saturation":"","scores":[]}},"task_id":"f5ba17d6-551b-5491-b7df-4eb900a33fe0","task_key":"default--test--10622","task_revision_id":"1","upstream_id":"10622","short_description":"Which of the following is (are) true about virtual memory systems that use…","config":"default","split":"test","body":"{\"category\":\"computer science\",\"options\":\"['I and III', 'None of the above', 'All of the above', 'I and II and not III', 'I, II and III', 'III only', 'I only', 'II and III', 'I and II', 'II only']\",\"question\":\"Which of the following is (are) true about virtual memory systems that use pages?\\nI. The virtual address space can be larger than the amount of physical memory.\\nII. Programs must be resident in main memory throughout their execution.\\nIII. Pages correspond to semantic characteristics of the program.\",\"src\":\"ori_mmlu-college_computer_science\"}","display_format":"text","language":"","answer_status":"published","assets":[],"source_url":"https://huggingface.co/datasets/TIGER-Lab/MMLU-Pro","history":"initial import","indexing_mode":"index","subproblems":[],"grids":[]}