{"kind":"task","effective_mode":"full","benchmark":{"kind":"benchmark","effective_mode":"full","slug":"mmlu-pro","formal_name":"MMLU-Pro","introduction":"MMLU-Pro rebuilds MMLU with ten answer options instead of four and removes items that no longer separate models. Questions span fourteen academic and professional subjects.","introduction_ja":"","introduction_en":"","category":"Category not supplied","task_count":null,"acquisition_status":"Acquisition status not supplied","official_url":"https://huggingface.co/datasets/TIGER-Lab/MMLU-Pro","indexing_mode":"index","profile":{"resources":[],"task_format":"","scoring":"","metric":"","size":"","answer_access":"","license":"","citation":"","maintainer":"","released":"","why_hard":"","related":[],"history":"","construction":"","limitations":"","saturation":"","scores":[]}},"task_id":"9a0b8332-ef81-55d5-bf5b-04395bc80073","task_key":"default--test--10412","task_revision_id":"1","upstream_id":"10412","short_description":"\"Magic memory\"' has two operations: Read and Clear. Both are indivisible and…","config":"default","split":"test","body":"{\"category\":\"computer science\",\"options\":\"['I only', 'II only', 'II and III', 'Only II and III', 'I, II and III', 'None of the above', 'All of the above', 'I and III', 'I and II', 'III only']\",\"question\":\"\\\"Magic memory\\\"' has two operations: Read and Clear. Both are indivisible and mutually exclusive. Clear sets the magic memory to zero. Read returns a value that represents the number of Read operations since the last Clear operation. Which of the following is (are) true of \\\"Magic memory\\\"?\\nI. It can provide the functionality of an atomic Test-and-Set.\\nII. It can be used to coordinate processes running on a shared-memory multiprocessor.\\nIII. It is only useful on a multiprocessor.\",\"src\":\"ori_mmlu-college_computer_science\"}","display_format":"text","language":"","answer_status":"published","assets":[],"source_url":"https://huggingface.co/datasets/TIGER-Lab/MMLU-Pro","history":"initial import","indexing_mode":"index","subproblems":[],"grids":[]}