{"kind":"task","effective_mode":"full","benchmark":{"kind":"benchmark","effective_mode":"full","slug":"mmlu-pro","formal_name":"MMLU-Pro","introduction":"MMLU-Pro rebuilds MMLU with ten answer options instead of four and removes items that no longer separate models. Questions span fourteen academic and professional subjects.","introduction_ja":"","introduction_en":"","category":"Category not supplied","task_count":null,"acquisition_status":"Acquisition status not supplied","official_url":"https://huggingface.co/datasets/TIGER-Lab/MMLU-Pro","indexing_mode":"index","profile":{"resources":[],"task_format":"","scoring":"","metric":"","size":"","answer_access":"","license":"","citation":"","maintainer":"","released":"","why_hard":"","related":[],"history":"","construction":"","limitations":"","saturation":"","scores":[]}},"task_id":"b1d93abf-9ae8-582e-b5a7-2ba66679baad","task_key":"default--test--10656","task_revision_id":"1","upstream_id":"10656","short_description":"Which of the following statements about horizontal versus vertical…","config":"default","split":"test","body":"{\"category\":\"computer science\",\"options\":\"['I and III only', 'I, II and III', 'III only', 'I and II only', 'All of the above', 'II only', 'II and III only', 'I only', 'None of the above', 'II and III']\",\"question\":\"Which of the following statements about horizontal versus vertical microarchitecture is (are) true?\\nI. Programs for horizontal architectures require more time steps than those for vertical architectures.\\nII. Horizontal microinstructions are unencoded.\\nIII. Horizontal microinstructions usually have a single opcode and multiple operand specifiers.\",\"src\":\"ori_mmlu-college_computer_science\"}","display_format":"text","language":"","answer_status":"published","assets":[],"source_url":"https://huggingface.co/datasets/TIGER-Lab/MMLU-Pro","history":"initial import","indexing_mode":"index","subproblems":[],"grids":[]}