{"kind":"task","effective_mode":"full","benchmark":{"kind":"benchmark","effective_mode":"full","slug":"mmlu-pro","formal_name":"MMLU-Pro","introduction":"MMLU-Pro rebuilds MMLU with ten answer options instead of four and removes items that no longer separate models. Questions span fourteen academic and professional subjects.","introduction_ja":"","introduction_en":"","category":"Category not supplied","task_count":null,"acquisition_status":"Acquisition status not supplied","official_url":"https://huggingface.co/datasets/TIGER-Lab/MMLU-Pro","indexing_mode":"index","profile":{"resources":[],"task_format":"","scoring":"","metric":"","size":"","answer_access":"","license":"","citation":"","maintainer":"","released":"","why_hard":"","related":[],"history":"","construction":"","limitations":"","saturation":"","scores":[]}},"task_id":"eb19e03c-dd05-5f97-aec9-f5aeb9313ea2","task_key":"default--test--10659","task_revision_id":"1","upstream_id":"10659","short_description":"Two processors, M-5 and M-7, implement the same instruction set. Processor M-5…","config":"default","split":"test","body":"{\"category\":\"computer science\",\"options\":\"['All three statements are true', 'None of the statements are true', 'I only', 'II only', 'I and II', 'I and II only', 'II and III only', 'II and III', 'I and III', 'III only']\",\"question\":\"Two processors, M-5 and M-7, implement the same instruction set. Processor M-5 uses a 5-stage pipeline and a clock cycle of 10 nanoseconds. Processor M-7 uses a 7-stage pipeline and a clock cycle of 7.5 nanoseconds. Which of the following is (are) true?\\nI. M-7’s pipeline has better maximum throughput than M-5’s pipeline.\\nII. The latency of a single instruction is shorter on M-7’s pipeline than on M-5’s pipeline.\\nIII. Programs executing on M-7 will always run faster than programs executing on M-5.\",\"src\":\"ori_mmlu-college_computer_science\"}","display_format":"text","language":"","answer_status":"published","assets":[],"source_url":"https://huggingface.co/datasets/TIGER-Lab/MMLU-Pro","history":"initial import","indexing_mode":"index","subproblems":[],"grids":[]}