{"kind":"task","effective_mode":"full","benchmark":{"kind":"benchmark","effective_mode":"full","slug":"mmlu-pro","formal_name":"MMLU-Pro","introduction":"MMLU-Pro rebuilds MMLU with ten answer options instead of four and removes items that no longer separate models. Questions span fourteen academic and professional subjects.","introduction_ja":"","introduction_en":"","category":"Category not supplied","task_count":null,"acquisition_status":"Acquisition status not supplied","official_url":"https://huggingface.co/datasets/TIGER-Lab/MMLU-Pro","indexing_mode":"index","profile":{"resources":[],"task_format":"","scoring":"","metric":"","size":"","answer_access":"","license":"","citation":"","maintainer":"","released":"","why_hard":"","related":[],"history":"","construction":"","limitations":"","saturation":"","scores":[]}},"task_id":"27c2e8cf-88d1-525f-9c42-a0cc0b7b2953","task_key":"default--test--10618","task_revision_id":"1","upstream_id":"10618","short_description":"A programmer is writing a program that is intended to be able to process large…","config":"default","split":"test","body":"{\"category\":\"computer science\",\"options\":\"['How much storage space the program requires as it runs', 'The number of users using the program simultaneously', 'How long the program takes to run', 'How many programming statements the program contains', 'The programming language in which the program is written', 'How much memory the program requires as it runs', 'The speed of the processor on the computer running the program']\",\"question\":\"A programmer is writing a program that is intended to be able to process large amounts of data. Which of the following considerations is LEAST likely to affect the ability of the program to process larger data sets?\",\"src\":\"ori_mmlu-high_school_computer_science\"}","display_format":"text","language":"","answer_status":"published","assets":[],"source_url":"https://huggingface.co/datasets/TIGER-Lab/MMLU-Pro","history":"initial import","indexing_mode":"index","subproblems":[],"grids":[]}