{"kind":"task","effective_mode":"full","benchmark":{"kind":"benchmark","effective_mode":"full","slug":"mmlu-pro","formal_name":"MMLU-Pro","introduction":"MMLU-Pro rebuilds MMLU with ten answer options instead of four and removes items that no longer separate models. Questions span fourteen academic and professional subjects.","introduction_ja":"","introduction_en":"","category":"Category not supplied","task_count":null,"acquisition_status":"Acquisition status not supplied","official_url":"https://huggingface.co/datasets/TIGER-Lab/MMLU-Pro","indexing_mode":"index","profile":{"resources":[],"task_format":"","scoring":"","metric":"","size":"","answer_access":"","license":"","citation":"","maintainer":"","released":"","why_hard":"","related":[],"history":"","construction":"","limitations":"","saturation":"","scores":[]}},"task_id":"38da1f20-784d-5d46-ad29-14fc174ad405","task_key":"default--test--10468","task_revision_id":"1","upstream_id":"10468","short_description":"A sign of a good computer programmer is the ability to write a program which…","config":"default","split":"test","body":"{\"category\":\"computer science\",\"options\":\"['They optimize the program by automatically correcting coding errors.', 'They allow for faster execution of programs.', 'The statements create a cloud-based backup of the program for additional storage.', 'The DSECT and CSECT statements allow for the definition and use of many variables without allocating storage space for them all at once, minimizing storage usage.', 'The statements divide the program into smaller, independent sections that can be loaded as needed.', 'They enable multithreading capabilities.', 'DSECT and CSECT statements provide a framework for artificial intelligence integration.', 'They increase the program’s compatibility with various operating systems.', 'They streamline the user input process to reduce the amount of code required.', 'They enhance the graphic interface of the program.']\",\"question\":\"A sign of a good computer programmer is the ability to write a program which minimizes storage usage. The IBM 360/370 computers provide us with the DSECT and CSECT statements . How do they enable us to becomebetter programmers ?\",\"src\":\"stemez-ComputerScience\"}","display_format":"text","language":"","answer_status":"published","assets":[],"source_url":"https://huggingface.co/datasets/TIGER-Lab/MMLU-Pro","history":"initial import","indexing_mode":"index","subproblems":[],"grids":[]}