{"kind":"task","effective_mode":"full","benchmark":{"kind":"benchmark","effective_mode":"full","slug":"mmlu-pro","formal_name":"MMLU-Pro","introduction":"MMLU-Pro rebuilds MMLU with ten answer options instead of four and removes items that no longer separate models. Questions span fourteen academic and professional subjects.","introduction_ja":"","introduction_en":"","category":"Category not supplied","task_count":null,"acquisition_status":"Acquisition status not supplied","official_url":"https://huggingface.co/datasets/TIGER-Lab/MMLU-Pro","indexing_mode":"index","profile":{"resources":[],"task_format":"","scoring":"","metric":"","size":"","answer_access":"","license":"","citation":"","maintainer":"","released":"","why_hard":"","related":[],"history":"","construction":"","limitations":"","saturation":"","scores":[]}},"task_id":"dcf57ad0-d2cc-50b2-b49f-5000d416bfc9","task_key":"default--test--10466","task_revision_id":"1","upstream_id":"10466","short_description":"What are the various methods by which PL/I debugs or notesdefects in a program…","config":"default","split":"test","body":"{\"category\":\"computer science\",\"options\":\"['Runtime error handling, Code annotations, Logic flow tracing', 'Real-time performance monitoring, Assert statements, Stack trace analysis', 'Syntax checking, Code optimization, Memory allocation', 'Syntax highlighting, Code formatting, Version control tracking', 'Automated unit tests, Integration testing, Regression testing', 'Conditional breakpoints, Watch variables, Call stack inspection', 'Profiling, Code coverage analysis, Static code analysis', 'CONVERSION condition, Overflow and underflow, Division by zero, Indicating the end of input data, ON statement', 'Code compilation, Linking, Loading', 'Error messages, Code review, Testing']\",\"question\":\"What are the various methods by which PL/I debugs or notesdefects in a program while processing it?\",\"src\":\"stemez-ComputerScience\"}","display_format":"text","language":"","answer_status":"published","assets":[],"source_url":"https://huggingface.co/datasets/TIGER-Lab/MMLU-Pro","history":"initial import","indexing_mode":"index","subproblems":[],"grids":[]}