{"kind":"task","effective_mode":"full","benchmark":{"kind":"benchmark","effective_mode":"full","slug":"mmlu-pro","formal_name":"MMLU-Pro","introduction":"MMLU-Pro rebuilds MMLU with ten answer options instead of four and removes items that no longer separate models. Questions span fourteen academic and professional subjects.","introduction_ja":"","introduction_en":"","category":"Category not supplied","task_count":null,"acquisition_status":"Acquisition status not supplied","official_url":"https://huggingface.co/datasets/TIGER-Lab/MMLU-Pro","indexing_mode":"index","profile":{"resources":[],"task_format":"","scoring":"","metric":"","size":"","answer_access":"","license":"","citation":"","maintainer":"","released":"","why_hard":"","related":[],"history":"","construction":"","limitations":"","saturation":"","scores":[]}},"task_id":"fcf5a26f-11c5-5580-82c2-c0997756bd78","task_key":"default--test--10614","task_revision_id":"1","upstream_id":"10614","short_description":"The following BASIC program is supposed to calculate the sum of the squares of…","config":"default","split":"test","body":"{\"category\":\"computer science\",\"options\":\"['Statements 10 and 20 are prefaced by REM, SUM is an invalid variable name, 60IF J = 11 THEN GO, END is succeeded by a period, squares have been calculated', 'Statements 10 and 20 are not prefaced by REM, SUM is an invalid variable name, 60IF X = 10 THEN GO, END is succeeded by a period, squares have not been calculated', 'Statements 30 and 40 are prefaced by REM, SUM is a valid variable name, 60IF J = 10 THEN GO, END is not succeeded by a period, no error in the sum calculation', 'Statements 30 and 40 are not prefaced by REM, SUMM is an invalid variable name, 60IF X = 10 THEN GO, END is not succeeded by a period, squares have been calculated', 'Statements 30 and 40 are not prefaced by REM, SUM is a valid variable name, 60IF J = 11 THEN GO, END is not succeeded by a period, squares have not been calculated', 'Statements 10 and 20 are prefaced by REM, SUM is a valid variable name, 60IF X = 10 THEN GO, END is succeeded by a period, squares have not been calculated', 'No statements are prefaced by REM, SUM is a valid variable name, 60IF J = 11 THEN GO, END is not succeeded by a period, squares have been calculated', 'Statements 10 and 20 are not prefaced by REM, SUM is an invalid variable name, 60IF X = 10 THEN GO, END is not succeeded by a period, squares have been calculated', 'Statements 10 and 20 are not prefaced by REM, SUM is a valid variable name, 60IF J = 9 THEN GO, END is succeeded by a period, squares have not been calculated', 'No statements are prefaced by REM, SUM is a valid variable name, 60IF J = 10 THEN GO, END is succeeded by a period, no error in the sum calculation']\",\"question\":\"The following BASIC program is supposed to calculate the sum of the squares of the first 10 digits. Find the syntactical and logical errors. 10PROGRAM TO CALCULATE SUM OF SQUARES OF 20FIRST TEN DIGITS. 30LET SUM = 0 40LET J = 1 50LET S = J 60IF J = 10 THEN GO TO 90 70LET J = J + 1 80GO TO 50 90PRINT S 100END.\",\"src\":\"stemez-ComputerScience\"}","display_format":"text","language":"","answer_status":"published","assets":[],"source_url":"https://huggingface.co/datasets/TIGER-Lab/MMLU-Pro","history":"initial import","indexing_mode":"index","subproblems":[],"grids":[]}