{"kind":"task","effective_mode":"full","benchmark":{"kind":"benchmark","effective_mode":"full","slug":"mmlu-pro","formal_name":"MMLU-Pro","introduction":"MMLU-Pro rebuilds MMLU with ten answer options instead of four and removes items that no longer separate models. Questions span fourteen academic and professional subjects.","introduction_ja":"","introduction_en":"","category":"Category not supplied","task_count":null,"acquisition_status":"Acquisition status not supplied","official_url":"https://huggingface.co/datasets/TIGER-Lab/MMLU-Pro","indexing_mode":"index","profile":{"resources":[],"task_format":"","scoring":"","metric":"","size":"","answer_access":"","license":"","citation":"","maintainer":"","released":"","why_hard":"","related":[],"history":"","construction":"","limitations":"","saturation":"","scores":[]}},"task_id":"f7a2f0ad-8757-5d4c-9250-2a5840e1b430","task_key":"default--test--10748","task_revision_id":"1","upstream_id":"10748","short_description":"Determine the output of the following program READ (2, 3) I, J 3FORMAT (2I11) IF…","config":"default","split":"test","body":"{\"category\":\"computer science\",\"options\":\"['Prints the absolute value of the difference of two numbers', 'Prints out the result of multiplying the two numbers', 'Prints the second number twice', 'Calculates the difference of two numbers', 'Calculates the sum of two numbers', 'Prints out the largest of two numbers', 'Exits without printing anything', 'Prints out the smallest of two numbers', 'Prints out the result of dividing the first number by the second', 'Prints both numbers on the same line']\",\"question\":\"Determine the output of the following program READ (2, 3) I, J 3FORMAT (2I11) IF (I - J) 5,6,6 6WRITE (3,10) I 10FORMAT (1X, I11) GO TO 7 5WRITE (3,10) J 7STOP\",\"src\":\"stemez-ComputerScience\"}","display_format":"text","language":"","answer_status":"published","assets":[],"source_url":"https://huggingface.co/datasets/TIGER-Lab/MMLU-Pro","history":"initial import","indexing_mode":"index","subproblems":[],"grids":[]}