{"kind":"task","effective_mode":"full","benchmark":{"kind":"benchmark","effective_mode":"full","slug":"mmlu-pro","formal_name":"MMLU-Pro","introduction":"MMLU-Pro rebuilds MMLU with ten answer options instead of four and removes items that no longer separate models. Questions span fourteen academic and professional subjects.","introduction_ja":"","introduction_en":"","category":"Category not supplied","task_count":null,"acquisition_status":"Acquisition status not supplied","official_url":"https://huggingface.co/datasets/TIGER-Lab/MMLU-Pro","indexing_mode":"index","profile":{"resources":[],"task_format":"","scoring":"","metric":"","size":"","answer_access":"","license":"","citation":"","maintainer":"","released":"","why_hard":"","related":[],"history":"","construction":"","limitations":"","saturation":"","scores":[]}},"task_id":"5ab4d72a-2259-5d14-91f0-c6c2d778f94e","task_key":"default--test--10479","task_revision_id":"1","upstream_id":"10479","short_description":"Which of the following statements about floating-point arithmetic is NOT true?","config":"default","split":"test","body":"{\"category\":\"computer science\",\"options\":\"['It is inherently nonassociative because some numbers have no exact representation.', 'All operations in floating-point arithmetic are commutative.', 'It is inherently nonassociative because there have to be upper and lower bounds on the size of numbers.', 'Associativity can be achieved with appropriate roundoff conventions.', 'Some rational numbers have no exact representation.']\",\"question\":\"Which of the following statements about floating-point arithmetic is NOT true?\",\"src\":\"ori_mmlu-college_computer_science\"}","display_format":"text","language":"","answer_status":"published","assets":[],"source_url":"https://huggingface.co/datasets/TIGER-Lab/MMLU-Pro","history":"initial import","indexing_mode":"index","subproblems":[],"grids":[]}