{"kind":"task","effective_mode":"full","benchmark":{"kind":"benchmark","effective_mode":"full","slug":"mmlu","formal_name":"MMLU","introduction":"MMLU asks four-way multiple-choice questions across 57 subjects, from elementary material to professional-level examinations. Since 2020 it has been the most widely cited single reference point for general capability.","introduction_ja":"","introduction_en":"","category":"Category not supplied","task_count":null,"acquisition_status":"Acquisition status not supplied","official_url":"https://github.com/hendrycks/test","indexing_mode":"index","profile":{"resources":[],"task_format":"","scoring":"","metric":"","size":"","answer_access":"","license":"","citation":"","maintainer":"","released":"","why_hard":"","related":[],"history":"","construction":"","limitations":"","saturation":"","scores":[]}},"task_id":"2d719b9d-62bf-5e7e-b046-88e1e3fdd417","task_key":"all--test--2d719b9d-62bf-5e7e-b046-88e1e3fdd417","task_revision_id":"1","upstream_id":"","short_description":"Suppose you toss a fair die three times and it comes up an even number each…","config":"all","split":"test","body":"{\"choices\":\"['By the law of large numbers, the next toss is more likely to be an odd number than another even number.', 'Based on the properties of conditional probability the next toss is more likely to be an even number given that three in a row have been even.', 'Dice actually do have memories, and thus the number that comes up on the next toss will be influenced by the previous tosses.', 'The probability that the next toss will again be even is 0.5.']\",\"question\":\"Suppose you toss a fair die three times and it comes up an even number each time. Which of the following is a true statement?\",\"subject\":\"high_school_statistics\"}","display_format":"text","language":"","answer_status":"published","assets":[],"source_url":"https://github.com/hendrycks/test","history":"initial import","indexing_mode":"index","subproblems":[],"grids":[]}