{"kind":"task","effective_mode":"full","benchmark":{"kind":"benchmark","effective_mode":"full","slug":"mmlu","formal_name":"MMLU","introduction":"MMLU asks four-way multiple-choice questions across 57 subjects, from elementary material to professional-level examinations. Since 2020 it has been the most widely cited single reference point for general capability.","introduction_ja":"","introduction_en":"","category":"Category not supplied","task_count":null,"acquisition_status":"Acquisition status not supplied","official_url":"https://github.com/hendrycks/test","indexing_mode":"index","profile":{"resources":[],"task_format":"","scoring":"","metric":"","size":"","answer_access":"","license":"","citation":"","maintainer":"","released":"","why_hard":"","related":[],"history":"","construction":"","limitations":"","saturation":"","scores":[]}},"task_id":"e96f4cc3-cadc-52ec-bec7-e45352f0349a","task_key":"all--test--e96f4cc3-cadc-52ec-bec7-e45352f0349a","task_revision_id":"1","upstream_id":"","short_description":"During an ice storm, a man's car slipped down an embankment and became lodged…","config":"all","split":"test","body":"{\"choices\":\"['No, because both parties were uncertain about the distance.', \\\"No, because the towing company assumed the risk by the manager's failure to examine the distance himself.\\\", 'Yes, because at the time of contracting, both parties were mistaken about a basic assumption on which the contract was based.', 'Yes, because the agreement did not allocate the risk of mistake to either party.']\",\"question\":\"During an ice storm, a man's car slipped down an embankment and became lodged against a large tree. The man called a towing company and told the company's manager that the car was 100 feet down the embankment. \\\"That's lucky,\\\" said the manager, \\\"because our winch only goes 100 feet.\\\" After the manager and the man agreed on a price, an employee of the company attempted to reach the car but could not because the car turned out to be 120 feet down the embankment. Is the towing company's performance excused on the grounds of mistake?\",\"subject\":\"professional_law\"}","display_format":"text","language":"","answer_status":"published","assets":[],"source_url":"https://github.com/hendrycks/test","history":"initial import","indexing_mode":"index","subproblems":[],"grids":[]}