{"kind":"task","effective_mode":"full","benchmark":{"kind":"benchmark","effective_mode":"full","slug":"mmlu","formal_name":"MMLU","introduction":"MMLU asks four-way multiple-choice questions across 57 subjects, from elementary material to professional-level examinations. Since 2020 it has been the most widely cited single reference point for general capability.","introduction_ja":"","introduction_en":"","category":"Category not supplied","task_count":null,"acquisition_status":"Acquisition status not supplied","official_url":"https://github.com/hendrycks/test","indexing_mode":"index","profile":{"resources":[],"task_format":"","scoring":"","metric":"","size":"","answer_access":"","license":"","citation":"","maintainer":"","released":"","why_hard":"","related":[],"history":"","construction":"","limitations":"","saturation":"","scores":[]}},"task_id":"20e71ba0-b790-549f-8809-6db845f029aa","task_key":"all--test--20e71ba0-b790-549f-8809-6db845f029aa","task_revision_id":"1","upstream_id":"","short_description":"A customer sued a printing company for negligence, claiming that the holiday…","config":"all","split":"test","body":"{\"choices\":\"['grant the request, because it promotes expeditious and economical disposition of the proceedings.', 'grant the request, because the standard of care is amply supported by information supplied to the court.', 'deny the request, because judicial notice cannot be taken of matters that are ultimate issues in the case.', 'deny the request if the judge determines that the statement is subject to reasonable dispute.']\",\"question\":\"A customer sued a printing company for negligence, claiming that the holiday cards he specially ordered were of poor quality and had his name misspelled. At trial, the customer shows the court a printed statement from a printer's association defining the minimum standard of skill for the printing industry in the community. The customer asks the court to take judicial notice that the printed statement constitutes the applicable standard of care for printing companies in the area. The court should\",\"subject\":\"professional_law\"}","display_format":"text","language":"","answer_status":"published","assets":[],"source_url":"https://github.com/hendrycks/test","history":"initial import","indexing_mode":"index","subproblems":[],"grids":[]}