{"kind":"task","effective_mode":"full","benchmark":{"kind":"benchmark","effective_mode":"full","slug":"mmlu","formal_name":"MMLU","introduction":"MMLU asks four-way multiple-choice questions across 57 subjects, from elementary material to professional-level examinations. Since 2020 it has been the most widely cited single reference point for general capability.","introduction_ja":"","introduction_en":"","category":"Category not supplied","task_count":null,"acquisition_status":"Acquisition status not supplied","official_url":"https://github.com/hendrycks/test","indexing_mode":"index","profile":{"resources":[],"task_format":"","scoring":"","metric":"","size":"","answer_access":"","license":"","citation":"","maintainer":"","released":"","why_hard":"","related":[],"history":"","construction":"","limitations":"","saturation":"","scores":[]}},"task_id":"9959cb1f-5efb-5b82-a116-b55a50114d6f","task_key":"all--test--9959cb1f-5efb-5b82-a116-b55a50114d6f","task_revision_id":"1","upstream_id":"","short_description":"A teenage male was on trial on charges of illegally entering an owner's property…","config":"all","split":"test","body":"{\"choices\":\"['Yes, because this is an objectionable question based on pure speculation.', 'Yes, because it is not permissible to question the credibility of the police without substantial supporting evidence.', 'No, questioning about the possibility of other perpetrators, or about police misconduct, is always permissible.', 'No, because the witness may know something that he is not revealing and for that reason, a cross-examiner is given wide latitude in questioning.']\",\"question\":\"A teenage male was on trial on charges of illegally entering an owner's property and stealing a car. The witness can only confirm for the prosecution that a car was stolen, the model and type of car, and that it was missing from the property on a certain date. The defense asks witness \\\"isn't it possible that my client did not steal the car, that someone else did it, and that the police are lying about the charges against him? The prosecution objected to the question. Will the court likely sustain the objection?\",\"subject\":\"professional_law\"}","display_format":"text","language":"","answer_status":"published","assets":[],"source_url":"https://github.com/hendrycks/test","history":"initial import","indexing_mode":"index","subproblems":[],"grids":[]}