{"kind":"task","effective_mode":"full","benchmark":{"kind":"benchmark","effective_mode":"full","slug":"mmlu-pro","formal_name":"MMLU-Pro","introduction":"MMLU-Pro rebuilds MMLU with ten answer options instead of four and removes items that no longer separate models. Questions span fourteen academic and professional subjects.","introduction_ja":"","introduction_en":"","category":"Category not supplied","task_count":null,"acquisition_status":"Acquisition status not supplied","official_url":"https://huggingface.co/datasets/TIGER-Lab/MMLU-Pro","indexing_mode":"index","profile":{"resources":[],"task_format":"","scoring":"","metric":"","size":"","answer_access":"","license":"","citation":"","maintainer":"","released":"","why_hard":"","related":[],"history":"","construction":"","limitations":"","saturation":"","scores":[]}},"task_id":"527ed08a-cb72-5392-bc08-28928fee58da","task_key":"default--test--881","task_revision_id":"1","upstream_id":"881","short_description":"A teenage male was on trial on charges of illegally entering an owner's property…","config":"default","split":"test","body":"{\"category\":\"law\",\"options\":\"['No, as it is essential for the defense to create reasonable doubt in the minds of the jury.', 'Yes, because the question is not directly related to the charges against the defendant.', 'Yes, because this is an objectionable question based on pure speculation.', 'No, because the defense is merely suggesting a possibility, not stating it as fact.', \\\"No, because the defense has the right to question every aspect of the prosecution's case.\\\", 'Yes, because it is not permissible to question the credibility of the police without substantial supporting evidence.', 'No, questioning about the possibility of other perpetrators, or about police misconduct, is always permissible.', 'No, because the witness may know something that he is not revealing and for that reason, a cross-examiner is given wide latitude in questioning.']\",\"question\":\"A teenage male was on trial on charges of illegally entering an owner's property and stealing a car. The witness can only confirm for the prosecution that a car was stolen, the model and type of car, and that it was missing from the property on a certain date. The defense asks witness \\\"isn't it possible that my client did not steal the car, that someone else did it, and that the police are lying about the charges against him? The prosecution objected to the question. Will the court likely sustain the objection?\",\"src\":\"ori_mmlu-professional_law\"}","display_format":"text","language":"","answer_status":"published","assets":[],"source_url":"https://huggingface.co/datasets/TIGER-Lab/MMLU-Pro","history":"initial import","indexing_mode":"index","subproblems":[],"grids":[]}