{"kind":"task","effective_mode":"full","benchmark":{"kind":"benchmark","effective_mode":"full","slug":"mmlu-pro","formal_name":"MMLU-Pro","introduction":"MMLU-Pro rebuilds MMLU with ten answer options instead of four and removes items that no longer separate models. Questions span fourteen academic and professional subjects.","introduction_ja":"","introduction_en":"","category":"Category not supplied","task_count":null,"acquisition_status":"Acquisition status not supplied","official_url":"https://huggingface.co/datasets/TIGER-Lab/MMLU-Pro","indexing_mode":"index","profile":{"resources":[],"task_format":"","scoring":"","metric":"","size":"","answer_access":"","license":"","citation":"","maintainer":"","released":"","why_hard":"","related":[],"history":"","construction":"","limitations":"","saturation":"","scores":[]}},"task_id":"2ba48299-b0c0-54b4-bcee-98f4603064c3","task_key":"default--test--1543","task_revision_id":"1","upstream_id":"1543","short_description":"One evening, a defendant set fire to a homeowner's occupied house. As a result…","config":"default","split":"test","body":"{\"category\":\"law\",\"options\":\"[\\\"granted, because the defendant's right to a fair trial was violated by the inconsistent verdicts.\\\", \\\"denied, because the jury's decision is final and cannot be overturned.\\\", \\\"denied, because the defendant's proper remedy is to seek an appellate review for a non-constitutional error.\\\", 'denied, because the verdicts do not amount to a reversible error.', \\\"granted, because the guilty verdict is plain error that adversely affects the defendant's constitutional rights.\\\", 'granted, because the verdicts contradict each other, raising reasonable doubt.', 'granted, because the verdicts are legally inconsistent and should lead to an acquittal of both charges.', 'denied, because the jury found sufficient evidence to convict on the first count, regardless of the second count outcome.', 'granted, because the defendant was not directly responsible for the death.']\",\"question\":\"One evening, a defendant set fire to a homeowner's occupied house. As a result of the blaze, the homeowner's daughter was killed. The defendant was charged with felony murder on the first count and arson on the second count of the two-count indictment. The jury found the defendant guilty on the first count, but returned a not guilty verdict on the second. The defendant's attorney's motion to set aside the guilty verdict on the felony murder charge will most likely be\",\"src\":\"ori_mmlu-professional_law\"}","display_format":"text","language":"","answer_status":"published","assets":[],"source_url":"https://huggingface.co/datasets/TIGER-Lab/MMLU-Pro","history":"initial import","indexing_mode":"index","subproblems":[],"grids":[]}