{"kind":"task","effective_mode":"full","benchmark":{"kind":"benchmark","effective_mode":"full","slug":"mmlu-pro","formal_name":"MMLU-Pro","introduction":"MMLU-Pro rebuilds MMLU with ten answer options instead of four and removes items that no longer separate models. Questions span fourteen academic and professional subjects.","introduction_ja":"","introduction_en":"","category":"Category not supplied","task_count":null,"acquisition_status":"Acquisition status not supplied","official_url":"https://huggingface.co/datasets/TIGER-Lab/MMLU-Pro","indexing_mode":"index","profile":{"resources":[],"task_format":"","scoring":"","metric":"","size":"","answer_access":"","license":"","citation":"","maintainer":"","released":"","why_hard":"","related":[],"history":"","construction":"","limitations":"","saturation":"","scores":[]}},"task_id":"a76002d8-938b-5a3c-a64e-d0e454f8050c","task_key":"default--test--1806","task_revision_id":"1","upstream_id":"1806","short_description":"A defendant is arrested and charged with shoplifting. He claims that he intended…","config":"default","split":"test","body":"{\"category\":\"law\",\"options\":\"['Yes, because the defendant was denied the right to counsel.', 'Yes, because the defendant was not allowed to pay for the item he took.', 'Yes, because the defendant was denied the right to a fair trial.', 'No, because the defendant admitted to taking the item.', 'Yes, because the maximum penalty for shoplifting was not applied.', 'Yes, because the defendant was denied the right to a jury trial.', 'No, because his actual sentence was only three months.', 'No, because the defendant was allowed to serve his sentence on weekends.', 'No, because the defendant was found guilty of the charges.']\",\"question\":\"A defendant is arrested and charged with shoplifting. He claims that he intended to pay for the item he took and requests an attorney and a jury trial. Both requests are denied by the judge, and the defendant is found guilty. The maximum penalty for shoplifting is a $500 fine and six months in jail. The defendant is sentenced to three months in jail, which he is allowed to serve on weekends in the county detention center. The defendant claims that his constitutional rights have been violated. Is he correct?\",\"src\":\"ori_mmlu-professional_law\"}","display_format":"text","language":"","answer_status":"published","assets":[],"source_url":"https://huggingface.co/datasets/TIGER-Lab/MMLU-Pro","history":"initial import","indexing_mode":"index","subproblems":[],"grids":[]}