{"kind":"task","effective_mode":"full","benchmark":{"kind":"benchmark","effective_mode":"full","slug":"mmlu-pro","formal_name":"MMLU-Pro","introduction":"MMLU-Pro rebuilds MMLU with ten answer options instead of four and removes items that no longer separate models. Questions span fourteen academic and professional subjects.","introduction_ja":"","introduction_en":"","category":"Category not supplied","task_count":null,"acquisition_status":"Acquisition status not supplied","official_url":"https://huggingface.co/datasets/TIGER-Lab/MMLU-Pro","indexing_mode":"index","profile":{"resources":[],"task_format":"","scoring":"","metric":"","size":"","answer_access":"","license":"","citation":"","maintainer":"","released":"","why_hard":"","related":[],"history":"","construction":"","limitations":"","saturation":"","scores":[]}},"task_id":"567c243d-ca3e-54e5-8cee-184ee0005012","task_key":"default--test--1507","task_revision_id":"1","upstream_id":"1507","short_description":"Believing she was pregnant, a woman went to see a doctor, a licensed…","config":"default","split":"test","body":"{\"category\":\"law\",\"options\":\"['murder and false pretenses.', 'battery and larceny by trick.', 'murder and larceny by trick.', 'manslaughter and larceny by trick.', 'manslaughter, battery, and false pretenses.', 'murder, manslaughter, and larceny by trick.', 'murder, battery, and false pretenses.', 'manslaughter and false pretenses.', 'battery, manslaughter, and larceny by trick.', 'battery and false pretenses.']\",\"question\":\"Believing she was pregnant, a woman went to see a doctor, a licensed gynecologist, because she wanted an abortion. The doctor examined her and determined that she was not pregnant, but decided that he would tell her she was, in order to earn his $500 fee. After receiving the $500, the doctor proceeded to have the woman admitted to a hospital by falsely informing the hospital authorities that she had a benign tumor on her uterus, which he was going to remove. He performed all the surgical procedures appropriate for an abortion under adequate hygienic conditions, but the woman began hemorrhaging and died. The doctor should be found guilty of\",\"src\":\"ori_mmlu-professional_law\"}","display_format":"text","language":"","answer_status":"published","assets":[],"source_url":"https://huggingface.co/datasets/TIGER-Lab/MMLU-Pro","history":"initial import","indexing_mode":"index","subproblems":[],"grids":[]}