{"kind":"task","effective_mode":"full","benchmark":{"kind":"benchmark","effective_mode":"full","slug":"mmlu-pro","formal_name":"MMLU-Pro","introduction":"MMLU-Pro rebuilds MMLU with ten answer options instead of four and removes items that no longer separate models. Questions span fourteen academic and professional subjects.","introduction_ja":"","introduction_en":"","category":"Category not supplied","task_count":null,"acquisition_status":"Acquisition status not supplied","official_url":"https://huggingface.co/datasets/TIGER-Lab/MMLU-Pro","indexing_mode":"index","profile":{"resources":[],"task_format":"","scoring":"","metric":"","size":"","answer_access":"","license":"","citation":"","maintainer":"","released":"","why_hard":"","related":[],"history":"","construction":"","limitations":"","saturation":"","scores":[]}},"task_id":"c68185ab-9057-580b-b861-8370d70da859","task_key":"default--test--866","task_revision_id":"1","upstream_id":"866","short_description":"A woman was standing in the aisle of a subway car and put her purse on the seat…","config":"default","split":"test","body":"{\"category\":\"law\",\"options\":\"[\\\"Fraud, because he took the purse without the woman's consent.\\\", \\\"Larceny, because he took the purse without the woman's permission.\\\", 'Burglary, because he entered the subway car with the intention of committing a theft.', 'Robbery, because he used force in leaving with the purse.', 'Robbery, because he used force to take possession of the purse.', \\\"Robbery, because he used force to remove the woman from the purse's vicinity.\\\", 'Larceny, because force was not used until after he took the purse.', 'Assault, because he pushed the woman out of the way.', 'Larceny, because he made no threat to use force.', \\\"Robbery, because he physically took the purse from the woman's presence.\\\"]\",\"question\":\"A woman was standing in the aisle of a subway car and put her purse on the seat next to her. A man approached the woman from behind and grabbed the purse off the seat. He then pushed the woman out of the way and ran out of the subway car while carrying the purse. The man was apprehended on the subway platform while in possession of the purse. In a jurisdiction that follows the common law with respect to criminal offenses, of what crime can the man properly be convicted?\",\"src\":\"ori_mmlu-professional_law\"}","display_format":"text","language":"","answer_status":"published","assets":[],"source_url":"https://huggingface.co/datasets/TIGER-Lab/MMLU-Pro","history":"initial import","indexing_mode":"index","subproblems":[],"grids":[]}