{"kind":"task","effective_mode":"full","benchmark":{"kind":"benchmark","effective_mode":"full","slug":"mmlu-pro","formal_name":"MMLU-Pro","introduction":"MMLU-Pro rebuilds MMLU with ten answer options instead of four and removes items that no longer separate models. Questions span fourteen academic and professional subjects.","introduction_ja":"","introduction_en":"","category":"Category not supplied","task_count":null,"acquisition_status":"Acquisition status not supplied","official_url":"https://huggingface.co/datasets/TIGER-Lab/MMLU-Pro","indexing_mode":"index","profile":{"resources":[],"task_format":"","scoring":"","metric":"","size":"","answer_access":"","license":"","citation":"","maintainer":"","released":"","why_hard":"","related":[],"history":"","construction":"","limitations":"","saturation":"","scores":[]}},"task_id":"456f9395-4bf7-5df6-a5ae-45b267f0a485","task_key":"default--test--1324","task_revision_id":"1","upstream_id":"1324","short_description":"One evening, a defendant was at a party and offered to sell an ounce of…","config":"default","split":"test","body":"{\"category\":\"law\",\"options\":\"['Solicitation, attempted sale of narcotics, and false pretenses.', 'False pretenses.', 'Solicitation and false pretenses.', 'Attempted sale of narcotics and false pretenses.', 'Attempted sale of narcotics.', 'Attempted fraud.', 'No crimes, as no actual narcotics were involved.', 'Theft by deception.', 'Solicitation and attempted sale of narcotics.', 'Solicitation, theft by deception, and false pretenses.']\",\"question\":\"One evening, a defendant was at a party and offered to sell an ounce of marijuana to a partygoer. The partygoer agreed to purchase the marijuana and gave the defendant $200. In return, the defendant handed the partygoer a bag containing what appeared to be marijuana. At the time of the transaction, the defendant knew that the bag did not contain marijuana but, instead, was oregano. The defendant is guilty for which, if any, of the following crimes?\",\"src\":\"ori_mmlu-professional_law\"}","display_format":"text","language":"","answer_status":"published","assets":[],"source_url":"https://huggingface.co/datasets/TIGER-Lab/MMLU-Pro","history":"initial import","indexing_mode":"index","subproblems":[],"grids":[]}