{"kind":"task","effective_mode":"full","benchmark":{"kind":"benchmark","effective_mode":"full","slug":"mmlu-pro","formal_name":"MMLU-Pro","introduction":"MMLU-Pro rebuilds MMLU with ten answer options instead of four and removes items that no longer separate models. Questions span fourteen academic and professional subjects.","introduction_ja":"","introduction_en":"","category":"Category not supplied","task_count":null,"acquisition_status":"Acquisition status not supplied","official_url":"https://huggingface.co/datasets/TIGER-Lab/MMLU-Pro","indexing_mode":"index","profile":{"resources":[],"task_format":"","scoring":"","metric":"","size":"","answer_access":"","license":"","citation":"","maintainer":"","released":"","why_hard":"","related":[],"history":"","construction":"","limitations":"","saturation":"","scores":[]}},"task_id":"4d2fa7e0-e835-5111-a129-a39b3047d797","task_key":"default--test--1025","task_revision_id":"1","upstream_id":"1025","short_description":"A defendant and his college roommate went home together for the holiday…","config":"default","split":"test","body":"{\"category\":\"law\",\"options\":\"[\\\"the package didn't belong to him.\\\", 'the cocaine was illegally seized.', \\\"the trooper didn't have a search warrant.\\\", \\\"the defendant didn't know what was in the package.\\\", \\\"the roommate didn't tell him he was carrying illegal narcotics.\\\", 'the roommate should be the one charged, not him.', 'the substance was not tested to confirm it was cocaine.', \\\"the defendant didn't willingly possess the substance.\\\", 'the cocaine was not in a recognizable form.', 'the defendant was not speeding.']\",\"question\":\"A defendant and his college roommate went home together for the holiday vacation. The defendant was to go back to college by himself. The day that the defendant was to drive back to college, his roommate asked him if he would deliver a package to one of their classmates. The defendant agreed, and the roommate gave him a small package, which the defendant placed in his glove compartment. While driving back to school, the defendant was stopped for speeding. When the state trooper asked the defendant for his license and registration, he reached into the glove compartment. As he did, the package that his roommate had given him fell onto the floor. When it hit the floor, the box broke open and a plastic envelope containing a white substance slid out. Seeing that the substance appeared to be cocaine, the state trooper arrested the defendant for possession of illegal narcotics under a state statute making it a felony to willfully possess a controlled substance. The package did, in fact, contain cocaine. If the defendant is later brought to trial for this charge, his best defense is that\",\"src\":\"ori_mmlu-professional_law\"}","display_format":"text","language":"","answer_status":"published","assets":[],"source_url":"https://huggingface.co/datasets/TIGER-Lab/MMLU-Pro","history":"initial import","indexing_mode":"index","subproblems":[],"grids":[]}