{"kind":"task","effective_mode":"full","benchmark":{"kind":"benchmark","effective_mode":"full","slug":"mmlu-pro","formal_name":"MMLU-Pro","introduction":"MMLU-Pro rebuilds MMLU with ten answer options instead of four and removes items that no longer separate models. Questions span fourteen academic and professional subjects.","introduction_ja":"","introduction_en":"","category":"Category not supplied","task_count":null,"acquisition_status":"Acquisition status not supplied","official_url":"https://huggingface.co/datasets/TIGER-Lab/MMLU-Pro","indexing_mode":"index","profile":{"resources":[],"task_format":"","scoring":"","metric":"","size":"","answer_access":"","license":"","citation":"","maintainer":"","released":"","why_hard":"","related":[],"history":"","construction":"","limitations":"","saturation":"","scores":[]}},"task_id":"b21ddb1a-df11-52d2-b12a-50c840662a68","task_key":"default--test--1708","task_revision_id":"1","upstream_id":"1708","short_description":"A student lent his classmate his car to attend a concert at a theater. Before…","config":"default","split":"test","body":"{\"category\":\"law\",\"options\":\"['the cost of repairing the car because the classmate used the car for an illegal purpose.', 'nothing, because the damage was caused by a third party not in control of the car.', 'the cost of repairing the car because the classmate was driving under the influence of marijuana.', 'nothing, because the car was damaged due to an incident that the classmate could not control.', 'the value of the car before it was damaged because the classmate used the car for a purpose other than that for which it was lent.', \\\"the value of the car before it was damaged because the car was damaged while under the classmate's dominion and control.\\\", \\\"nothing, because the classmate was not negligent in causing the car's damage.\\\", 'the cost of repairing the car because the classmate used the car for a purpose other than attending the concert.', \\\"the cost of repairing the car because the car was damaged while under the classmate's dominion and control.\\\", 'the value of the car before it was damaged because the classmate was using illegal substances while driving the car.']\",\"question\":\"A student lent his classmate his car to attend a concert at a theater. Before going to the theater, the classmate drove to a dealer's home to buy some marijuana. The dealer lived approximately two miles from the theater. After picking up the marijuana, the classmate then drove to the concert. As he was driving to the concert, the classmate smoked two marijuana cigarettes so that he could be \\\"high\\\" for the show. While the car was parked outside the theater, through no fault of the classmate, it was struck by another car and damaged. Repairs will cost $750. The driver of the vehicle that struck the student's car fled the scene and cannot be identified. If the student asserts a claim against the classmate for the damage to the student's car, the plaintiff will recover\",\"src\":\"ori_mmlu-professional_law\"}","display_format":"text","language":"","answer_status":"published","assets":[],"source_url":"https://huggingface.co/datasets/TIGER-Lab/MMLU-Pro","history":"initial import","indexing_mode":"index","subproblems":[],"grids":[]}