{"kind":"task","effective_mode":"full","benchmark":{"kind":"benchmark","effective_mode":"full","slug":"mmlu-pro","formal_name":"MMLU-Pro","introduction":"MMLU-Pro rebuilds MMLU with ten answer options instead of four and removes items that no longer separate models. Questions span fourteen academic and professional subjects.","introduction_ja":"","introduction_en":"","category":"Category not supplied","task_count":null,"acquisition_status":"Acquisition status not supplied","official_url":"https://huggingface.co/datasets/TIGER-Lab/MMLU-Pro","indexing_mode":"index","profile":{"resources":[],"task_format":"","scoring":"","metric":"","size":"","answer_access":"","license":"","citation":"","maintainer":"","released":"","why_hard":"","related":[],"history":"","construction":"","limitations":"","saturation":"","scores":[]}},"task_id":"968eed7b-f1e9-517f-a2e8-e6096a6dd499","task_key":"default--test--1136","task_revision_id":"1","upstream_id":"1136","short_description":"A pharmacist was employed by a drug store. A long-time customer of the drug…","config":"default","split":"test","body":"{\"category\":\"law\",\"options\":\"['The customer did not suffer any injuries due to the medication.', 'The customer did not ask for clarification about the medication.', 'The physician had prescribed the wrong medication.', 'The drug store was not the manufacturer upon whom ultimate liability falls.', 'The medication was properly labeled with the correct medication name.', 'It was the cashier and not he who personally received the money for the medication.', 'The customer should have been aware that the medication was not correct.', 'He was not a seller of the product upon whom strict liability may be imposed.', 'He exercised reasonable care under the circumstances.', 'The customer misread the label on the medication.']\",\"question\":\"A pharmacist was employed by a drug store. A long-time customer of the drug store presented a prescription to the pharmacist for the medication Clinoril, which was prescribed by her physician to treat her arthritic condition. The pharmacist was in a rush that day because of a backlog of prescriptions to fill, and he misread the customer's prescription. He filled her prescription with Clinitest, a caustic substance not for internal use. He labeled the container with the instructions for Clinoril: \\\"Take one tablet twice a day. \\\"The customer followed the instructions, consumed the Clinitest, and suffered severe injuries to her stomach and esophagus. If the customer brings a strict liability in tort action against the pharmacist, what is his best defense?\",\"src\":\"ori_mmlu-professional_law\"}","display_format":"text","language":"","answer_status":"published","assets":[],"source_url":"https://huggingface.co/datasets/TIGER-Lab/MMLU-Pro","history":"initial import","indexing_mode":"index","subproblems":[],"grids":[]}