{"kind":"task","effective_mode":"full","benchmark":{"kind":"benchmark","effective_mode":"full","slug":"mmlu-pro","formal_name":"MMLU-Pro","introduction":"MMLU-Pro rebuilds MMLU with ten answer options instead of four and removes items that no longer separate models. Questions span fourteen academic and professional subjects.","introduction_ja":"","introduction_en":"","category":"Category not supplied","task_count":null,"acquisition_status":"Acquisition status not supplied","official_url":"https://huggingface.co/datasets/TIGER-Lab/MMLU-Pro","indexing_mode":"index","profile":{"resources":[],"task_format":"","scoring":"","metric":"","size":"","answer_access":"","license":"","citation":"","maintainer":"","released":"","why_hard":"","related":[],"history":"","construction":"","limitations":"","saturation":"","scores":[]}},"task_id":"fc9eb69e-957b-5ad3-acf2-8a65461a2430","task_key":"default--test--1291","task_revision_id":"1","upstream_id":"1291","short_description":"Bill purchased a can of Sipep from the Ajax Minimart. After he finished drinking…","config":"default","split":"test","body":"{\"category\":\"law\",\"options\":\"['Ajax failed to warn Bill about the potential for insects in the Sipep can.', 'Ajax had prior notice of other similar problems with Sipep products.', 'Ajax actually placed the dead insects into the can.', 'The insects in the can caused Bill physical harm.', 'Ajax is a merchant selling Sipep.', 'Ajax knew or should have known of the defective condition', 'The Sipep can was sealed when Bill purchased it.', 'Bill consumed the entire can of Sipep before noticing the defects.', 'Ajax had a duty to inspect each can of Sipep before selling it.', 'Ajax is the manufacturer of the Sipep.']\",\"question\":\"Bill purchased a can of Sipep from the Ajax Minimart. After he finished drinking the Sipep, Bill noticed that the can contained dead insects stuck on the inside bottom of the can. In a strict product liability tort action against Ajax, Bill must prove, among other things, that:\",\"src\":\"ori_mmlu-jurisprudence\"}","display_format":"text","language":"","answer_status":"published","assets":[],"source_url":"https://huggingface.co/datasets/TIGER-Lab/MMLU-Pro","history":"initial import","indexing_mode":"index","subproblems":[],"grids":[]}