{"kind":"task","effective_mode":"full","benchmark":{"kind":"benchmark","effective_mode":"full","slug":"mmlu-pro","formal_name":"MMLU-Pro","introduction":"MMLU-Pro rebuilds MMLU with ten answer options instead of four and removes items that no longer separate models. Questions span fourteen academic and professional subjects.","introduction_ja":"","introduction_en":"","category":"Category not supplied","task_count":null,"acquisition_status":"Acquisition status not supplied","official_url":"https://huggingface.co/datasets/TIGER-Lab/MMLU-Pro","indexing_mode":"index","profile":{"resources":[],"task_format":"","scoring":"","metric":"","size":"","answer_access":"","license":"","citation":"","maintainer":"","released":"","why_hard":"","related":[],"history":"","construction":"","limitations":"","saturation":"","scores":[]}},"task_id":"548a9918-eb19-5c8f-9a64-2f6c4b271d51","task_key":"default--test--1284","task_revision_id":"1","upstream_id":"1284","short_description":"A shopper purchased a can of sardines, which had been produced by a fish…","config":"default","split":"test","body":"{\"category\":\"law\",\"options\":\"[\\\"overrule the objection, because the label is direct evidence of the company's responsibility.\\\", 'overrule the objection, because the label is the best evidence showing that the can was produced by the company.', 'sustain the objection, because the company could not have foreseen the mouse in the can.', \\\"sustain the objection, because the label does not prove the company's negligence.\\\", 'overrule the objection, as the label is the primary link between the product and the company.', 'overrule the objection, because the label is self-authenticating.', 'overrule the objection, because the label is a significant piece of evidence in strict liability cases.', \\\"sustain the objection, because the shopper's proper cause of action is for infliction of emotional distress, not strict liability.\\\", 'sustain the objection, because the label is not a valid form of evidence.', 'sustain the objection, because there is no corroborating evidence that the company placed the label on the can.']\",\"question\":\"A shopper purchased a can of sardines, which had been produced by a fish company. A short while later at home, the shopper was preparing a sardine sandwich when she opened the can and, to her astonishment, she saw a decomposed mouse. The shopper became nauseated and vomited upon seeing the mouse. After consulting her attorney, the shopper initiated a strict liability action against the company. At trial, the shopper presented into evidence the sardine can with the company's label, along with other corroborating evidence. The attorney representing the company objected to the introduction of the label on the can. The trial court should\",\"src\":\"ori_mmlu-professional_law\"}","display_format":"text","language":"","answer_status":"published","assets":[],"source_url":"https://huggingface.co/datasets/TIGER-Lab/MMLU-Pro","history":"initial import","indexing_mode":"index","subproblems":[],"grids":[]}