{"kind":"task","effective_mode":"full","benchmark":{"kind":"benchmark","effective_mode":"full","slug":"mmlu-pro","formal_name":"MMLU-Pro","introduction":"MMLU-Pro rebuilds MMLU with ten answer options instead of four and removes items that no longer separate models. Questions span fourteen academic and professional subjects.","introduction_ja":"","introduction_en":"","category":"Category not supplied","task_count":null,"acquisition_status":"Acquisition status not supplied","official_url":"https://huggingface.co/datasets/TIGER-Lab/MMLU-Pro","indexing_mode":"index","profile":{"resources":[],"task_format":"","scoring":"","metric":"","size":"","answer_access":"","license":"","citation":"","maintainer":"","released":"","why_hard":"","related":[],"history":"","construction":"","limitations":"","saturation":"","scores":[]}},"task_id":"d629b30c-4f56-5212-8de6-910dba2c22f1","task_key":"default--test--1153","task_revision_id":"1","upstream_id":"1153","short_description":"Carol Collector was a serious antique car buyer, and was always searching for…","config":"default","split":"test","body":"{\"category\":\"law\",\"options\":\"['Carol, because the internet picture was of a car that had never been driven, not the actual used car she was to buy.', 'Carol, because the seller was not negligent in providing a picture of the car.', 'Carol, because the \\\"as is\\\" disclosure was not totally effective.', 'The seller, unless they can prove Carol had knowledge that the car was a 1968 model.', 'The seller, because the car was misrepresented in the advertisement.', 'The seller, because the buyer was aware the Thunderbird was sold \\\"as is.\\\"', 'Carol, unless the seller can prove the 1968 model was of equal value to the 1964 model.', 'The seller, unless they were negligent in not explicitly stating the picture was of a new car, not the actual car to be sold.', 'The seller, if Carol can prove she relied on the picture when making her purchase.', 'Carol, because the seller provided a picture of the car, regardless of its condition.']\",\"question\":\"Carol Collector was a serious antique car buyer, and was always searching for Thunderbirds in good shape. She saw a newspaper ad offering a 1964 Thunderbird sports coupe for $25,000, The ad also gave a website address \\\"to view a picture of the car,\\\" which Carol visited, and which showed a T-Bird with a perfect body and interior. Carol paid the $25,000 and signed a sales contract which specified that the Thunderbird was \\\"used and sold as is.\\\" When Carol went to pick up the car, she learned it was a 1968 model, which was not at all rare and worth much less than the advertised 1964 model. If Carol brings suit against the seller, the likely outcome is for\",\"src\":\"ori_mmlu-professional_law\"}","display_format":"text","language":"","answer_status":"published","assets":[],"source_url":"https://huggingface.co/datasets/TIGER-Lab/MMLU-Pro","history":"initial import","indexing_mode":"index","subproblems":[],"grids":[]}