{"kind":"task","effective_mode":"full","benchmark":{"kind":"benchmark","effective_mode":"full","slug":"mmlu","formal_name":"MMLU","introduction":"MMLU asks four-way multiple-choice questions across 57 subjects, from elementary material to professional-level examinations. Since 2020 it has been the most widely cited single reference point for general capability.","introduction_ja":"","introduction_en":"","category":"Category not supplied","task_count":null,"acquisition_status":"Acquisition status not supplied","official_url":"https://github.com/hendrycks/test","indexing_mode":"index","profile":{"resources":[],"task_format":"","scoring":"","metric":"","size":"","answer_access":"","license":"","citation":"","maintainer":"","released":"","why_hard":"","related":[],"history":"","construction":"","limitations":"","saturation":"","scores":[]}},"task_id":"2f4b41ec-f7d4-52ba-b034-e6900ab2ef96","task_key":"all--test--2f4b41ec-f7d4-52ba-b034-e6900ab2ef96","task_revision_id":"1","upstream_id":"","short_description":"An obese woman signed up for a weight loss program that promised weight loss…","config":"all","split":"test","body":"{\"choices\":\"['No, because statements that are mere \\\"puffing\\\" or opinions cannot be the basis for misrepresentation.', 'Yes, because the program had a fiduciary duty to make full disclosure to her of all of the facts.', 'No, because she did not justifiably rely on the incredible representations that were made.', 'Yes, because the program knowingly made false factual representations that the customer relied on.']\",\"question\":\"An obese woman signed up for a weight loss program that promised weight loss down to her medicallyrecommended range. The promotional material claimed patented methods and nutritional supplements that were a \\\"miraculous discovery\\\" that would work for anyone within 24 months. The cost was $1,000 per month; the woman paid a total of $24,000 for two years but she lost only 6 pounds and remained excessively obese. A former program employee told her that there was no patent, no miraculous discovery, and the supplements were the same as a common vitamin brand. She sued the program for misrepresentation. Will she likely prevail?\",\"subject\":\"professional_law\"}","display_format":"text","language":"","answer_status":"published","assets":[],"source_url":"https://github.com/hendrycks/test","history":"initial import","indexing_mode":"index","subproblems":[],"grids":[]}