{"kind":"task","effective_mode":"full","benchmark":{"kind":"benchmark","effective_mode":"full","slug":"mmlu","formal_name":"MMLU","introduction":"MMLU asks four-way multiple-choice questions across 57 subjects, from elementary material to professional-level examinations. Since 2020 it has been the most widely cited single reference point for general capability.","introduction_ja":"","introduction_en":"","category":"Category not supplied","task_count":null,"acquisition_status":"Acquisition status not supplied","official_url":"https://github.com/hendrycks/test","indexing_mode":"index","profile":{"resources":[],"task_format":"","scoring":"","metric":"","size":"","answer_access":"","license":"","citation":"","maintainer":"","released":"","why_hard":"","related":[],"history":"","construction":"","limitations":"","saturation":"","scores":[]}},"task_id":"fbbb7bbf-4436-5891-8694-e434e0388709","task_key":"all--test--fbbb7bbf-4436-5891-8694-e434e0388709","task_revision_id":"1","upstream_id":"","short_description":"Charlie Contractor entered into a contract with Nancy Non-cooperative to remodel…","config":"all","split":"test","body":"{\"choices\":\"['Charlie in an amount of $25,000 since he alleges that Nancy breached the contract.', 'Charlie in an amount of $12,500 since he completed the bathroom construction.', 'Charlie in an amount of $10,000 if the court finds the contract divisible and the $10,000 is an equivalent amount for the completed portion of the agreement.', 'Nancy since Charlie will not complete the contract, he is not entitled to any compensation.']\",\"question\":\"Charlie Contractor entered into a contract with Nancy Non-cooperative to remodel a bathroom and kitchen in Nancy's home. The contract assigned $10,000 to the bathroom and $15,000 to the kitchen with the $25,000 total due when all the remodeling was complete. Charlie completed the bathroom, but refused to begin the kitchen because Nancy did not cooperate. If Charlie sues Nancy, the likely outcome is for\",\"subject\":\"professional_law\"}","display_format":"text","language":"","answer_status":"published","assets":[],"source_url":"https://github.com/hendrycks/test","history":"initial import","indexing_mode":"index","subproblems":[],"grids":[]}