{"kind":"task","effective_mode":"full","benchmark":{"kind":"benchmark","effective_mode":"full","slug":"mmlu","formal_name":"MMLU","introduction":"MMLU asks four-way multiple-choice questions across 57 subjects, from elementary material to professional-level examinations. Since 2020 it has been the most widely cited single reference point for general capability.","introduction_ja":"","introduction_en":"","category":"Category not supplied","task_count":null,"acquisition_status":"Acquisition status not supplied","official_url":"https://github.com/hendrycks/test","indexing_mode":"index","profile":{"resources":[],"task_format":"","scoring":"","metric":"","size":"","answer_access":"","license":"","citation":"","maintainer":"","released":"","why_hard":"","related":[],"history":"","construction":"","limitations":"","saturation":"","scores":[]}},"task_id":"e6609b0a-c771-5587-9791-38a77dd0e65d","task_key":"all--test--e6609b0a-c771-5587-9791-38a77dd0e65d","task_revision_id":"1","upstream_id":"","short_description":"A pet store owner opened a store in a new town. She was not aware that the city…","config":"all","split":"test","body":"{\"choices\":\"['Yes, because the pet store owner was not treated differently than any other pet store owner.', 'Yes, because the correct test requires only that the law in question will satisfy any set of facts that could establish a rational basis for achieving legitimate government ends.', 'No, the pet store owner has a vested constitutional right to earn a living and that is unreasonably precluded by the law in question.', 'No, because this ordinance is so invidiously discriminatory that it must pass the strict scrutiny test, which it fails to do.']\",\"question\":\"A pet store owner opened a store in a new town. She was not aware that the city council had recently passed a law prohibiting the sale of cats and dogs from a retail pet store. The statute had a provision that allowed the pet store to give space to a non-profit humane society to adopt animals from the pet store. The owner sued the city in federal court, claiming a violation of due process and equal protection under the Fourteenth Amendment. The court used the rational basis test to conclude that the statute served a legitimate state interest and was enforceable. The pet store owner appealed the decision. Will the appellate court likely affirm the decision of the lower court?\",\"subject\":\"professional_law\"}","display_format":"text","language":"","answer_status":"published","assets":[],"source_url":"https://github.com/hendrycks/test","history":"initial import","indexing_mode":"index","subproblems":[],"grids":[]}