{"kind":"task","effective_mode":"full","benchmark":{"kind":"benchmark","effective_mode":"full","slug":"mmlu","formal_name":"MMLU","introduction":"MMLU asks four-way multiple-choice questions across 57 subjects, from elementary material to professional-level examinations. Since 2020 it has been the most widely cited single reference point for general capability.","introduction_ja":"","introduction_en":"","category":"Category not supplied","task_count":null,"acquisition_status":"Acquisition status not supplied","official_url":"https://github.com/hendrycks/test","indexing_mode":"index","profile":{"resources":[],"task_format":"","scoring":"","metric":"","size":"","answer_access":"","license":"","citation":"","maintainer":"","released":"","why_hard":"","related":[],"history":"","construction":"","limitations":"","saturation":"","scores":[]}},"task_id":"e7f79179-a8d5-5cb4-bab0-0b23855b7532","task_key":"all--test--e7f79179-a8d5-5cb4-bab0-0b23855b7532","task_revision_id":"1","upstream_id":"","short_description":"Joey has an orchard on the south part of town and recently found out that a…","config":"all","split":"test","body":"{\"choices\":\"['If the action is a physical or actual appropriation', 'If the action denies all economic use of the land', 'If the action just decreases the value of property', 'If the action affects the property taxes']\",\"question\":\"Joey has an orchard on the south part of town and recently found out that a sizable number of the trees are diseased. He was further dismayed when he got an order from the city to destroy the affected trees. He wonders if he will be compensated for this, and the answers depends on whether this action constitutes a regulation or a taking. Which of the following is not a consideration in deciding this question?\",\"subject\":\"professional_law\"}","display_format":"text","language":"","answer_status":"published","assets":[],"source_url":"https://github.com/hendrycks/test","history":"initial import","indexing_mode":"index","subproblems":[],"grids":[]}