{"kind":"task","effective_mode":"full","benchmark":{"kind":"benchmark","effective_mode":"full","slug":"mmlu","formal_name":"MMLU","introduction":"MMLU asks four-way multiple-choice questions across 57 subjects, from elementary material to professional-level examinations. Since 2020 it has been the most widely cited single reference point for general capability.","introduction_ja":"","introduction_en":"","category":"Category not supplied","task_count":null,"acquisition_status":"Acquisition status not supplied","official_url":"https://github.com/hendrycks/test","indexing_mode":"index","profile":{"resources":[],"task_format":"","scoring":"","metric":"","size":"","answer_access":"","license":"","citation":"","maintainer":"","released":"","why_hard":"","related":[],"history":"","construction":"","limitations":"","saturation":"","scores":[]}},"task_id":"bc6472ce-6947-5fed-ad43-fc91946401a7","task_key":"all--test--bc6472ce-6947-5fed-ad43-fc91946401a7","task_revision_id":"1","upstream_id":"","short_description":"A woman inherited her father's farm when he died testate. Prior to her father's…","config":"all","split":"test","body":"{\"choices\":\"['profit-a-prendre.', 'license.', 'easement in gross.', 'voluntary waste.']\",\"question\":\"A woman inherited her father's farm when he died testate. Prior to her father's death, the woman had already taken over the operations on the father's farm due to his declining health. Recently, the woman had granted a coal company rights to strip-mine coal from underneath the farm. Their agreement stipulated that the coal company would pay the woman a per-ton royalty for the coal extracted. In addition, the coal company agreed to fill in the excavated area and replace top soil on the surface of the land. After the coal company commenced its strip-mining operations, the woman noticed that the company was not filling in the excavated area as previously agreed. However, because the company paid the woman all the coal royalties from the strip mining, she did not voice any objection regarding its failure to replace the top soil. Two years later, the coal company had now completed its strip-mining operation under its arrangement with the woman. The coal company's right to strip-mine coal from the woman's property would be an example of a (an)\",\"subject\":\"professional_law\"}","display_format":"text","language":"","answer_status":"published","assets":[],"source_url":"https://github.com/hendrycks/test","history":"initial import","indexing_mode":"index","subproblems":[],"grids":[]}