{"kind":"task","effective_mode":"full","benchmark":{"kind":"benchmark","effective_mode":"full","slug":"mmlu","formal_name":"MMLU","introduction":"MMLU asks four-way multiple-choice questions across 57 subjects, from elementary material to professional-level examinations. Since 2020 it has been the most widely cited single reference point for general capability.","introduction_ja":"","introduction_en":"","category":"Category not supplied","task_count":null,"acquisition_status":"Acquisition status not supplied","official_url":"https://github.com/hendrycks/test","indexing_mode":"index","profile":{"resources":[],"task_format":"","scoring":"","metric":"","size":"","answer_access":"","license":"","citation":"","maintainer":"","released":"","why_hard":"","related":[],"history":"","construction":"","limitations":"","saturation":"","scores":[]}},"task_id":"18a64603-d46a-566c-a4c4-c764c1a91535","task_key":"all--test--18a64603-d46a-566c-a4c4-c764c1a91535","task_revision_id":"1","upstream_id":"","short_description":"A man who owned riverfront property sued an upstream factory in federal court…","config":"all","split":"test","body":"{\"choices\":\"['The factory may appeal if the appellate court finds that the case involves a controlling question of law upon which the courts are divided.', 'The factory may appeal if the trial court certifies that there is no just reason for delay.', 'The factory may not appeal until after a trial on the merits or other disposition resulting in a final judgment.', 'The factory may not appeal, because the denial of summary judgment is a collateral order.']\",\"question\":\"A man who owned riverfront property sued an upstream factory in federal court for polluting the river, seeking injunctive relief and $250,000 in damages. The factory moved for summary judgment on the ground of res judicata (claim preclusion), arguing that the man had sued on and lost an identical claim one year before. The court denied the motion. The factory has asked its attorney's advice as to whether it may appeal the court's denial of summary judgment in order to avoid an expensive trial. What advice should the attorney give?\",\"subject\":\"professional_law\"}","display_format":"text","language":"","answer_status":"published","assets":[],"source_url":"https://github.com/hendrycks/test","history":"initial import","indexing_mode":"index","subproblems":[],"grids":[]}