{"kind":"task","effective_mode":"full","benchmark":{"kind":"benchmark","effective_mode":"full","slug":"mmlu-pro","formal_name":"MMLU-Pro","introduction":"MMLU-Pro rebuilds MMLU with ten answer options instead of four and removes items that no longer separate models. Questions span fourteen academic and professional subjects.","introduction_ja":"","introduction_en":"","category":"Category not supplied","task_count":null,"acquisition_status":"Acquisition status not supplied","official_url":"https://huggingface.co/datasets/TIGER-Lab/MMLU-Pro","indexing_mode":"index","profile":{"resources":[],"task_format":"","scoring":"","metric":"","size":"","answer_access":"","license":"","citation":"","maintainer":"","released":"","why_hard":"","related":[],"history":"","construction":"","limitations":"","saturation":"","scores":[]}},"task_id":"b36922ca-fb0a-5b8c-a823-1b97e5eae0f6","task_key":"default--test--1519","task_revision_id":"1","upstream_id":"1519","short_description":"A woman was hit in the right eye by a shard of glass when her car was rear-ended…","config":"default","split":"test","body":"{\"category\":\"law\",\"options\":\"['Yes, because the damages awarded are excessive and not substantiated by concrete proof of earning capacity.', 'Yes, because the plaintiff failed to prove that she would have definitely become a successful professional ballet dancer.', 'Yes, because the defendant has the right to challenge the amount of the award given by the jury.', 'Yes, the evidence did not guarantee or assure anyone that she really would have been a successful professional ballet dancer.', \\\"No, the jury's award is discretionary and inviolate, and it may not be disturbed with respect to personal injury damages.\\\", 'Yes, she can only collect for lost work that she held prior to the accident; her future lost earning capacity could be based on an apprentice ballet dancer.', 'No, because some imprecision in the evidence regarding loss of earning capacity is to be expected and is not grounds for excluding the evidence.']\",\"question\":\"A woman was hit in the right eye by a shard of glass when her car was rear-ended by an industrial truck. She was hospitalized, lost her eye, and had several operations over several years. She could not return to work as an apprentice ballet dancer due to loss of coordination, constant headaches, and severe disfigurement. She is considered totally disabled. She had been training for the ballet for many years and, at the trial against the tortfeasor, she proved that she was considered a bright and uncoming student. The defendant claimed that this was still too speculative whether she would have been successful. The trial court disagreed and allowed evidence on the lost earning capacity of a ballet dancer. The jury awarded $3 million. Will the appellate court likely reverse the verdict due to the lost earning capacity issue?\",\"src\":\"ori_mmlu-professional_law\"}","display_format":"text","language":"","answer_status":"published","assets":[],"source_url":"https://huggingface.co/datasets/TIGER-Lab/MMLU-Pro","history":"initial import","indexing_mode":"index","subproblems":[],"grids":[]}