{"kind":"task","effective_mode":"full","benchmark":{"kind":"benchmark","effective_mode":"full","slug":"mmlu-pro","formal_name":"MMLU-Pro","introduction":"MMLU-Pro rebuilds MMLU with ten answer options instead of four and removes items that no longer separate models. Questions span fourteen academic and professional subjects.","introduction_ja":"","introduction_en":"","category":"Category not supplied","task_count":null,"acquisition_status":"Acquisition status not supplied","official_url":"https://huggingface.co/datasets/TIGER-Lab/MMLU-Pro","indexing_mode":"index","profile":{"resources":[],"task_format":"","scoring":"","metric":"","size":"","answer_access":"","license":"","citation":"","maintainer":"","released":"","why_hard":"","related":[],"history":"","construction":"","limitations":"","saturation":"","scores":[]}},"task_id":"40ccfd8f-4eb0-523f-940b-cc94844ff4a2","task_key":"default--test--1468","task_revision_id":"1","upstream_id":"1468","short_description":"A wife is the beneficiary of a policy issued by an insurance company, insuring…","config":"default","split":"test","body":"{\"category\":\"law\",\"options\":\"['competent, because she can provide a first-hand account of the incident.', 'incompetent, because she was not an eyewitness to the actual event.', 'incompetent, because her testimony is based on her perception of events.', 'competent, because she was present on the scene after the event occurred.', 'competent, because she had personal knowledge of the matter.', 'competent, because the neighbor is available to testify.', 'incompetent, because her testimony could potentially be biased.', \\\"incompetent, because she was testifying to facts occurring after her husband's death.\\\", \\\"competent, because she can corroborate her account with the neighbor's testimony.\\\", 'incompetent, because she had a personal interest in the outcome of the lawsuit.']\",\"question\":\"A wife is the beneficiary of a policy issued by an insurance company, insuring the life of her husband, now deceased. The policy contained a clause providing that double indemnity is payable in the event that death of the insured \\\"results directly, and independently of all other causes, from bodily injury effected solely through external violent and unexpected means. \\\"The husband was found dead in the chicken shed of his farm. His death resulted from wounds caused by a shotgun blast. The wife filed the necessary papers with the insurance company concerning proof of her husband's death. The insurance company admitted liability for the face amount of the policy but rejected the wife's claim for double indemnity. The wife then instituted suit against the insurance company demanding judgment according to the double indemnity provisions of the husband's insurance policy. At trial, the wife was called to testify about the events on the day of her husband's death. The wife said that she was in the kitchen when she heard a gunshot in the shed. As she rushed out of the house, she saw their neighbor running from the shed. The neighbor is present in court. As a witness, the wife was\",\"src\":\"ori_mmlu-professional_law\"}","display_format":"text","language":"","answer_status":"published","assets":[],"source_url":"https://huggingface.co/datasets/TIGER-Lab/MMLU-Pro","history":"initial import","indexing_mode":"index","subproblems":[],"grids":[]}