{"kind":"task","effective_mode":"full","benchmark":{"kind":"benchmark","effective_mode":"full","slug":"mmlu-pro","formal_name":"MMLU-Pro","introduction":"MMLU-Pro rebuilds MMLU with ten answer options instead of four and removes items that no longer separate models. Questions span fourteen academic and professional subjects.","introduction_ja":"","introduction_en":"","category":"Category not supplied","task_count":null,"acquisition_status":"Acquisition status not supplied","official_url":"https://huggingface.co/datasets/TIGER-Lab/MMLU-Pro","indexing_mode":"index","profile":{"resources":[],"task_format":"","scoring":"","metric":"","size":"","answer_access":"","license":"","citation":"","maintainer":"","released":"","why_hard":"","related":[],"history":"","construction":"","limitations":"","saturation":"","scores":[]}},"task_id":"2989f108-bbbd-5b63-a51f-2622c5f851dd","task_key":"default--test--1236","task_revision_id":"1","upstream_id":"1236","short_description":"A husband and a wife are filing for divorce. At issue is the market value of the…","config":"default","split":"test","body":"{\"category\":\"law\",\"options\":\"[\\\"admissible, because the neighbor is a witness to the house's value.\\\", \\\"admissible, because it supports the husband's claim.\\\", 'inadmissible, because the neighbor has not been shown to be an expert on real estate market values.', 'admissible, because it is based on a reliable source.', 'inadmissible, because the neighbor is not a party involved in the divorce.', 'inadmissible, because the appraisal report was not officially presented in court.', 'admissible, because she had firsthand knowledge of the estimate contained in the appraisal report.', 'admissible as proper lay opinion.', 'inadmissible, because it involves a privileged communication between husband and wife.']\",\"question\":\"A husband and a wife are filing for divorce. At issue is the market value of the house they own. During the divorce proceedings, the husband's lawyer calls a neighbor to testify that the market value of the home exceeds $100,000. She states that her testimony is based on an estimate contained in an appraisal report that she saw. The appraisal had been prepared at the husband's request by a well-known real estate company in the area. Upon objection by the wife's attorney, the neighbor's testimony will be held\",\"src\":\"ori_mmlu-professional_law\"}","display_format":"text","language":"","answer_status":"published","assets":[],"source_url":"https://huggingface.co/datasets/TIGER-Lab/MMLU-Pro","history":"initial import","indexing_mode":"index","subproblems":[],"grids":[]}