{"kind":"task","effective_mode":"full","benchmark":{"kind":"benchmark","effective_mode":"full","slug":"mmlu-pro","formal_name":"MMLU-Pro","introduction":"MMLU-Pro rebuilds MMLU with ten answer options instead of four and removes items that no longer separate models. Questions span fourteen academic and professional subjects.","introduction_ja":"","introduction_en":"","category":"Category not supplied","task_count":null,"acquisition_status":"Acquisition status not supplied","official_url":"https://huggingface.co/datasets/TIGER-Lab/MMLU-Pro","indexing_mode":"index","profile":{"resources":[],"task_format":"","scoring":"","metric":"","size":"","answer_access":"","license":"","citation":"","maintainer":"","released":"","why_hard":"","related":[],"history":"","construction":"","limitations":"","saturation":"","scores":[]}},"task_id":"a6e5f40e-0286-56c1-a605-3c8a10006925","task_key":"default--test--5632","task_revision_id":"1","upstream_id":"5632","short_description":"On January 1, a company enters into an operating lease for office space and…","config":"default","split":"test","body":"{\"category\":\"other\",\"options\":\"['April.', 'June.', 'January.', 'May.', 'July.', 'February.', 'September.', 'December.', 'August.', 'March.']\",\"question\":\"On January 1, a company enters into an operating lease for office space and receives control of the property to make leasehold improvements. The company begins alterations to the property on March 1 and the company’s staff moves into the property on May 1. The monthly rental payments begin on July 1. The recognition of rental expense for the new offices should begin in which of the following months?\",\"src\":\"ori_mmlu-professional_accounting\"}","display_format":"text","language":"","answer_status":"published","assets":[],"source_url":"https://huggingface.co/datasets/TIGER-Lab/MMLU-Pro","history":"initial import","indexing_mode":"index","subproblems":[],"grids":[]}