{"kind":"task","effective_mode":"full","benchmark":{"kind":"benchmark","effective_mode":"full","slug":"mmlu-pro","formal_name":"MMLU-Pro","introduction":"MMLU-Pro rebuilds MMLU with ten answer options instead of four and removes items that no longer separate models. Questions span fourteen academic and professional subjects.","introduction_ja":"","introduction_en":"","category":"Category not supplied","task_count":null,"acquisition_status":"Acquisition status not supplied","official_url":"https://huggingface.co/datasets/TIGER-Lab/MMLU-Pro","indexing_mode":"index","profile":{"resources":[],"task_format":"","scoring":"","metric":"","size":"","answer_access":"","license":"","citation":"","maintainer":"","released":"","why_hard":"","related":[],"history":"","construction":"","limitations":"","saturation":"","scores":[]}},"task_id":"12bdbde4-f75d-592e-a177-ec1c86de4ef8","task_key":"default--test--4796","task_revision_id":"1","upstream_id":"4796","short_description":"This question refers to the following information.","config":"default","split":"test","body":"{\"category\":\"history\",\"options\":\"['Housing was a large part of the British budget.', 'The government Housing Board was indifferent towards the housing program.', \\\"The Borough's Kingfield Street scheme was approved without any issues.\\\", 'The government Housing Board wanted to build more houses.', 'The government announced an increase in the housing programme in May 1921.', 'The government was planning to increase the housing target instead of cutting it.', 'The Council was expecting the deferment of the Kingfield Street scheme.', 'Workers were threatening communist revolution.', 'Welfare programs were questioned after the First World War.', \\\"Addison's Act was widely appreciated and supported without any criticism.\\\"]\",\"question\":\"This question refers to the following information.\\nAddison's Act had been conceived in the belief that, unless working-class aspirations were quickly met after the war, Britain might experience a revolution similar to that in Russia. By 1920 that fear was beginning to recede and Addison's policy was being regarded as extravagant. An immediate victim of the new attitude was the Borough's Kingfield Street scheme, provisional plans for which were approved in September 1920. Much to the Council's surprise, the Government's Housing Board deferred the scheme, \\\"having regard to the Council's present commitments and the money available at the present time, and in May 1921 the Government announced a drastic curtailment of the housing programme, cutting the housing target by half.\\n—Excerpt from a work by English historian Hermione Hobhouse Public Housing in Poplar: The Inter-war Years, 1994\\nWhich of the following is the best description of the plans being changed above?\",\"src\":\"ori_mmlu-high_school_european_history\"}","display_format":"text","language":"","answer_status":"published","assets":[],"source_url":"https://huggingface.co/datasets/TIGER-Lab/MMLU-Pro","history":"initial import","indexing_mode":"index","subproblems":[],"grids":[]}