{"kind":"task","effective_mode":"full","benchmark":{"kind":"benchmark","effective_mode":"full","slug":"mmlu-pro","formal_name":"MMLU-Pro","introduction":"MMLU-Pro rebuilds MMLU with ten answer options instead of four and removes items that no longer separate models. Questions span fourteen academic and professional subjects.","introduction_ja":"","introduction_en":"","category":"Category not supplied","task_count":null,"acquisition_status":"Acquisition status not supplied","official_url":"https://huggingface.co/datasets/TIGER-Lab/MMLU-Pro","indexing_mode":"index","profile":{"resources":[],"task_format":"","scoring":"","metric":"","size":"","answer_access":"","license":"","citation":"","maintainer":"","released":"","why_hard":"","related":[],"history":"","construction":"","limitations":"","saturation":"","scores":[]}},"task_id":"7687934f-5976-58ba-9cdb-56d78ccc1d76","task_key":"default--test--4879","task_revision_id":"1","upstream_id":"4879","short_description":"This question refers to the following information.","config":"default","split":"test","body":"{\"category\":\"history\",\"options\":\"['Greater rights for unions', 'Increasing minimum wage', 'Mandatory health insurance for workers', \\\"Women's suffrage\\\", 'Equal pay for equal work', 'Weakening political machines', 'Rent control laws', 'Child labor laws', 'Improved working conditions in factories', 'Trust-busting']\",\"question\":\"This question refers to the following information.\\nAt a meeting of working girls held at Hull House during a strike in a large shoe factory, the discussions made it clear that the strikers who had been most easily frightened, and therefore the first to capitulate, were naturally those girls who were paying board and were afraid of being put out if they fell too far behind. After a recital of a case of peculiar hardship one of them exclaimed: \\\"Wouldn't it be fine if we had a boarding club of our own, and then we could stand behind each other in a time like this?\\\" After that events moved quickly. We … discussed all the difficulties and fascinations of such an undertaking, and on the first of May, 1891, two comfortable apartments near Hull House were rented and furnished. The Settlement was responsible for the furniture and paid the first month's rent, but beyond that the members managed the club themselves.… At the end of the third year the club occupied all of the six apartments which the original building contained, and numbered fifty members.\\n—Jane Addams, Twenty Years at Hull House, 1912\\nThe perspective of the passage above would most directly support which of the following political goals?\",\"src\":\"ori_mmlu-high_school_us_history\"}","display_format":"text","language":"","answer_status":"published","assets":[],"source_url":"https://huggingface.co/datasets/TIGER-Lab/MMLU-Pro","history":"initial import","indexing_mode":"index","subproblems":[],"grids":[]}