{"kind":"task","effective_mode":"full","benchmark":{"kind":"benchmark","effective_mode":"full","slug":"mmlu-pro","formal_name":"MMLU-Pro","introduction":"MMLU-Pro rebuilds MMLU with ten answer options instead of four and removes items that no longer separate models. Questions span fourteen academic and professional subjects.","introduction_ja":"","introduction_en":"","category":"Category not supplied","task_count":null,"acquisition_status":"Acquisition status not supplied","official_url":"https://huggingface.co/datasets/TIGER-Lab/MMLU-Pro","indexing_mode":"index","profile":{"resources":[],"task_format":"","scoring":"","metric":"","size":"","answer_access":"","license":"","citation":"","maintainer":"","released":"","why_hard":"","related":[],"history":"","construction":"","limitations":"","saturation":"","scores":[]}},"task_id":"0321b9b6-2c44-571b-9bc5-05bbfb3ae1ee","task_key":"default--test--4682","task_revision_id":"1","upstream_id":"4682","short_description":"This question refers to the following information.","config":"default","split":"test","body":"{\"category\":\"history\",\"options\":\"['Jewish immigrants in New York in the 1880s', 'Detroit autoworkers in the 1930s', 'Populist farmers in the 1890s', 'Factory workers in the Industrial Revolution', 'British soldiers in the 1800s', 'Coal miners in the 1900s', 'Civil Rights activists in the 1960s', 'Women suffragettes in the early 20th century', 'American revolutionaries in the 1770s', 'Slaves in the antebellum South']\",\"question\":\"This question refers to the following information.\\nHere is the case of a woman employed in the manufacturing department of a Broadway house. It stands for a hundred like her own. She averages three dollars a week. Pay is $1.50 for her room; for breakfast she has a cup of coffee; lunch she cannot afford. One meal a day is her allowance. This woman is young, she is pretty. She has \\\"the world before her.\\\" Is it anything less than a miracle if she is guilty of nothing less than the \\\"early and improvident marriage,\\\" against which moralists exclaim as one of the prolific causes of the distresses of the poor? Almost any door might seem to offer a welcome escape from such slavery as this. \\\"I feel so much healthier since I got three square meals a day,\\\" said a lodger in one of the Girls' Homes. Two young sewing-girls came in seeking domestic service, so that they might get enough to eat. They had been only half-fed for some time, and starvation had driven them to the one door at which the pride of the American-born girl will not permit her to knock, though poverty be the price of her independence.\\n—Jacob Riis, How the Other Half Lives, 1890\\nThe situation faced by the young women in the passage above is most directly comparable to which of the following?\",\"src\":\"ori_mmlu-high_school_us_history\"}","display_format":"text","language":"","answer_status":"published","assets":[],"source_url":"https://huggingface.co/datasets/TIGER-Lab/MMLU-Pro","history":"initial import","indexing_mode":"index","subproblems":[],"grids":[]}