{"kind":"task","effective_mode":"full","benchmark":{"kind":"benchmark","effective_mode":"full","slug":"mmlu","formal_name":"MMLU","introduction":"MMLU asks four-way multiple-choice questions across 57 subjects, from elementary material to professional-level examinations. Since 2020 it has been the most widely cited single reference point for general capability.","introduction_ja":"","introduction_en":"","category":"Category not supplied","task_count":null,"acquisition_status":"Acquisition status not supplied","official_url":"https://github.com/hendrycks/test","indexing_mode":"index","profile":{"resources":[],"task_format":"","scoring":"","metric":"","size":"","answer_access":"","license":"","citation":"","maintainer":"","released":"","why_hard":"","related":[],"history":"","construction":"","limitations":"","saturation":"","scores":[]}},"task_id":"b7ba9511-63ea-5102-917a-c2a1abc2a1ef","task_key":"all--test--b7ba9511-63ea-5102-917a-c2a1abc2a1ef","task_revision_id":"1","upstream_id":"","short_description":"This question refers to the following information.","config":"all","split":"test","body":"{\"choices\":\"['The elimination of the Black Plague in Europe during the seventeenth century', 'The elimination of fallow fields through the agricultural revolution during the seventeenth century', 'The creation of a strong putting-out system that allowed people to be paid for their work', 'A declining population during the seventeenth century that led to better food yields and less hunger as Europe urbanized']\",\"question\":\"This question refers to the following information.\\nRead the following governmental report.\\nOf the 450 sick persons whom the inhabitants were unable to relieve, 200 were turned out, and these we saw die one by one as they lay on the roadside. A large number still remain, and to each of them it is only possible to dole out the least scrap of bread. We only give bread to those who would otherwise die. The staple dish here consists of mice, which the inhabitants hunt, so desperate are they from hunger. They devour roots which the animals cannot eat; one can, in fact, not put into words the things one sees. . . . This narrative, far from exaggerating, rather understates the horror of the case, for it does not record the hundredth part of the misery in this district. Those who have not witnessed it with their own eyes cannot imagine how great it is. Not a day passes but at least 200 people die of famine in the two provinces. We certify to having ourselves seen herds, not of cattle, but of men and women, wandering about the fields between Rheims and Rhétel, turning up the earth like pigs to find a few roots; and as they can only find rotten ones, and not half enough of them, they become so weak that they have not strength left to seek food. The parish priest at Boult, whose letter we enclose, tells us he has buried three of his parishioners who died of hunger. The rest subsisted on chopped straw mixed with earth, of which they composed a food which cannot be called bread. Other persons in the same place lived on the bodies of animals which had died of disease, and which the curé, otherwise unable to help his people, allowed them to roast at the presbytery fire.\\n—Report of the Estates of Normandy, 1651\\nWhich of the following would be most responsible for ending the problems of hunger mentioned above?\",\"subject\":\"high_school_european_history\"}","display_format":"text","language":"","answer_status":"published","assets":[],"source_url":"https://github.com/hendrycks/test","history":"initial import","indexing_mode":"index","subproblems":[],"grids":[]}