{"kind":"task","effective_mode":"full","benchmark":{"kind":"benchmark","effective_mode":"full","slug":"mmlu-pro","formal_name":"MMLU-Pro","introduction":"MMLU-Pro rebuilds MMLU with ten answer options instead of four and removes items that no longer separate models. Questions span fourteen academic and professional subjects.","introduction_ja":"","introduction_en":"","category":"Category not supplied","task_count":null,"acquisition_status":"Acquisition status not supplied","official_url":"https://huggingface.co/datasets/TIGER-Lab/MMLU-Pro","indexing_mode":"index","profile":{"resources":[],"task_format":"","scoring":"","metric":"","size":"","answer_access":"","license":"","citation":"","maintainer":"","released":"","why_hard":"","related":[],"history":"","construction":"","limitations":"","saturation":"","scores":[]}},"task_id":"4e5d548a-a4f2-5898-aee4-22464276491c","task_key":"default--test--4891","task_revision_id":"1","upstream_id":"4891","short_description":"This question refers to the following information.","config":"default","split":"test","body":"{\"category\":\"history\",\"options\":\"['The establishment of a state-funded program for teaching etiquette and manners to young men', 'The creation of coeducational public high schools paid for by the state', 'The creation of public music schools where students can learn to play various instruments', 'The creation of military training schools for young men', 'The creation of private universities that focus on the humanities and the arts.', 'The creation of private high schools for elite young men with many extracurricular activities', 'The creation of public universities without tuition where all students of both genders could learn', 'The creation of male-only public high schools paid for by the state', 'The establishment of state-funded language schools teaching Latin and Greek', 'The creation of public schools focused on physical education and sports training']\",\"question\":\"This question refers to the following information.\\nI am of opinion that the principal and true profession of the Courtier ought to be that of arms; which I would have him follow actively above all else, and be known among others as bold and strong, and loyal to whomsoever he serves. . . .\\nI would have him well built and shapely of limb, and would have him show strength and lightness and suppleness, and know all bodily exercises that befit a man of war; whereof I think the first should be to handle every sort of weapon well on foot and on horse, to understand the advantages of each, and especially to be familiar with those weapons that are ordinarily used among gentlemen. . . .\\nOur Courtier then will be esteemed excellent and will attain grace in everything, particularly in speaking. . . .\\nI would have him more than passably accomplished in letters, at least in those studies that are called the humanities, and conversant not only with the Latin language but with the Greek, for the sake of the many different things that have been admirably written therein. Let him be well versed in the poets, and not less in the orators and historians, and also proficient in writing verse and prose, especially in this vulgar [vernacular] tongue of ours. . . .\\nYou must know that I am not content with the Courtier unless he be also a musician and unless, besides understanding and being able to read notes, he can play upon divers instruments. . . .\\nI wish our Courtier to guard against getting the name of a liar or a boaster, which sometimes befalls even those who do not deserve it.\\n—From Baldassare Castiglione's Book of the Courtier, 1528\\nWhich of the following reforms would Castiglione most likely want to institute?\",\"src\":\"ori_mmlu-high_school_european_history\"}","display_format":"text","language":"","answer_status":"published","assets":[],"source_url":"https://huggingface.co/datasets/TIGER-Lab/MMLU-Pro","history":"initial import","indexing_mode":"index","subproblems":[],"grids":[]}