{"kind":"task","effective_mode":"full","benchmark":{"kind":"benchmark","effective_mode":"full","slug":"mmlu-pro","formal_name":"MMLU-Pro","introduction":"MMLU-Pro rebuilds MMLU with ten answer options instead of four and removes items that no longer separate models. Questions span fourteen academic and professional subjects.","introduction_ja":"","introduction_en":"","category":"Category not supplied","task_count":null,"acquisition_status":"Acquisition status not supplied","official_url":"https://huggingface.co/datasets/TIGER-Lab/MMLU-Pro","indexing_mode":"index","profile":{"resources":[],"task_format":"","scoring":"","metric":"","size":"","answer_access":"","license":"","citation":"","maintainer":"","released":"","why_hard":"","related":[],"history":"","construction":"","limitations":"","saturation":"","scores":[]}},"task_id":"7290c478-08e4-508d-af00-3d0561f96bf3","task_key":"default--test--11043","task_revision_id":"1","upstream_id":"11043","short_description":"According to Singer, which of the following historical defenders of the…","config":"default","split":"test","body":"{\"category\":\"philosophy\",\"options\":\"['Aristotle', 'Friedrich Nietzsche', 'David Hume', 'Jeremy Bentham', 'John Stuart Mill', 'Thomas Hobbes', 'Immanuel Kant', 'Jean-Jacques Rousseau', 'Socrates', 'Rene Descartes']\",\"question\":\" According to Singer, which of the following historical defenders of the principle of equal consideration of interests was one of the few to recognize that the principle applies to members of other species as well as to our own?\",\"src\":\"ori_mmlu-moral_disputes\"}","display_format":"text","language":"","answer_status":"published","assets":[],"source_url":"https://huggingface.co/datasets/TIGER-Lab/MMLU-Pro","history":"initial import","indexing_mode":"index","subproblems":[],"grids":[]}