{"kind":"task","effective_mode":"full","benchmark":{"kind":"benchmark","effective_mode":"full","slug":"mmlu-pro","formal_name":"MMLU-Pro","introduction":"MMLU-Pro rebuilds MMLU with ten answer options instead of four and removes items that no longer separate models. Questions span fourteen academic and professional subjects.","introduction_ja":"","introduction_en":"","category":"Category not supplied","task_count":null,"acquisition_status":"Acquisition status not supplied","official_url":"https://huggingface.co/datasets/TIGER-Lab/MMLU-Pro","indexing_mode":"index","profile":{"resources":[],"task_format":"","scoring":"","metric":"","size":"","answer_access":"","license":"","citation":"","maintainer":"","released":"","why_hard":"","related":[],"history":"","construction":"","limitations":"","saturation":"","scores":[]}},"task_id":"9e665839-37af-5275-b899-6015b429c5b1","task_key":"default--test--11109","task_revision_id":"1","upstream_id":"11109","short_description":"According to Mill, to determine whether one pleasure is more valuable than…","config":"default","split":"test","body":"{\"category\":\"philosophy\",\"options\":\"['determine which pleasure most experienced people prefer', 'consult science', 'consult religious leaders', 'determine which one is objectively most pleasurable', 'measure the intensity of each pleasure', 'consider which pleasure is most easily attainable', 'consult our personal preferences', 'determine which pleasure is most universally preferred', 'determine which pleasure lasts the longest', 'consult philosophers of the past']\",\"question\":\"According to Mill, to determine whether one pleasure is more valuable than another, we must _____.\",\"src\":\"ori_mmlu-philosophy\"}","display_format":"text","language":"","answer_status":"published","assets":[],"source_url":"https://huggingface.co/datasets/TIGER-Lab/MMLU-Pro","history":"initial import","indexing_mode":"index","subproblems":[],"grids":[]}