{"kind":"task","effective_mode":"full","benchmark":{"kind":"benchmark","effective_mode":"full","slug":"mmlu-pro","formal_name":"MMLU-Pro","introduction":"MMLU-Pro rebuilds MMLU with ten answer options instead of four and removes items that no longer separate models. Questions span fourteen academic and professional subjects.","introduction_ja":"","introduction_en":"","category":"Category not supplied","task_count":null,"acquisition_status":"Acquisition status not supplied","official_url":"https://huggingface.co/datasets/TIGER-Lab/MMLU-Pro","indexing_mode":"index","profile":{"resources":[],"task_format":"","scoring":"","metric":"","size":"","answer_access":"","license":"","citation":"","maintainer":"","released":"","why_hard":"","related":[],"history":"","construction":"","limitations":"","saturation":"","scores":[]}},"task_id":"62473730-9326-51c3-b7a2-108c16469aff","task_key":"default--test--10851","task_revision_id":"1","upstream_id":"10851","short_description":"According to Socrates, it is important that we discover what makes a particular…","config":"default","split":"test","body":"{\"category\":\"philosophy\",\"options\":\"['we would not be able to categorize actions as good or bad.', 'we would not be able to understand the true motivations behind our actions.', 'there would be no universal standard for morality.', 'the primary purpose of human existence--which is to think and to know--is replaced by a focus on morality (acting and doing).', 'we would not be able to improve or evolve our actions and behavior.', 'we can refer only to how people characterize actions without knowing why such actions should be characterized that way.', 'there would be no way to distinguish one kind of action (e.g., a merciful action) from another kind of action (e.g., a just action).', 'people would only act based on their personal beliefs and not on universal truths.', 'no one in society will ever do any action that really is merciful or just, only those actions that they think are merciful or just.', 'we would live in a society where actions are arbitrary and not based on any concrete principle.']\",\"question\":\"According to Socrates, it is important that we discover what makes a particular action (e.g., a merciful or just act) the kind of action that it is, because without such knowledge:\",\"src\":\"ori_mmlu-philosophy\"}","display_format":"text","language":"","answer_status":"published","assets":[],"source_url":"https://huggingface.co/datasets/TIGER-Lab/MMLU-Pro","history":"initial import","indexing_mode":"index","subproblems":[],"grids":[]}