{"kind":"task","effective_mode":"full","benchmark":{"kind":"benchmark","effective_mode":"full","slug":"mmlu-pro","formal_name":"MMLU-Pro","introduction":"MMLU-Pro rebuilds MMLU with ten answer options instead of four and removes items that no longer separate models. Questions span fourteen academic and professional subjects.","introduction_ja":"","introduction_en":"","category":"Category not supplied","task_count":null,"acquisition_status":"Acquisition status not supplied","official_url":"https://huggingface.co/datasets/TIGER-Lab/MMLU-Pro","indexing_mode":"index","profile":{"resources":[],"task_format":"","scoring":"","metric":"","size":"","answer_access":"","license":"","citation":"","maintainer":"","released":"","why_hard":"","related":[],"history":"","construction":"","limitations":"","saturation":"","scores":[]}},"task_id":"7703d19f-1edf-5c63-b2fe-543d86ebfdac","task_key":"default--test--11012","task_revision_id":"1","upstream_id":"11012","short_description":"Which of the following describes the fallacy of appeal to loyalty?","config":"default","split":"test","body":"{\"category\":\"philosophy\",\"options\":\"['arguing that someone should do something to be loyal to an opposing person or group', 'arguing that someone should do something to be loyal to a person or a group', 'arguing that someone should do something because it is the traditional thing to do', 'arguing that someone should do something because it will make him or her better liked by others', 'arguing that an idea should be rejected only because of who originated the idea', 'responding to an argument with an expression of derision', 'arguing that an idea should be accepted because it is the most recent or modern idea', 'arguing that an idea should be accepted only because of who originated the idea', 'arguing that someone should do something because it is the popular thing to do', 'arguing that an idea should be accepted because it has been around for a long time']\",\"question\":\"Which of the following describes the fallacy of appeal to loyalty?\",\"src\":\"ori_mmlu-logical_fallacies\"}","display_format":"text","language":"","answer_status":"published","assets":[],"source_url":"https://huggingface.co/datasets/TIGER-Lab/MMLU-Pro","history":"initial import","indexing_mode":"index","subproblems":[],"grids":[]}