{"kind":"task","effective_mode":"full","benchmark":{"kind":"benchmark","effective_mode":"full","slug":"mmlu-pro","formal_name":"MMLU-Pro","introduction":"MMLU-Pro rebuilds MMLU with ten answer options instead of four and removes items that no longer separate models. Questions span fourteen academic and professional subjects.","introduction_ja":"","introduction_en":"","category":"Category not supplied","task_count":null,"acquisition_status":"Acquisition status not supplied","official_url":"https://huggingface.co/datasets/TIGER-Lab/MMLU-Pro","indexing_mode":"index","profile":{"resources":[],"task_format":"","scoring":"","metric":"","size":"","answer_access":"","license":"","citation":"","maintainer":"","released":"","why_hard":"","related":[],"history":"","construction":"","limitations":"","saturation":"","scores":[]}},"task_id":"22f39341-acdb-5dc7-9bbe-1d2db7a01696","task_key":"default--test--11181","task_revision_id":"1","upstream_id":"11181","short_description":"Before his conversion, what was Tolstoy surrounded by?","config":"default","split":"test","body":"{\"category\":\"philosophy\",\"options\":\"['things that no one would want', 'what is considered deep insight', 'a life of poverty and hardship', 'the pressures of aristocratic society', 'what is considered utter despair', 'an abundance of wealth and power', 'what is considered complete happiness', 'what is considered heavenly bliss', 'a world of literature and art', 'the turmoil of a war']\",\"question\":\"Before his conversion, what was Tolstoy surrounded by?\",\"src\":\"ori_mmlu-philosophy\"}","display_format":"text","language":"","answer_status":"published","assets":[],"source_url":"https://huggingface.co/datasets/TIGER-Lab/MMLU-Pro","history":"initial import","indexing_mode":"index","subproblems":[],"grids":[]}