{"kind":"task","effective_mode":"full","benchmark":{"kind":"benchmark","effective_mode":"full","slug":"mmlu-pro","formal_name":"MMLU-Pro","introduction":"MMLU-Pro rebuilds MMLU with ten answer options instead of four and removes items that no longer separate models. Questions span fourteen academic and professional subjects.","introduction_ja":"","introduction_en":"","category":"Category not supplied","task_count":null,"acquisition_status":"Acquisition status not supplied","official_url":"https://huggingface.co/datasets/TIGER-Lab/MMLU-Pro","indexing_mode":"index","profile":{"resources":[],"task_format":"","scoring":"","metric":"","size":"","answer_access":"","license":"","citation":"","maintainer":"","released":"","why_hard":"","related":[],"history":"","construction":"","limitations":"","saturation":"","scores":[]}},"task_id":"15ae6fd8-1e6f-538e-aabf-beb434a7c483","task_key":"default--test--10586","task_revision_id":"1","upstream_id":"10586","short_description":"A search engine has a trend-tracking feature that provides information on how…","config":"default","split":"test","body":"{\"category\":\"computer science\",\"options\":\"['In which political candidates are people interested?', 'Which region of the country has the greatest number of people searching for opera performances?', 'What is the cost of a certain electronics product?', 'In what month does a particular sport receive the most searches?']\",\"question\":\"A search engine has a trend-tracking feature that provides information on how popular a search term is. The data can be filtered by geographic region, date, and category. Categories include arts and entertainment, computers and electronics, games, news, people and society, shopping, sports, and travel. Which of the following questions is LEAST likely to be answerable using the trends feature?\",\"src\":\"ori_mmlu-high_school_computer_science\"}","display_format":"text","language":"","answer_status":"published","assets":[],"source_url":"https://huggingface.co/datasets/TIGER-Lab/MMLU-Pro","history":"initial import","indexing_mode":"index","subproblems":[],"grids":[]}