{"kind":"task","effective_mode":"full","benchmark":{"kind":"benchmark","effective_mode":"full","slug":"mmlu-pro","formal_name":"MMLU-Pro","introduction":"MMLU-Pro rebuilds MMLU with ten answer options instead of four and removes items that no longer separate models. Questions span fourteen academic and professional subjects.","introduction_ja":"","introduction_en":"","category":"Category not supplied","task_count":null,"acquisition_status":"Acquisition status not supplied","official_url":"https://huggingface.co/datasets/TIGER-Lab/MMLU-Pro","indexing_mode":"index","profile":{"resources":[],"task_format":"","scoring":"","metric":"","size":"","answer_access":"","license":"","citation":"","maintainer":"","released":"","why_hard":"","related":[],"history":"","construction":"","limitations":"","saturation":"","scores":[]}},"task_id":"66ef6b99-4dc6-50de-8861-4ddc2863e863","task_key":"default--test--2492","task_revision_id":"1","upstream_id":"2492","short_description":"If a group of five people are watching a movie and upon conclusionfour of them…","config":"default","split":"test","body":"{\"category\":\"psychology\",\"options\":\"['it was a terrible movie', 'it was an excellent movie', 'I liked the movie, despite what others think', 'the movie was okay', \\\"The movie was not as bad as you're saying\\\", 'The special effects were good, but the story was lacking', \\\"The acting was good, but the plot didn't make sense\\\", 'no opinion about the movie', \\\"The movie had its moments but wasn't great overall\\\", 'Everyone has their own taste in movies']\",\"question\":\"If a group of five people are watching a movie and upon conclusionfour of them state that the movie was terrible, whatwill the fifth person (who happened to like the movie) probablysay?\",\"src\":\"stemez-Psychology\"}","display_format":"text","language":"","answer_status":"published","assets":[],"source_url":"https://huggingface.co/datasets/TIGER-Lab/MMLU-Pro","history":"initial import","indexing_mode":"index","subproblems":[],"grids":[]}