{"kind":"task","effective_mode":"full","benchmark":{"kind":"benchmark","effective_mode":"full","slug":"mmlu-pro","formal_name":"MMLU-Pro","introduction":"MMLU-Pro rebuilds MMLU with ten answer options instead of four and removes items that no longer separate models. Questions span fourteen academic and professional subjects.","introduction_ja":"","introduction_en":"","category":"Category not supplied","task_count":null,"acquisition_status":"Acquisition status not supplied","official_url":"https://huggingface.co/datasets/TIGER-Lab/MMLU-Pro","indexing_mode":"index","profile":{"resources":[],"task_format":"","scoring":"","metric":"","size":"","answer_access":"","license":"","citation":"","maintainer":"","released":"","why_hard":"","related":[],"history":"","construction":"","limitations":"","saturation":"","scores":[]}},"task_id":"ce38e848-341b-59dd-9f36-f1b054ff2a12","task_key":"default--test--2573","task_revision_id":"1","upstream_id":"2573","short_description":"At the termination of an extensive training program, management may expect to…","config":"default","split":"test","body":"{\"category\":\"psychology\",\"options\":\"['primarily high-ability trainees have improved in performance', 'most trainees have improved, but relative standing is generally maintained', 'all trainees have significantly decreased in performance', 'there is no noticeable change in the performance of the trainees', 'the performance of the trainees is directly proportional to their initial ability', 'only the trainees who were initially performing poorly have improved', 'the improvement in performance of trainees is not related to the training program', 'primarily low-ability trainees have improved in performance', 'all trainees are at the same performance level', 'only the trainees with previous experience have improved']\",\"question\":\"At the termination of an extensive training program, management may expect to find chat\",\"src\":\"ori_mmlu-professional_psychology\"}","display_format":"text","language":"","answer_status":"published","assets":[],"source_url":"https://huggingface.co/datasets/TIGER-Lab/MMLU-Pro","history":"initial import","indexing_mode":"index","subproblems":[],"grids":[]}