{"kind":"task","effective_mode":"full","benchmark":{"kind":"benchmark","effective_mode":"full","slug":"mmlu-pro","formal_name":"MMLU-Pro","introduction":"MMLU-Pro rebuilds MMLU with ten answer options instead of four and removes items that no longer separate models. Questions span fourteen academic and professional subjects.","introduction_ja":"","introduction_en":"","category":"Category not supplied","task_count":null,"acquisition_status":"Acquisition status not supplied","official_url":"https://huggingface.co/datasets/TIGER-Lab/MMLU-Pro","indexing_mode":"index","profile":{"resources":[],"task_format":"","scoring":"","metric":"","size":"","answer_access":"","license":"","citation":"","maintainer":"","released":"","why_hard":"","related":[],"history":"","construction":"","limitations":"","saturation":"","scores":[]}},"task_id":"c99e41a4-f2ad-51c2-841e-23780290bba4","task_key":"default--test--6257","task_revision_id":"1","upstream_id":"6257","short_description":"A 22-year-old woman comes to the office because of urticaria. This is her first…","config":"default","split":"test","body":"{\"category\":\"health\",\"options\":\"['Refer her to an allergist', 'Initiate immunotherapy', 'Advise her to avoid all potential allergens', 'Determine the serum IgE concentration', 'Prescribe antibiotics immediately', 'Treat the symptoms', 'Schedule her for allergy testing', 'Determine the total eosinophil count', 'Recommend a skin biopsy', 'Suggest immediate hospitalization']\",\"question\":\"A 22-year-old woman comes to the office because of urticaria. This is her first episode of urticaria and it has occurred and then resolved several times in the past week. The history and physical examination disclose no abnormalities. Which of the following is the most appropriate course of action?\",\"src\":\"ori_mmlu-professional_medicine\"}","display_format":"text","language":"","answer_status":"published","assets":[],"source_url":"https://huggingface.co/datasets/TIGER-Lab/MMLU-Pro","history":"initial import","indexing_mode":"index","subproblems":[],"grids":[]}