{"kind":"task","effective_mode":"full","benchmark":{"kind":"benchmark","effective_mode":"full","slug":"mmlu-pro","formal_name":"MMLU-Pro","introduction":"MMLU-Pro rebuilds MMLU with ten answer options instead of four and removes items that no longer separate models. Questions span fourteen academic and professional subjects.","introduction_ja":"","introduction_en":"","category":"Category not supplied","task_count":null,"acquisition_status":"Acquisition status not supplied","official_url":"https://huggingface.co/datasets/TIGER-Lab/MMLU-Pro","indexing_mode":"index","profile":{"resources":[],"task_format":"","scoring":"","metric":"","size":"","answer_access":"","license":"","citation":"","maintainer":"","released":"","why_hard":"","related":[],"history":"","construction":"","limitations":"","saturation":"","scores":[]}},"task_id":"a5ba7b74-f4ba-5eda-bad5-a9c0e79da99c","task_key":"default--test--5932","task_revision_id":"1","upstream_id":"5932","short_description":"In the public relations field, what is the most common threat to a client-firm…","config":"default","split":"test","body":"{\"category\":\"other\",\"options\":\"['Resistance to outside advice', \\\"Disagreements over the firm's strategic direction\\\", 'Personality conflicts', \\\"The firm's inability to meet the client's expectations\\\", 'Lack of transparency from the firm', \\\"Clients' questions about costs\\\", \\\"The client's dissatisfaction with the firm's performance\\\", \\\"Superficial grasp of the client's unique problems\\\", 'Inconsistent communication between the client and the firm', \\\"The client's lack of trust in the firm's expertise\\\"]\",\"question\":\"In the public relations field, what is the most common threat to a client-firm relationship?\",\"src\":\"ori_mmlu-public_relations\"}","display_format":"text","language":"","answer_status":"published","assets":[],"source_url":"https://huggingface.co/datasets/TIGER-Lab/MMLU-Pro","history":"initial import","indexing_mode":"index","subproblems":[],"grids":[]}