{"kind":"task","effective_mode":"full","benchmark":{"kind":"benchmark","effective_mode":"full","slug":"mmlu-pro","formal_name":"MMLU-Pro","introduction":"MMLU-Pro rebuilds MMLU with ten answer options instead of four and removes items that no longer separate models. Questions span fourteen academic and professional subjects.","introduction_ja":"","introduction_en":"","category":"Category not supplied","task_count":null,"acquisition_status":"Acquisition status not supplied","official_url":"https://huggingface.co/datasets/TIGER-Lab/MMLU-Pro","indexing_mode":"index","profile":{"resources":[],"task_format":"","scoring":"","metric":"","size":"","answer_access":"","license":"","citation":"","maintainer":"","released":"","why_hard":"","related":[],"history":"","construction":"","limitations":"","saturation":"","scores":[]}},"task_id":"8645a1eb-57a3-5e9b-85c4-44fcc4bdf418","task_key":"default--test--8037","task_revision_id":"1","upstream_id":"8037","short_description":"Suppose A and B are n × n matrices with real entries. Which of the follow are…","config":"default","split":"test","body":"{\"category\":\"math\",\"options\":\"['All of them', 'I, II and III', 'I only', 'I and III only', 'II, III and I', 'III only', 'II only', 'I and II only', 'II and III only', 'None of them']\",\"question\":\"Suppose A and B are n × n matrices with real entries. Which of the follow are TRUE?\\nI. The trace of A^2 is nonnegative.\\nII. If A^2 = A, then the trace of A is nonnegative.\\nIII. The trace of AB is the product of the traces of A and B.\",\"src\":\"ori_mmlu-college_mathematics\"}","display_format":"text","language":"","answer_status":"published","assets":[],"source_url":"https://huggingface.co/datasets/TIGER-Lab/MMLU-Pro","history":"initial import","indexing_mode":"index","subproblems":[],"grids":[]}