{"kind":"task","effective_mode":"full","benchmark":{"kind":"benchmark","effective_mode":"full","slug":"mmlu-pro","formal_name":"MMLU-Pro","introduction":"MMLU-Pro rebuilds MMLU with ten answer options instead of four and removes items that no longer separate models. Questions span fourteen academic and professional subjects.","introduction_ja":"","introduction_en":"","category":"Category not supplied","task_count":null,"acquisition_status":"Acquisition status not supplied","official_url":"https://huggingface.co/datasets/TIGER-Lab/MMLU-Pro","indexing_mode":"index","profile":{"resources":[],"task_format":"","scoring":"","metric":"","size":"","answer_access":"","license":"","citation":"","maintainer":"","released":"","why_hard":"","related":[],"history":"","construction":"","limitations":"","saturation":"","scores":[]}},"task_id":"ad1e8223-6d29-5e34-9959-8f5b3bef9a06","task_key":"default--test--10483","task_revision_id":"1","upstream_id":"10483","short_description":"In which of the following representations of numbers by 8-bit words is the…","config":"default","split":"test","body":"{\"category\":\"computer science\",\"options\":\"['I and II only', 'I, II, and III', 'III only', 'II and III only', 'I and III only', 'I only', 'II only', 'All are out of range', 'None of the above', \\\"Only in signed magnitude and one's complement\\\"]\",\"question\":\"In which of the following representations of numbers by 8-bit words is the addition of the integers 109 and -42 within range?\\nI. One's complement\\nII. Two's complement\\nIII. Sign and magnitude\",\"src\":\"ori_mmlu-college_computer_science\"}","display_format":"text","language":"","answer_status":"published","assets":[],"source_url":"https://huggingface.co/datasets/TIGER-Lab/MMLU-Pro","history":"initial import","indexing_mode":"index","subproblems":[],"grids":[]}