{"kind":"task","effective_mode":"full","benchmark":{"kind":"benchmark","effective_mode":"full","slug":"mmlu-pro","formal_name":"MMLU-Pro","introduction":"MMLU-Pro rebuilds MMLU with ten answer options instead of four and removes items that no longer separate models. Questions span fourteen academic and professional subjects.","introduction_ja":"","introduction_en":"","category":"Category not supplied","task_count":null,"acquisition_status":"Acquisition status not supplied","official_url":"https://huggingface.co/datasets/TIGER-Lab/MMLU-Pro","indexing_mode":"index","profile":{"resources":[],"task_format":"","scoring":"","metric":"","size":"","answer_access":"","license":"","citation":"","maintainer":"","released":"","why_hard":"","related":[],"history":"","construction":"","limitations":"","saturation":"","scores":[]}},"task_id":"5a0c4289-655c-552f-b59f-9101dccbe5bf","task_key":"default--test--10408","task_revision_id":"1","upstream_id":"10408","short_description":"An online store uses 6-bit binary sequences to identify each unique item for…","config":"default","split":"test","body":"{\"category\":\"computer science\",\"options\":\"['10 times as many items can be uniquely identified.', '2 more items can be uniquely identified.', '2 times as many items can be uniquely identified.', 'The number of uniquely identified items will remain the same.', '3 times as many items can be uniquely identified.', 'Half as many items can be uniquely identified.', '128 more items can be uniquely identified.', '10 more items can be uniquely identified.', '4 times as many items can be uniquely identified.']\",\"question\":\"An online store uses 6-bit binary sequences to identify each unique item for sale. The store plans to increase the number of items it sells and is considering using 7-bit binary sequences. Which of the following best describes the result of using 7-bit sequences instead of 6- bit sequences?\",\"src\":\"ori_mmlu-high_school_computer_science\"}","display_format":"text","language":"","answer_status":"published","assets":[],"source_url":"https://huggingface.co/datasets/TIGER-Lab/MMLU-Pro","history":"initial import","indexing_mode":"index","subproblems":[],"grids":[]}