{"kind":"task","effective_mode":"full","benchmark":{"kind":"benchmark","effective_mode":"full","slug":"mmlu-pro","formal_name":"MMLU-Pro","introduction":"MMLU-Pro rebuilds MMLU with ten answer options instead of four and removes items that no longer separate models. Questions span fourteen academic and professional subjects.","introduction_ja":"","introduction_en":"","category":"Category not supplied","task_count":null,"acquisition_status":"Acquisition status not supplied","official_url":"https://huggingface.co/datasets/TIGER-Lab/MMLU-Pro","indexing_mode":"index","profile":{"resources":[],"task_format":"","scoring":"","metric":"","size":"","answer_access":"","license":"","citation":"","maintainer":"","released":"","why_hard":"","related":[],"history":"","construction":"","limitations":"","saturation":"","scores":[]}},"task_id":"c50e4a22-db32-55fb-a9b4-1dcc6c082cac","task_key":"default--test--10554","task_revision_id":"1","upstream_id":"10554","short_description":"Consider an object-oriented language in which all entities are objects. Two…","config":"default","split":"test","body":"{\"category\":\"computer science\",\"options\":\"['one or more instance links and then one or more subclass links', 'one or more subclass links', 'one or more subclass links and then one or more instance links', 'one instance link and then zero or more subclass links', 'one or more instance links', 'one subclass link only', 'zero or more instance links', 'zero or more instance links and then one or more subclass links', 'one subclass link and then one or more instance links', 'one instance link only']\",\"question\":\"Consider an object-oriented language in which all entities are objects. Two relationships arise: (1) the instance relationship, between an object and the class of which that object is a member, and (2) the subclass relationship, between a class and the superclass from which that class inherits properties. In such a language, when a message is sent to an object requesting execution of one of its methods (procedures), the method is located by following\",\"src\":\"ori_mmlu-college_computer_science\"}","display_format":"text","language":"","answer_status":"published","assets":[],"source_url":"https://huggingface.co/datasets/TIGER-Lab/MMLU-Pro","history":"initial import","indexing_mode":"index","subproblems":[],"grids":[]}