{"kind":"task","effective_mode":"full","benchmark":{"kind":"benchmark","effective_mode":"full","slug":"mmlu-pro","formal_name":"MMLU-Pro","introduction":"MMLU-Pro rebuilds MMLU with ten answer options instead of four and removes items that no longer separate models. Questions span fourteen academic and professional subjects.","introduction_ja":"","introduction_en":"","category":"Category not supplied","task_count":null,"acquisition_status":"Acquisition status not supplied","official_url":"https://huggingface.co/datasets/TIGER-Lab/MMLU-Pro","indexing_mode":"index","profile":{"resources":[],"task_format":"","scoring":"","metric":"","size":"","answer_access":"","license":"","citation":"","maintainer":"","released":"","why_hard":"","related":[],"history":"","construction":"","limitations":"","saturation":"","scores":[]}},"task_id":"b50f3a38-9c0c-5d89-bbcc-7a9c29117872","task_key":"default--test--3449","task_revision_id":"1","upstream_id":"3449","short_description":"A women feeds her new cat daily with canned cat food. As she opens the cans with…","config":"default","split":"test","body":"{\"category\":\"biology\",\"options\":\"['The cat believes that the can opener itself is a toy.', 'The cat has learned that the can opener noise means someone is in the kitchen.', 'The cat recognizes the can opener as a signal that its owner is nearby.', 'The cat enjoys the routine of the feeding process and the sound signifies the start.', 'The cat is trained to respond to the sound of the can opener.', 'The cat is attracted by the noise of the can opener.', 'The cat likes the sound of the can opener.', 'The cat has associated the sound of the can opener with the presentation of food.', 'The cat is curious about the source of the noise and comes to investigate.', 'The cat has been conditioned to come for a treat whenever it hears any electrical appliance.']\",\"question\":\"A women feeds her new cat daily with canned cat food. As she opens the cans with a noisy electric can opener, she calls the cat. The cat comes in response, to its name, and the woman feeds it. Why does the cat eventually come running when it hears the electric can opener?\",\"src\":\"stemez-Biology\"}","display_format":"text","language":"","answer_status":"published","assets":[],"source_url":"https://huggingface.co/datasets/TIGER-Lab/MMLU-Pro","history":"initial import","indexing_mode":"index","subproblems":[],"grids":[]}