{"kind":"task","effective_mode":"full","benchmark":{"kind":"benchmark","effective_mode":"full","slug":"mmlu-pro","formal_name":"MMLU-Pro","introduction":"MMLU-Pro rebuilds MMLU with ten answer options instead of four and removes items that no longer separate models. Questions span fourteen academic and professional subjects.","introduction_ja":"","introduction_en":"","category":"Category not supplied","task_count":null,"acquisition_status":"Acquisition status not supplied","official_url":"https://huggingface.co/datasets/TIGER-Lab/MMLU-Pro","indexing_mode":"index","profile":{"resources":[],"task_format":"","scoring":"","metric":"","size":"","answer_access":"","license":"","citation":"","maintainer":"","released":"","why_hard":"","related":[],"history":"","construction":"","limitations":"","saturation":"","scores":[]}},"task_id":"5bb1ee69-ad0d-549e-bf27-f566b157162a","task_key":"default--test--1046","task_revision_id":"1","upstream_id":"1046","short_description":"A hiker was enjoying a journey through a large forest when he was caught in an…","config":"default","split":"test","body":"{\"category\":\"law\",\"options\":\"[\\\"The hiker does not owe anything because he was not aware of the cabin's ownership.\\\", 'The hiker must pay for the cost of the food and an additional charge for the inconvenience.', 'The hiker must pay for the cost of the food, but nothing else.', 'The hiker must pay for the cost of the food, the use of the cabin, and any damage caused.', 'The hiker does not owe anything because the importance of his survival outweighs the property interests of the cabin owner.', 'The hiker must pay double the cost of the food as a penalty for unauthorized access.', 'The hiker must pay for the cost of the food and a fine for trespassing.', 'The hiker must pay for the cost of the food and the rental value of the cabin for one night.', 'The hiker must pay for the use of the cabin, but nothing else.']\",\"question\":\"A hiker was enjoying a journey through a large forest when he was caught in an unexpected blizzard. As it grew dark and the snow continued, he realized he was completely lost. He felt sure that he would freeze to death during the night. When all hope seemed lost, he stumbled across a small cabin. He knocked on the door, but no one answered. He tried to open the door, realized it was not locked, and went inside. In the cabin, he found some food to eat and some blankets that kept him warm through the night. By the next morning, the snow had stopped, and he was able to make his way out of the forest. If the owner of the cabin sues the hiker, seeking to recover damages for the cost of the food and for the use of his cabin, what is the most likely outcome of the case?\",\"src\":\"ori_mmlu-professional_law\"}","display_format":"text","language":"","answer_status":"published","assets":[],"source_url":"https://huggingface.co/datasets/TIGER-Lab/MMLU-Pro","history":"initial import","indexing_mode":"index","subproblems":[],"grids":[]}