{"kind":"task","effective_mode":"full","benchmark":{"kind":"benchmark","effective_mode":"full","slug":"mmlu-pro","formal_name":"MMLU-Pro","introduction":"MMLU-Pro rebuilds MMLU with ten answer options instead of four and removes items that no longer separate models. Questions span fourteen academic and professional subjects.","introduction_ja":"","introduction_en":"","category":"Category not supplied","task_count":null,"acquisition_status":"Acquisition status not supplied","official_url":"https://huggingface.co/datasets/TIGER-Lab/MMLU-Pro","indexing_mode":"index","profile":{"resources":[],"task_format":"","scoring":"","metric":"","size":"","answer_access":"","license":"","citation":"","maintainer":"","released":"","why_hard":"","related":[],"history":"","construction":"","limitations":"","saturation":"","scores":[]}},"task_id":"7d1ba0f3-9334-5d49-8757-c9a0e404a190","task_key":"default--test--1528","task_revision_id":"1","upstream_id":"1528","short_description":"A man worked as a clerk in a gourmet coffee shop. His friend often stopped by…","config":"default","split":"test","body":"{\"category\":\"law\",\"options\":\"['prevail, because the company should have had a warning sign about the dangers of the machine.', \\\"not prevail, because the company is not liable for the friend's negligence.\\\", \\\"prevail, because the company would be vicariously liable for the man's unauthorized actions.\\\", 'not prevail, because the customer should have waited for an employee to operate the machine.', 'not prevail, because the friend was acting as a Good Samaritan.', 'not prevail, because the man acted outside the scope of employment by entrusting the supervision of the shop to his friend.', 'prevail, because the company failed to properly train the man on how to operate the machine.', 'not prevail, because the friend was not an employee of the coffee shop.', 'prevail, because the company failed to ensure that the man was never left alone in the shop.', \\\"prevail, because the friend's negligence would be imputed to the company.\\\"]\",\"question\":\"A man worked as a clerk in a gourmet coffee shop. His friend often stopped by the shop to chat with the man while the man was working. One afternoon, while the friend was visiting the shop, the man needed to use the bathroom. There were no customers in the shop at that moment, so the man asked the friend to watch the store for him while he went in the back to use the bathroom. The friend agreed to do so. While the man was in the bathroom, a customer came into the store. She wanted to purchase some freshly ground coffee. The shop had a grinding machine that customers could use. The customer selected a bag of coffee beans and then took it to the grinding machine. She was unsure how to start the machine, so she asked the friend for help. The friend tried to show the customer how to use the machine, but he was not familiar with how it worked, and he gave the customer erroneous instructions. The customer tried to follow the friend's instructions, but this led to her getting her finger pinched and cut by the machine. If the customer asserts a claim against the coffee shop company for her finger injury, the customer will most likely\",\"src\":\"ori_mmlu-professional_law\"}","display_format":"text","language":"","answer_status":"published","assets":[],"source_url":"https://huggingface.co/datasets/TIGER-Lab/MMLU-Pro","history":"initial import","indexing_mode":"index","subproblems":[],"grids":[]}