{"kind":"task","effective_mode":"full","benchmark":{"kind":"benchmark","effective_mode":"full","slug":"mmlu-pro","formal_name":"MMLU-Pro","introduction":"MMLU-Pro rebuilds MMLU with ten answer options instead of four and removes items that no longer separate models. Questions span fourteen academic and professional subjects.","introduction_ja":"","introduction_en":"","category":"Category not supplied","task_count":null,"acquisition_status":"Acquisition status not supplied","official_url":"https://huggingface.co/datasets/TIGER-Lab/MMLU-Pro","indexing_mode":"index","profile":{"resources":[],"task_format":"","scoring":"","metric":"","size":"","answer_access":"","license":"","citation":"","maintainer":"","released":"","why_hard":"","related":[],"history":"","construction":"","limitations":"","saturation":"","scores":[]}},"task_id":"c08ef3f9-8688-5157-b165-d23c1b63532e","task_key":"default--test--1087","task_revision_id":"1","upstream_id":"1087","short_description":"A wife and her husband were having dinner at a restaurant when the wife excused…","config":"default","split":"test","body":"{\"category\":\"law\",\"options\":\"['recover, because the customer was negligent in not warning her about the egg roll.', 'not recover, because the customer could have prevented the injury by warning the wife of the presences of the egg roll.', 'recover, because the egg roll was on the floor for a substantial period of time before the accident.', 'not recover, because she should have been more careful while walking.', 'recover, because the egg roll on the floor constituted an unsafe condition of the premises.', 'not recover, because the chef was not directly responsible for the egg roll being on the floor.', 'not recover, because the chef did not know that the egg roll was on the floor.', \\\"not recover, because it is not the chef's responsibility to monitor the floor at all times.\\\", 'recover, because the serving tray was placed in a dangerous location.']\",\"question\":\"A wife and her husband were having dinner at a restaurant when the wife excused herself to go to the bathroom. The restaurant was owned and operated by a chef. As the wife was walking past a table where another customer was seated, she slipped and fell on an egg roll that had been lying on the floor for quite some time, although the chef was unaware that it had fallen onto the floor. When she fell, her head struck a serving tray that was located in the aisle. The fall caused the wife to suffer a severe concussion. The customer knew that the egg roll was on the floor, and although he could have done so, he did not warn the wife. If the wife asserts a claim against the chef for the injuries she suffered from the fall, she will most likely\",\"src\":\"ori_mmlu-professional_law\"}","display_format":"text","language":"","answer_status":"published","assets":[],"source_url":"https://huggingface.co/datasets/TIGER-Lab/MMLU-Pro","history":"initial import","indexing_mode":"index","subproblems":[],"grids":[]}