{"kind":"task","effective_mode":"full","benchmark":{"kind":"benchmark","effective_mode":"full","slug":"mmlu-pro","formal_name":"MMLU-Pro","introduction":"MMLU-Pro rebuilds MMLU with ten answer options instead of four and removes items that no longer separate models. Questions span fourteen academic and professional subjects.","introduction_ja":"","introduction_en":"","category":"Category not supplied","task_count":null,"acquisition_status":"Acquisition status not supplied","official_url":"https://huggingface.co/datasets/TIGER-Lab/MMLU-Pro","indexing_mode":"index","profile":{"resources":[],"task_format":"","scoring":"","metric":"","size":"","answer_access":"","license":"","citation":"","maintainer":"","released":"","why_hard":"","related":[],"history":"","construction":"","limitations":"","saturation":"","scores":[]}},"task_id":"afb7c2bc-610e-5c03-88a2-5d964538aa3b","task_key":"default--test--1940","task_revision_id":"1","upstream_id":"1940","short_description":"At 1:00 a. m. one night, two police officers were driving past a tavern when…","config":"default","split":"test","body":"{\"category\":\"law\",\"options\":\"['recover, because the police officers were responsible for his safety after detaining him.', 'not recover, because he was already intoxicated and causing a disturbance before the police intervened.', \\\"not recover, because the motorist's act was a supervening cause of his injury.\\\", 'not recover, because he willingly chose to be dropped off at the city limits.', 'recover, because the police knew that the man was intoxicated when they dropped him off.', 'not recover, because the man chose to be driven to the city limits.', 'recover, because the police should have realized it was dangerous to drop him off where they did.', 'recover, because the police officers should have provided him with medical assistance due to his intoxication.', 'recover, because the police officers did not ensure his safety before leaving him.', 'not recover, because the police officers gave him a choice and he made his decision.']\",\"question\":\"At 1:00 a. m. one night, two police officers were driving past a tavern when they noticed a man vomiting outside. They stopped their patrol car and walked over to the man, whom they recognized as a troublemaker from the neighboring town. Realizing that the man was extremely intoxicated, they grabbed him and shoved him into their patrol car. They drove the man around in the police car for a few minutes and told him he had two choices. They would either take him to jail for the night or drive him to the city limits and drop him off there. The man indicated that he didn't want to go to jail. One of the officers then said, \\\"Does that mean you want us to drop you off at the city limits?\\\" The man replied, \\\"Yes. \\\" They then drove him to the city limits and left him along a busy highway. Moments later, the man staggered onto the street and was struck by a car driven by a motorist. He suffered multiple injuries. If the man asserts a claim against the police department based on negligence, he will most likely\",\"src\":\"ori_mmlu-professional_law\"}","display_format":"text","language":"","answer_status":"published","assets":[],"source_url":"https://huggingface.co/datasets/TIGER-Lab/MMLU-Pro","history":"initial import","indexing_mode":"index","subproblems":[],"grids":[]}