{"kind":"task","effective_mode":"full","benchmark":{"kind":"benchmark","effective_mode":"full","slug":"mmlu-pro","formal_name":"MMLU-Pro","introduction":"MMLU-Pro rebuilds MMLU with ten answer options instead of four and removes items that no longer separate models. Questions span fourteen academic and professional subjects.","introduction_ja":"","introduction_en":"","category":"Category not supplied","task_count":null,"acquisition_status":"Acquisition status not supplied","official_url":"https://huggingface.co/datasets/TIGER-Lab/MMLU-Pro","indexing_mode":"index","profile":{"resources":[],"task_format":"","scoring":"","metric":"","size":"","answer_access":"","license":"","citation":"","maintainer":"","released":"","why_hard":"","related":[],"history":"","construction":"","limitations":"","saturation":"","scores":[]}},"task_id":"408aeb6c-79aa-5f85-8d5d-a9c6f8de8556","task_key":"default--test--1638","task_revision_id":"1","upstream_id":"1638","short_description":"A rider entered a subway car at the 42nd Street station. Because all of the…","config":"default","split":"test","body":"{\"category\":\"law\",\"options\":\"['liable, because the rider believed that the girl consented to the contact under the circumstances.', \\\"not liable, because the rider's actions were unintentional.\\\", 'not liable, because the girl did not express any discomfort at the time of contact.', \\\"not liable, because the subway's sudden stop was an unforeseeable event.\\\", 'liable, as he should have found another way to secure his balance.', 'liable, because the rider did not apologize for the contact.', 'liable for battery.', 'not liable, because the girl was not harmed by the contact.', 'liable for assault.', \\\"not liable, because the rider's conduct was socially acceptable under the circumstances.\\\"]\",\"question\":\"A rider entered a subway car at the 42nd Street station. Because all of the seats were occupied, the rider stood in the subway car and grabbed a pole to secure his balance. As the subway car was proceeding downtown, the rider glanced at a girl standing next to him. Suddenly, the subway car made an unexpected stop. The rider momentarily lost his balance and grabbed the girl around the waist (to avoid falling). Once the rider regained his balance, he removed his hands from the girl's waist and grasped the pole again. In a civil action instituted by the girl against the rider, he will most likely be found\",\"src\":\"ori_mmlu-professional_law\"}","display_format":"text","language":"","answer_status":"published","assets":[],"source_url":"https://huggingface.co/datasets/TIGER-Lab/MMLU-Pro","history":"initial import","indexing_mode":"index","subproblems":[],"grids":[]}