{"kind":"task","effective_mode":"full","benchmark":{"kind":"benchmark","effective_mode":"full","slug":"mmlu-pro","formal_name":"MMLU-Pro","introduction":"MMLU-Pro rebuilds MMLU with ten answer options instead of four and removes items that no longer separate models. Questions span fourteen academic and professional subjects.","introduction_ja":"","introduction_en":"","category":"Category not supplied","task_count":null,"acquisition_status":"Acquisition status not supplied","official_url":"https://huggingface.co/datasets/TIGER-Lab/MMLU-Pro","indexing_mode":"index","profile":{"resources":[],"task_format":"","scoring":"","metric":"","size":"","answer_access":"","license":"","citation":"","maintainer":"","released":"","why_hard":"","related":[],"history":"","construction":"","limitations":"","saturation":"","scores":[]}},"task_id":"c96c1662-be81-5fc2-bb41-f61661a654e5","task_key":"default--test--1968","task_revision_id":"1","upstream_id":"1968","short_description":"One Sunday afternoon, a sports fan went to a professional football game. While…","config":"default","split":"test","body":"{\"category\":\"law\",\"options\":\"['No, because the sports fan should not have argued with the three men in the first place.', 'No, because the sports fan does not have a legal basis to sue the man who punched him in the mouth.', 'No, because the man who punched him in the mouth did not actually strike the sports fan with the bottle.', 'No, because the sports fan could have avoided the fight.', 'Yes, because the man who punched him in the mouth was the instigator who struck the first blow that started the fracas.', 'Yes, because the sports fan was defending himself from an unprovoked attack.', 'Yes, because the man who punched him in the mouth and the other men were acting in concert.', 'No, because the sports fan cannot offer proof as to which of the men struck the sports fan over the head with the bottle.', 'Yes, because the three men were acting as a group and are therefore jointly responsible.', 'Yes, because all the men were involved in the fight and are therefore liable.']\",\"question\":\"One Sunday afternoon, a sports fan went to a professional football game. While standing in line at a concession stand, he got into an argument with three men ahead of him in the line. The three men were friends attending the game together. One of the men punched the sports fan in the mouth. A fight ensued. During the fracas, one of the other three men hit the sports fan over the head with a bottle, which caused him to suffer a serious concussion. The sports fan does not know for certain which of the three men struck him with the bottle. If the sports fan asserts a claim against the man who punched him in the mouth to recover damages for the head injury, will the sports fan prevail?\",\"src\":\"ori_mmlu-professional_law\"}","display_format":"text","language":"","answer_status":"published","assets":[],"source_url":"https://huggingface.co/datasets/TIGER-Lab/MMLU-Pro","history":"initial import","indexing_mode":"index","subproblems":[],"grids":[]}