{"kind":"task","effective_mode":"full","benchmark":{"kind":"benchmark","effective_mode":"full","slug":"mmlu-pro","formal_name":"MMLU-Pro","introduction":"MMLU-Pro rebuilds MMLU with ten answer options instead of four and removes items that no longer separate models. Questions span fourteen academic and professional subjects.","introduction_ja":"","introduction_en":"","category":"Category not supplied","task_count":null,"acquisition_status":"Acquisition status not supplied","official_url":"https://huggingface.co/datasets/TIGER-Lab/MMLU-Pro","indexing_mode":"index","profile":{"resources":[],"task_format":"","scoring":"","metric":"","size":"","answer_access":"","license":"","citation":"","maintainer":"","released":"","why_hard":"","related":[],"history":"","construction":"","limitations":"","saturation":"","scores":[]}},"task_id":"a7795171-b167-5881-b20d-93c6290283f2","task_key":"default--test--6647","task_revision_id":"1","upstream_id":"6647","short_description":"Walking down a street late at night, an adult male pedestrian notices a young…","config":"default","split":"test","body":"{\"category\":\"health\",\"options\":\"['The presence of a security camera on the street', 'The presence of one other person who is already approaching the girl', 'The person requiring aid appearing to be of a lower socioeconomic class rather than a higher one', 'The presence of a large, intimidating dog near the girl', 'The person requiring aid appearing to be the same ethnicity rather than a different one', 'The person requiring aid appearing to be less wealthy than initially thought', 'The person requiring aid appearing to be older rather than younger', 'The person requiring aid appearing to be of the same gender rather than a different one']\",\"question\":\"Walking down a street late at night, an adult male pedestrian notices a young female on the ground, not moving. The female is on the opposite side of the street. Crossing the street, the pedestrian notices that the young woman appears both much wealthier than he is and is of a different ethnicity. Seeing no one else present, the pedestrian renders aid. According to the bystander effect, which of the following would change how the pedestrian reacts?\",\"src\":\"ori_mmlu-college_medicine\"}","display_format":"text","language":"","answer_status":"published","assets":[],"source_url":"https://huggingface.co/datasets/TIGER-Lab/MMLU-Pro","history":"initial import","indexing_mode":"index","subproblems":[],"grids":[]}