{"kind":"task","effective_mode":"full","benchmark":{"kind":"benchmark","effective_mode":"full","slug":"mmlu","formal_name":"MMLU","introduction":"MMLU asks four-way multiple-choice questions across 57 subjects, from elementary material to professional-level examinations. Since 2020 it has been the most widely cited single reference point for general capability.","introduction_ja":"","introduction_en":"","category":"Category not supplied","task_count":null,"acquisition_status":"Acquisition status not supplied","official_url":"https://github.com/hendrycks/test","indexing_mode":"index","profile":{"resources":[],"task_format":"","scoring":"","metric":"","size":"","answer_access":"","license":"","citation":"","maintainer":"","released":"","why_hard":"","related":[],"history":"","construction":"","limitations":"","saturation":"","scores":[]}},"task_id":"2a839896-ad0c-57f5-a513-ee1393ba4d69","task_key":"all--test--2a839896-ad0c-57f5-a513-ee1393ba4d69","task_revision_id":"1","upstream_id":"","short_description":"Walking down a street late at night, an adult male pedestrian notices a young…","config":"all","split":"test","body":"{\"choices\":\"['The person requiring aid appearing to be of a lower socioeconomic class rather than a higher one', 'The presence of another group of people one block up the street', 'The person requiring aid appearing to be the same ethnicity rather than a different one', 'The presence of one other person who is already approaching the girl']\",\"question\":\"Walking down a street late at night, an adult male pedestrian notices a young female on the ground, not moving. The female is on the opposite side of the street. Crossing the street, the pedestrian notices that the young woman appears both much wealthier than he is and is of a different ethnicity. Seeing no one else present, the pedestrian renders aid. According to the bystander effect, which of the following would change how the pedestrian reacts?\",\"subject\":\"college_medicine\"}","display_format":"text","language":"","answer_status":"published","assets":[],"source_url":"https://github.com/hendrycks/test","history":"initial import","indexing_mode":"index","subproblems":[],"grids":[]}