{"kind":"task","effective_mode":"full","benchmark":{"kind":"benchmark","effective_mode":"full","slug":"mmlu-pro","formal_name":"MMLU-Pro","introduction":"MMLU-Pro rebuilds MMLU with ten answer options instead of four and removes items that no longer separate models. Questions span fourteen academic and professional subjects.","introduction_ja":"","introduction_en":"","category":"Category not supplied","task_count":null,"acquisition_status":"Acquisition status not supplied","official_url":"https://huggingface.co/datasets/TIGER-Lab/MMLU-Pro","indexing_mode":"index","profile":{"resources":[],"task_format":"","scoring":"","metric":"","size":"","answer_access":"","license":"","citation":"","maintainer":"","released":"","why_hard":"","related":[],"history":"","construction":"","limitations":"","saturation":"","scores":[]}},"task_id":"b9a63258-5f29-5171-8bd8-64a79b77b35d","task_key":"default--test--1563","task_revision_id":"1","upstream_id":"1563","short_description":"A man entered a half-opened window to a neighbor's trailer and waited in the…","config":"default","split":"test","body":"{\"category\":\"law\",\"options\":\"['No, his belief that she consented in advance was unreasonable and impermissible.', 'No, because he did not receive explicit consent for this specific encounter.', 'Yes, because he entered through an unlocked window which could be seen as an invitation.', 'Yes, because they had a history of consensual relations and this could be seen as a continuation of that.', \\\"Yes, rape is a specific intent crime and there's no evidence that he intended a criminal result.\\\", 'Yes, his belief that she consented was reasonable based on prior experience.', 'No, rape of an unconscious woman is a strict liability crime that does not permit a defense.', 'No, because the woman was asleep and unable to give consent.', \\\"Yes, because he had no intent to harm or force her into anything she didn't want.\\\", 'No, because consent cannot be assumed from past experiences.']\",\"question\":\"A man entered a half-opened window to a neighbor's trailer and waited in the dark bedroom inside. After a while, he got in bed and had sexual relations with the woman while she slept. At trial, he claimed that he believed that the woman had consented because he had dated her many times and had consensual sexual relations with her many times in the past. He hadn't seen her in about one week but assumed her consent. He denied specific intent to rape and claimed a desire to surprise her with a \\\"romantic\\\" variation of their usual activities. The woman admitted to prior consensual relations but denied consent on this occasion. Has he stated a viable defense to rape?\",\"src\":\"ori_mmlu-professional_law\"}","display_format":"text","language":"","answer_status":"published","assets":[],"source_url":"https://huggingface.co/datasets/TIGER-Lab/MMLU-Pro","history":"initial import","indexing_mode":"index","subproblems":[],"grids":[]}