{"kind":"task","effective_mode":"full","benchmark":{"kind":"benchmark","effective_mode":"full","slug":"mmlu-pro","formal_name":"MMLU-Pro","introduction":"MMLU-Pro rebuilds MMLU with ten answer options instead of four and removes items that no longer separate models. Questions span fourteen academic and professional subjects.","introduction_ja":"","introduction_en":"","category":"Category not supplied","task_count":null,"acquisition_status":"Acquisition status not supplied","official_url":"https://huggingface.co/datasets/TIGER-Lab/MMLU-Pro","indexing_mode":"index","profile":{"resources":[],"task_format":"","scoring":"","metric":"","size":"","answer_access":"","license":"","citation":"","maintainer":"","released":"","why_hard":"","related":[],"history":"","construction":"","limitations":"","saturation":"","scores":[]}},"task_id":"e872dce7-5b81-59a2-a765-57db62c7ab75","task_key":"default--test--1257","task_revision_id":"1","upstream_id":"1257","short_description":"An older and younger brother had always done everything together. They had grown…","config":"default","split":"test","body":"{\"category\":\"law\",\"options\":\"[\\\"did not violate the older brother's constitutional rights if he separately paid for his representation.\\\", \\\"violated the older brother's Sixth Amendment right to counsel.\\\", \\\"violated both brothers' Sixth Amendment right to counsel.\\\", 'did not violate any constitutional rights as long as the attorney had the consent of both brothers.', 'constituted an impermissible conflict of interest only if the brothers had not been informed of the potential conflict.', \\\"violated the younger brother's constitutional rights, but did not infringe on the older brother's rights if he was not threatened with a gun.\\\", \\\"violated the younger brother's constitutional rights, but did not infringe on the older brother's constitutional rights.\\\", \\\"violated the younger brother's Sixth Amendment right to counsel.\\\", 'did not constitute a conflict of interest if the attorney had the consent of both brothers.']\",\"question\":\"An older and younger brother had always done everything together. They had grown up together, gone to school together, and partied together. One night, while drinking at a bar, both brothers decided that they wanted a little more action than the bar was offering up, so they left and went to a strip club. At the strip club, they paid for private dances and attempted to have conversations with the women. As hard as they tried, the women who worked in the club did not seem as interested in the brothers as they were in the women. As the night wore on, the brothers decided that they would wait behind the club until closing for the women to come out. The victim, who had given a dance to one of the brothers, came out and proceeded to walk to her car. One of the brothers grabbed the victim from behind and pulled her to the ground while the other brother sexually assaulted her. Both brothers were arrested and charged with the rape of the victim. Their long-time family attorney represented them both at their trial. The younger brother intended to testify that his older brother had forced him to hold the victim down by threatening him with a gun. If appropriate objections are made by either brother, the court should rule that the attorney's representation of the brothers at trial\",\"src\":\"ori_mmlu-professional_law\"}","display_format":"text","language":"","answer_status":"published","assets":[],"source_url":"https://huggingface.co/datasets/TIGER-Lab/MMLU-Pro","history":"initial import","indexing_mode":"index","subproblems":[],"grids":[]}