{"kind":"task","effective_mode":"full","benchmark":{"kind":"benchmark","effective_mode":"full","slug":"mmlu-pro","formal_name":"MMLU-Pro","introduction":"MMLU-Pro rebuilds MMLU with ten answer options instead of four and removes items that no longer separate models. Questions span fourteen academic and professional subjects.","introduction_ja":"","introduction_en":"","category":"Category not supplied","task_count":null,"acquisition_status":"Acquisition status not supplied","official_url":"https://huggingface.co/datasets/TIGER-Lab/MMLU-Pro","indexing_mode":"index","profile":{"resources":[],"task_format":"","scoring":"","metric":"","size":"","answer_access":"","license":"","citation":"","maintainer":"","released":"","why_hard":"","related":[],"history":"","construction":"","limitations":"","saturation":"","scores":[]}},"task_id":"6e56d535-67da-53c7-a408-95238d982cff","task_key":"default--test--1589","task_revision_id":"1","upstream_id":"1589","short_description":"This jurisdiction has the following bribery statute in effect:\"Any person who…","config":"default","split":"test","body":"{\"category\":\"law\",\"options\":\"[\\\"guilty, because the commission member's agreement to accept the $10,000 was sufficient to form a conspiratorial objective.\\\", 'guilty, because he gave the commission member the $10,000 in exchange for his approval of the zoning variance.', 'not guilty, because the commission member did not receive a thing of value, since he would have approved the variance regardless of receiving any payment from the developer.', 'not guilty, because there was no true agreement between the parties.']\",\"question\":\"This jurisdiction has the following bribery statute in effect:\\\"Any person who offers or gives a thing of value to a government officeholder in exchange for official action is guilty of bribery. \\\"A real estate developer owned a large parcel of land in the suburbs. Although the developer wanted to build an office building on the property, the land was zoned residential. Due to the residential zoning, the developer could not pursue his planned development unless he received a variance from the building commission. The developer held a meeting with a member of the building commission to solicit his approval in securing a zoning variance. To do so, the developer gave the commission member $10,000 in exchange for his support in approving the zoning variance. Thereupon, the commission member voted to approve the variance, thus making it possible for the developer to commence construction of the office building. The developer was subsequently prosecuted for conspiracy to commit bribery. During the course of the trial, the commission member testified that he faked the agreement with the developer and would have approved the zoning variance regardless of whether the developer gave him any money. Furthermore, in his defense, the developer presented evidence that the other six members of the building commission voted affirmatively to approve the variance. If the jury believed that the commission member would have approved the variance even had he not received the $10,000, the developer should be found\",\"src\":\"ori_mmlu-professional_law\"}","display_format":"text","language":"","answer_status":"published","assets":[],"source_url":"https://huggingface.co/datasets/TIGER-Lab/MMLU-Pro","history":"initial import","indexing_mode":"index","subproblems":[],"grids":[]}