{"kind":"task","effective_mode":"full","benchmark":{"kind":"benchmark","effective_mode":"full","slug":"mmlu-pro","formal_name":"MMLU-Pro","introduction":"MMLU-Pro rebuilds MMLU with ten answer options instead of four and removes items that no longer separate models. Questions span fourteen academic and professional subjects.","introduction_ja":"","introduction_en":"","category":"Category not supplied","task_count":null,"acquisition_status":"Acquisition status not supplied","official_url":"https://huggingface.co/datasets/TIGER-Lab/MMLU-Pro","indexing_mode":"index","profile":{"resources":[],"task_format":"","scoring":"","metric":"","size":"","answer_access":"","license":"","citation":"","maintainer":"","released":"","why_hard":"","related":[],"history":"","construction":"","limitations":"","saturation":"","scores":[]}},"task_id":"19eb63f5-870f-5bbf-ba6e-b108538384e1","task_key":"default--test--1203","task_revision_id":"1","upstream_id":"1203","short_description":"A widow owned an apartment building. Several years ago, a student, in a signed…","config":"default","split":"test","body":"{\"category\":\"law\",\"options\":\"['the student had a right to stay as he had been residing there for more than three years.', 'the widow had not given the student requisitenotice for terminating a term of years.', 'the student had made improvements to the property, thus extending the leasehold.', 'the student had a verbal agreement with the widow to extend the lease.', 'the terms of the leasehold contract were binding on both parties.', 'the widow had accepted rent for the following month before the three-year period ended.', 'the student was protected from eviction due to his status as a student.', 'the widow had not clearly communicated the end of the lease term.', 'the widow had not given the student requisitenotice for terminating a periodic tenancy.', 'the leasehold contract was void ab initio.']\",\"question\":\"A widow owned an apartment building. Several years ago, a student, in a signed writing, entered a three-year lease agreement with the widow. The student agreed to pay $800 monthly rent for the third-floor apartment in the widow's building. The original three-year term had long since expired, but the student had continued to pay his rent, and the widow continued to accept it. If the widow had chosen to evict the student immediately upon the end of the three-year period, and before she had accepted the next rent check, the student's strongest argument to prevent eviction, if true, would have been that\",\"src\":\"ori_mmlu-professional_law\"}","display_format":"text","language":"","answer_status":"published","assets":[],"source_url":"https://huggingface.co/datasets/TIGER-Lab/MMLU-Pro","history":"initial import","indexing_mode":"index","subproblems":[],"grids":[]}