{"kind":"task","effective_mode":"full","benchmark":{"kind":"benchmark","effective_mode":"full","slug":"mmlu-pro","formal_name":"MMLU-Pro","introduction":"MMLU-Pro rebuilds MMLU with ten answer options instead of four and removes items that no longer separate models. Questions span fourteen academic and professional subjects.","introduction_ja":"","introduction_en":"","category":"Category not supplied","task_count":null,"acquisition_status":"Acquisition status not supplied","official_url":"https://huggingface.co/datasets/TIGER-Lab/MMLU-Pro","indexing_mode":"index","profile":{"resources":[],"task_format":"","scoring":"","metric":"","size":"","answer_access":"","license":"","citation":"","maintainer":"","released":"","why_hard":"","related":[],"history":"","construction":"","limitations":"","saturation":"","scores":[]}},"task_id":"4b26712d-e236-5f4b-95fa-4e9cde672bae","task_key":"default--test--1790","task_revision_id":"1","upstream_id":"1790","short_description":"A partnership conveyed a general warranty deed representing a vacant parcel of…","config":"default","split":"test","body":"{\"category\":\"law\",\"options\":\"['No, because the couple were not aware of the covenant due to not having a lawyer and not conducting a title search.', 'No, because the general warranty of the original sellers did not apply beyond the first purchaser.', 'Yes, because a landowner is in any event responsible for her share of improvements to any abutting private roads.', 'Yes, this is a covenant running with the land because it was intended to apply to the first purchaser\\\\'s heirs and \\\"assigns\\\".', 'Yes, because the covenant was part of the original agreement and therefore applies to all subsequent owners.', 'Yes, because the couple should have conducted a title search to understand the obligations associated with the property.', 'No, because the sellers did not explicitly communicate the covenant to the couple.', 'Yes, because the couple should have hired an attorney to understand the obligations of the property.', 'No, because the special warranty deed given to the couple did not include any obligations related to the paving of the road.', 'No, because the paving requirement had to be placed in the deed to the married couple and it was not.']\",\"question\":\"A partnership conveyed a general warranty deed representing a vacant parcel of land to the purchaser. In the deed, the grantee and her assigns were required by covenant to share proportionately in the paving of a road that abutted the property. The grantee conveyed the property to a married couple by special warranty deed that did not contain any of the conditions of the covenant regarding paving the new road. The buyers did not have an attorney and did not do a title search. When the road was completed, the married couple buyers refused to pay the share that was demanded of them. The original sellers of the parcel sued the couple in an equity claim without a jury. The judge ruled that the couple had to pay their share of the paving. On appeal, will the appellate court affirm the trial judge's order?\",\"src\":\"ori_mmlu-professional_law\"}","display_format":"text","language":"","answer_status":"published","assets":[],"source_url":"https://huggingface.co/datasets/TIGER-Lab/MMLU-Pro","history":"initial import","indexing_mode":"index","subproblems":[],"grids":[]}