{"kind":"task","effective_mode":"full","benchmark":{"kind":"benchmark","effective_mode":"full","slug":"mmlu-pro","formal_name":"MMLU-Pro","introduction":"MMLU-Pro rebuilds MMLU with ten answer options instead of four and removes items that no longer separate models. Questions span fourteen academic and professional subjects.","introduction_ja":"","introduction_en":"","category":"Category not supplied","task_count":null,"acquisition_status":"Acquisition status not supplied","official_url":"https://huggingface.co/datasets/TIGER-Lab/MMLU-Pro","indexing_mode":"index","profile":{"resources":[],"task_format":"","scoring":"","metric":"","size":"","answer_access":"","license":"","citation":"","maintainer":"","released":"","why_hard":"","related":[],"history":"","construction":"","limitations":"","saturation":"","scores":[]}},"task_id":"db97ed8f-a692-5b23-9c2c-ca82547fdd0b","task_key":"default--test--1718","task_revision_id":"1","upstream_id":"1718","short_description":"A builder sold a new house to a buyer for use as the buyer's residence. The…","config":"default","split":"test","body":"{\"category\":\"law\",\"options\":\"['No, because the builder cannot accept the deed while also holding the mortgage.', 'No, because the owner of a personal residence cannot waive the right to foreclosure.', 'Yes, because the transaction was reasonable and fair under the circumstances.', 'No, because the builder coerced the buyer into deeding the house back.', 'Yes, because the buyer defaulted on the mortgage payments.', 'Yes, because the builder originally held the mortgage.', 'Yes, because of the doctrine of equitable redemption.', 'No, because the buyer still technically owns the house until foreclosure.', 'No, because the deed back to the builder constitutes a disguised mortgage.']\",\"question\":\"A builder sold a new house to a buyer for use as the buyer's residence. The buyer paid 10% of the purchase price and financed the rest by executing a promissory note and purchase money mortgage to the builder. A year later, the buyer missed several mortgage payments to the builder and became unable to make payments to the builder and became unable to make payments. During that year, property values in the neighborhood declined substantially. The builder suggested that the buyer deed the house back to the builder to settle all claims and avoid the costs and other disadvantages of foreclosure. The buyer deeded the house back to the builder. Does the builder now own fee simple title to the house?\",\"src\":\"ori_mmlu-professional_law\"}","display_format":"text","language":"","answer_status":"published","assets":[],"source_url":"https://huggingface.co/datasets/TIGER-Lab/MMLU-Pro","history":"initial import","indexing_mode":"index","subproblems":[],"grids":[]}