{"kind":"task","effective_mode":"full","benchmark":{"kind":"benchmark","effective_mode":"full","slug":"mmlu-pro","formal_name":"MMLU-Pro","introduction":"MMLU-Pro rebuilds MMLU with ten answer options instead of four and removes items that no longer separate models. Questions span fourteen academic and professional subjects.","introduction_ja":"","introduction_en":"","category":"Category not supplied","task_count":null,"acquisition_status":"Acquisition status not supplied","official_url":"https://huggingface.co/datasets/TIGER-Lab/MMLU-Pro","indexing_mode":"index","profile":{"resources":[],"task_format":"","scoring":"","metric":"","size":"","answer_access":"","license":"","citation":"","maintainer":"","released":"","why_hard":"","related":[],"history":"","construction":"","limitations":"","saturation":"","scores":[]}},"task_id":"11ebce7d-9afd-5c8b-aac0-4bc274e9b8e9","task_key":"default--test--1741","task_revision_id":"1","upstream_id":"1741","short_description":"A man purchased a house that needed substantial repairs. The man financed the…","config":"default","split":"test","body":"{\"category\":\"law\",\"options\":\"['The credit union, because the man is personally liable on the debt.', 'The bank, because it initiated the foreclosure proceedings.', 'The man, because the outside party received title to the house subject to the second mortgage.', 'The man, because he was the original owner of the house.', 'The credit union, because the credit union has priority.', 'The man, because of his equitable right of redemption.', 'The outside party, because they purchased the house at the foreclosure sale.', 'The man, because he has made significant improvements to the property.', 'The credit union, because it provided the funds for the necessary repairs.', 'The credit union, because the man defaulted on the loan to the bank.']\",\"question\":\"A man purchased a house that needed substantial repairs. The man financed the purchase of the house by borrowing funds from a bank. He signed a note to the bank and secured repayment of the loan with a mortgage. After purchasing the house, the man borrowed money from his credit union to make the repairs. The man signed a note to the credit union; this note was also secured by a mortgage on the house. Both mortgages were promptly recorded in the order granted. The man was personally liable on both loans. The man moved into the house and made the necessary repairs. He later defaulted on the debt to the bank, and the bank initiated judicial foreclosure proceedings, naming both the man and the credit union as parties to the action. An outside party acquired the house at the foreclosure sale. After the expenses of the sale and the balance due the bank have been paid, $5,000 remains in the sale the bank have been paid, $5,000 remains in the sale proceeds. The outstanding balance of the credit union loan is $20,000. The man and the credit union both claim the $5,000. There is no applicable statute. Who is entitled to the $5,000?\",\"src\":\"ori_mmlu-professional_law\"}","display_format":"text","language":"","answer_status":"published","assets":[],"source_url":"https://huggingface.co/datasets/TIGER-Lab/MMLU-Pro","history":"initial import","indexing_mode":"index","subproblems":[],"grids":[]}