{"kind":"task","effective_mode":"full","benchmark":{"kind":"benchmark","effective_mode":"full","slug":"mmlu-pro","formal_name":"MMLU-Pro","introduction":"MMLU-Pro rebuilds MMLU with ten answer options instead of four and removes items that no longer separate models. Questions span fourteen academic and professional subjects.","introduction_ja":"","introduction_en":"","category":"Category not supplied","task_count":null,"acquisition_status":"Acquisition status not supplied","official_url":"https://huggingface.co/datasets/TIGER-Lab/MMLU-Pro","indexing_mode":"index","profile":{"resources":[],"task_format":"","scoring":"","metric":"","size":"","answer_access":"","license":"","citation":"","maintainer":"","released":"","why_hard":"","related":[],"history":"","construction":"","limitations":"","saturation":"","scores":[]}},"task_id":"e5bae81c-be64-5340-b924-e2191bd77fe3","task_key":"default--test--1237","task_revision_id":"1","upstream_id":"1237","short_description":"Two years ago, a developer conveyed title to a lot to a woman by warranty deed.…","config":"default","split":"test","body":"{\"category\":\"law\",\"options\":\"['No, because the friend did not conduct a title search before purchasing the lot.', 'No, because the friend did not purchase her own title insurance.', 'No, because the title insurance policy expired when the woman conveyed the lot to the friend.', 'Yes, because the tax lien was not listed as an exception on the title insurance policy.', 'No, because the woman no longer owns the property, and thus her title insurance is no longer valid.', 'Yes, because the woman used a quitclaim deed, which means all encumbrances should transfer to the new owner.', 'No, because the friend is not a named insured on the title insurance policy.', 'Yes, because the friend paid cash, making her eligible for claims.', \\\"Yes, because the woman is liable to the friend for breach of the covenant against encumbrances, and the title policy should cover the friend's potential loss.\\\", 'Yes, because the title insurance company made an error in not listing the tax lien.']\",\"question\":\"Two years ago, a developer conveyed title to a lot to a woman by warranty deed. The woman purchased an owner's policy of title insurance before the an owner's policy of title insurance before the closing. Because of an error by the title insurance company, the title commitment and title policy failed to list a county tax lien encumbering the lot as an exception to coverage. Last year, the woman conveyed the lot for a bargain price by a quitclaim deed to a friend, who paid cash, did not conduct a title search, and did not purchase title insurance. Subsequently, the county began proceedings to foreclose the tax lien, and the friend filed a claim with the woman's title insurance company, demanding that the company reimburse her for the tax lien amount. Is the title insurance company obligated to pay the friend's claim?\",\"src\":\"ori_mmlu-professional_law\"}","display_format":"text","language":"","answer_status":"published","assets":[],"source_url":"https://huggingface.co/datasets/TIGER-Lab/MMLU-Pro","history":"initial import","indexing_mode":"index","subproblems":[],"grids":[]}