{"kind":"task","effective_mode":"full","benchmark":{"kind":"benchmark","effective_mode":"full","slug":"mmlu-pro","formal_name":"MMLU-Pro","introduction":"MMLU-Pro rebuilds MMLU with ten answer options instead of four and removes items that no longer separate models. Questions span fourteen academic and professional subjects.","introduction_ja":"","introduction_en":"","category":"Category not supplied","task_count":null,"acquisition_status":"Acquisition status not supplied","official_url":"https://huggingface.co/datasets/TIGER-Lab/MMLU-Pro","indexing_mode":"index","profile":{"resources":[],"task_format":"","scoring":"","metric":"","size":"","answer_access":"","license":"","citation":"","maintainer":"","released":"","why_hard":"","related":[],"history":"","construction":"","limitations":"","saturation":"","scores":[]}},"task_id":"1b52fa99-ee92-5f86-b45e-d7fdba0752b1","task_key":"default--test--1020","task_revision_id":"1","upstream_id":"1020","short_description":"An attorney received a document at his office with an attached note from a…","config":"default","split":"test","body":"{\"category\":\"law\",\"options\":\"['No, because the client did not sign a formal agreement with the attorney.', 'No, because the attorney and the client never agreed on the essential terms of a contract.', 'No, because the attorney did not confirm the fee before providing the service.', \\\"No, because the client's note did not explicitly ask for a billable service.\\\", 'Yes, because the attorney provided a service that is usually billable.', \\\"Yes, because the client was aware that the attorney's services are typically charged.\\\", \\\"Yes, because the attorney took action on the client's note to his detriment.\\\", 'No, because even if the parties had an agreement, that agreement was discharged under the doctrine of frustration of purpose.', \\\"Yes, because the client benefited from the attorney's professional advice.\\\"]\",\"question\":\"An attorney received a document at his office with an attached note from a client for whom he had just finished drafting a will. The note read as follows: \\\"Do you think this contract of sale for my boat complies with state law? I would have talked to you in person about this, but I'm on my way out of town. I will be back next week.\\\" The attorney reviewed the document and wrote a one-page letter to the client stating that the document complied with state law. The lawyer included a bill for $500, which was a lawyer included a bill for $500, which was a reasonable fee. The client refused to pay the attorney anything, arguing that she had never agreed to retain the attorney and that she had received nothing of value from the attorney because the sales transaction was never concluded. Assume that there is no applicable statute or rule governing the formation of attorney-client relationships in the jurisdiction. If the attorney sues the client for the $500, will the attorney be likely to prevail?\",\"src\":\"ori_mmlu-professional_law\"}","display_format":"text","language":"","answer_status":"published","assets":[],"source_url":"https://huggingface.co/datasets/TIGER-Lab/MMLU-Pro","history":"initial import","indexing_mode":"index","subproblems":[],"grids":[]}