{"kind":"task","effective_mode":"full","benchmark":{"kind":"benchmark","effective_mode":"full","slug":"mmlu-pro","formal_name":"MMLU-Pro","introduction":"MMLU-Pro rebuilds MMLU with ten answer options instead of four and removes items that no longer separate models. Questions span fourteen academic and professional subjects.","introduction_ja":"","introduction_en":"","category":"Category not supplied","task_count":null,"acquisition_status":"Acquisition status not supplied","official_url":"https://huggingface.co/datasets/TIGER-Lab/MMLU-Pro","indexing_mode":"index","profile":{"resources":[],"task_format":"","scoring":"","metric":"","size":"","answer_access":"","license":"","citation":"","maintainer":"","released":"","why_hard":"","related":[],"history":"","construction":"","limitations":"","saturation":"","scores":[]}},"task_id":"2aebc172-118b-54b5-ac1b-9e2e5e67e2db","task_key":"default--test--1807","task_revision_id":"1","upstream_id":"1807","short_description":"A man told his neighbor, \"If you will mow my lawn, I will pay you $50. \" The…","config":"default","split":"test","body":"{\"category\":\"law\",\"options\":\"[\\\"Yes, because there was an offer for a unilateral contract that became irrevocable prior to the man's attempted revocation.\\\", \\\"No, because the neighbor had done nothing more than prepare to accept the offer prior to the man's revocation.\\\", 'Yes, because the neighbor can sue for breach of oral contract.', 'Yes, because the neighbor had begun the task before the man revoked his offer.', 'Yes, under the doctrine of quasi-contract.', 'No, because there was no written contract and hence no legal obligation for the man to pay.', 'No, because the neighbor did not get a written agreement from the man.', \\\"No, because the man's revocation was effective, since the neighbor had not completed performance.\\\", 'No, because the man revoked his offer before the task was completed.', 'Yes, because the neighbor had already incurred expenses in preparation to fulfill the contract.']\",\"question\":\"A man told his neighbor, \\\"If you will mow my lawn, I will pay you $50. \\\" The neighbor then purchased gasoline for the mower so he could mow the lawn. The neighbor wheeled his lawn mower to the man's property and started the engine. At that moment, the man suddenly yelled, \\\"I hereby revoke my offer. \\\"If the neighbor thereafter mows the man's lawn, will he recover the $50?\",\"src\":\"ori_mmlu-professional_law\"}","display_format":"text","language":"","answer_status":"published","assets":[],"source_url":"https://huggingface.co/datasets/TIGER-Lab/MMLU-Pro","history":"initial import","indexing_mode":"index","subproblems":[],"grids":[]}