{"kind":"task","effective_mode":"full","benchmark":{"kind":"benchmark","effective_mode":"full","slug":"mmlu-pro","formal_name":"MMLU-Pro","introduction":"MMLU-Pro rebuilds MMLU with ten answer options instead of four and removes items that no longer separate models. Questions span fourteen academic and professional subjects.","introduction_ja":"","introduction_en":"","category":"Category not supplied","task_count":null,"acquisition_status":"Acquisition status not supplied","official_url":"https://huggingface.co/datasets/TIGER-Lab/MMLU-Pro","indexing_mode":"index","profile":{"resources":[],"task_format":"","scoring":"","metric":"","size":"","answer_access":"","license":"","citation":"","maintainer":"","released":"","why_hard":"","related":[],"history":"","construction":"","limitations":"","saturation":"","scores":[]}},"task_id":"68d9627b-6d8f-5d67-9b51-976a09269f41","task_key":"default--test--957","task_revision_id":"1","upstream_id":"957","short_description":"A man entered into a written contract with a nationally famous artist. Their…","config":"default","split":"test","body":"{\"category\":\"law\",\"options\":\"[\\\"The man's dissatisfaction is subjective and therefore not a valid reason for non-payment.\\\", 'The man failed to make an adequate examination of the quality of the portrait.', \\\"The man's dissatisfaction with the quality of the portrait was not genuine.\\\", 'The man had unrealistic expectations of the portrait.', 'The man did not provide clear guidelines for the portrait.', \\\"The artist's reputation should guarantee the quality of the portrait.\\\", 'Because of a recent hand injury, the artist did not have sufficient time to produce a portrait of a quality satisfactory to the man.', 'The artist used high-quality materials to create the portrait.', 'The delay in delivery was due to unforeseen circumstances.']\",\"question\":\"A man entered into a written contract with a nationally famous artist. Their contract provided that the artist would paint a full-color portrait of the man, to be delivered by November 1, in exchange for $15,000. A contractual provision recited that the artist \\\"guarantees that the quality of the portrait will be fully satisfactory. \\\"The artist finished the portrait and presented it to the man on November 2. The man looked at the portrait and said, \\\"This is awful! I really expected better. I'm not paying for this!\\\" The artist was enraged by the man's reaction, as he felt the finished portrait was one of his best works. Which of the following arguments would be LEAST helpful to the artist in his efforts to recover on a breach of contract theory?\",\"src\":\"ori_mmlu-professional_law\"}","display_format":"text","language":"","answer_status":"published","assets":[],"source_url":"https://huggingface.co/datasets/TIGER-Lab/MMLU-Pro","history":"initial import","indexing_mode":"index","subproblems":[],"grids":[]}