{"kind":"task","effective_mode":"full","benchmark":{"kind":"benchmark","effective_mode":"full","slug":"mmlu-pro","formal_name":"MMLU-Pro","introduction":"MMLU-Pro rebuilds MMLU with ten answer options instead of four and removes items that no longer separate models. Questions span fourteen academic and professional subjects.","introduction_ja":"","introduction_en":"","category":"Category not supplied","task_count":null,"acquisition_status":"Acquisition status not supplied","official_url":"https://huggingface.co/datasets/TIGER-Lab/MMLU-Pro","indexing_mode":"index","profile":{"resources":[],"task_format":"","scoring":"","metric":"","size":"","answer_access":"","license":"","citation":"","maintainer":"","released":"","why_hard":"","related":[],"history":"","construction":"","limitations":"","saturation":"","scores":[]}},"task_id":"aecc05fe-b203-5a7f-9202-e07c21abc1a4","task_key":"default--test--905","task_revision_id":"1","upstream_id":"905","short_description":"A buyer contracted in writing to purchase 1,000 gidgets (which are similar to…","config":"default","split":"test","body":"{\"category\":\"law\",\"options\":\"['it violates the statute of frauds.', 'the buyer did not provide sufficient notice.', 'it was a verbal agreement, not a written one.', 'the parol evidence rule bars contradictory terms.', 'the manufacturer did not agree to the modification.', 'the original contract was in writing.', 'the price was not adjusted to reflect the change in quantity.', 'there was no consideration.', 'the telephone conversation was not recorded.', 'there was no written agreement to the modification.']\",\"question\":\"A buyer contracted in writing to purchase 1,000 gidgets (which are similar to widgets but are more popular because of their greater durability and- resiliency) from a manufacturer for a total contract price of $4,000. After this agreement was executed and before delivery, the buyer and the manufacturer agreed in a telephone conversation to change the quantity term from 1,000 to 800 gidgets. This modification is probably unenforceable, because\",\"src\":\"ori_mmlu-professional_law\"}","display_format":"text","language":"","answer_status":"published","assets":[],"source_url":"https://huggingface.co/datasets/TIGER-Lab/MMLU-Pro","history":"initial import","indexing_mode":"index","subproblems":[],"grids":[]}