{"kind":"task","effective_mode":"full","benchmark":{"kind":"benchmark","effective_mode":"full","slug":"mmlu-pro","formal_name":"MMLU-Pro","introduction":"MMLU-Pro rebuilds MMLU with ten answer options instead of four and removes items that no longer separate models. Questions span fourteen academic and professional subjects.","introduction_ja":"","introduction_en":"","category":"Category not supplied","task_count":null,"acquisition_status":"Acquisition status not supplied","official_url":"https://huggingface.co/datasets/TIGER-Lab/MMLU-Pro","indexing_mode":"index","profile":{"resources":[],"task_format":"","scoring":"","metric":"","size":"","answer_access":"","license":"","citation":"","maintainer":"","released":"","why_hard":"","related":[],"history":"","construction":"","limitations":"","saturation":"","scores":[]}},"task_id":"3ef97631-2427-567d-a42f-786dd2a8d5dd","task_key":"default--test--1894","task_revision_id":"1","upstream_id":"1894","short_description":"A cashier worked part-time at a convenience store. As the cashier was sitting…","config":"default","split":"test","body":"{\"category\":\"law\",\"options\":\"['False pretenses.', 'No crime.', 'Robbery.', 'Theft by deception.', 'Embezzlement.', 'Fraud.', 'Larceny.', 'Petit theft.', 'Larceny by trick.']\",\"question\":\"A cashier worked part-time at a convenience store. As the cashier was sitting behind the cash register, he noticed a boy enter the store. While the boy walked through the store, the cashier watched him on a hidden camera located behind the cashier's counter. Thereupon, the cashier saw the boy pick up two packs of baseball cards and place them in his coat pocket. The boy then walked to the rear of the store, where he picked up a package of cupcakes. The boy took the cupcakes to the front counter and paid the cashier for the cupcakes only. The boy started to walk out of the store when the cashier said, \\\"Hey, kid, I'm not charging you for the baseball cards because you paid for the cupcakes. But next time I'm not going to let you get away with it. \\\" The baseball cards sold for $5 per pack. The boy has committed which, if any, crimes?\",\"src\":\"ori_mmlu-professional_law\"}","display_format":"text","language":"","answer_status":"published","assets":[],"source_url":"https://huggingface.co/datasets/TIGER-Lab/MMLU-Pro","history":"initial import","indexing_mode":"index","subproblems":[],"grids":[]}