{"kind":"task","effective_mode":"full","benchmark":{"kind":"benchmark","effective_mode":"full","slug":"mmlu-pro","formal_name":"MMLU-Pro","introduction":"MMLU-Pro rebuilds MMLU with ten answer options instead of four and removes items that no longer separate models. Questions span fourteen academic and professional subjects.","introduction_ja":"","introduction_en":"","category":"Category not supplied","task_count":null,"acquisition_status":"Acquisition status not supplied","official_url":"https://huggingface.co/datasets/TIGER-Lab/MMLU-Pro","indexing_mode":"index","profile":{"resources":[],"task_format":"","scoring":"","metric":"","size":"","answer_access":"","license":"","citation":"","maintainer":"","released":"","why_hard":"","related":[],"history":"","construction":"","limitations":"","saturation":"","scores":[]}},"task_id":"545a6dab-7a6c-533a-8b56-94da406b2229","task_key":"default--test--1234","task_revision_id":"1","upstream_id":"1234","short_description":"After the murder of his brother, a man published the following notice in the…","config":"default","split":"test","body":"{\"category\":\"law\",\"options\":\"['succeed, because his actions led to the arrest and conviction of the murderer.', 'not succeed, because he did not accept the offer before fulfilling the conditions.', 'not succeed, because he did not have knowledge of the reward.', 'not succeed, because the reward offer was not communicated to him directly.', 'succeed, because the reward notice was a unilateral contract.', 'succeed, because he was unaware of the offer.', 'succeed, because his apprehension of the murderer created a contract implied in law.', 'not succeed, because he was not officially hired to investigate the case.', 'not succeed, because his investigation was not a bargained-for exchange.']\",\"question\":\"After the murder of his brother, a man published the following notice in the local newspaper:\\\"REWARDAny person who supplies information leading to the arrest and conviction of the murderer of my brother will be paid $10,000. \\\"An amateur detective, without knowledge of the reward notice, began investigating the matter as a result of his own curiosity kindled by the sensationalism surrounding the brother's murder. One week later, the detective secured information that led to the arrest and later conviction of the murderer. During the murder trial, the detective found out about the reward and demanded the $10,000 from the man. In an action by the detective to recover the $10,000 reward, he will\",\"src\":\"ori_mmlu-professional_law\"}","display_format":"text","language":"","answer_status":"published","assets":[],"source_url":"https://huggingface.co/datasets/TIGER-Lab/MMLU-Pro","history":"initial import","indexing_mode":"index","subproblems":[],"grids":[]}