{"kind":"task","effective_mode":"full","benchmark":{"kind":"benchmark","effective_mode":"full","slug":"mmlu-pro","formal_name":"MMLU-Pro","introduction":"MMLU-Pro rebuilds MMLU with ten answer options instead of four and removes items that no longer separate models. Questions span fourteen academic and professional subjects.","introduction_ja":"","introduction_en":"","category":"Category not supplied","task_count":null,"acquisition_status":"Acquisition status not supplied","official_url":"https://huggingface.co/datasets/TIGER-Lab/MMLU-Pro","indexing_mode":"index","profile":{"resources":[],"task_format":"","scoring":"","metric":"","size":"","answer_access":"","license":"","citation":"","maintainer":"","released":"","why_hard":"","related":[],"history":"","construction":"","limitations":"","saturation":"","scores":[]}},"task_id":"dcc358f6-947c-5eab-aece-5687be80f424","task_key":"default--test--1508","task_revision_id":"1","upstream_id":"1508","short_description":"After a wife found out about her husband's infidelity, she decided to have him…","config":"default","split":"test","body":"{\"category\":\"law\",\"options\":\"['not guilty of either solicitation or conspiracy to commit murder.', 'not guilty due to lack of intent.', 'guilty of attempted murder.', 'guilty of solicitation and conspiracy to commit murder.', 'not guilty due to renouncing her participation.', 'guilty of solicitation.', 'guilty of conspiracy to commit murder and attempted murder.', 'not guilty of conspiracy to commit murder, but guilty of solicitation.', 'guilty of attempted murder and solicitation.', 'guilty of conspiracy to commit murder.']\",\"question\":\"After a wife found out about her husband's infidelity, she decided to have him killed. The defendant approached a hit man to solicit his assistance in carrying out the planned murder. Although the wife believed that he was a hit man, he was, in fact, an undercover agent. The wife told the undercovr agent she would pay him $20,000 if he accepted the job and an additional $20,000 after the killing was completed. Upon hearing the wife's proposal, the undercover agent told the wife he would kill her husband. However, the undercover agent secretly intended not to go through with the plan, and he merely feigned agreement because he wished to trap the wife. The wife told the undercover agent that she would deliver the first payment the next day. However, before making the initial payment, the wife learned that the hit man was really an undercover agent. Fearful that she might be prosecuted for planning her husband's murder, the wife contacted the police and renounced her participation in the criminal endeavor. The defendant will most likely be found\",\"src\":\"ori_mmlu-professional_law\"}","display_format":"text","language":"","answer_status":"published","assets":[],"source_url":"https://huggingface.co/datasets/TIGER-Lab/MMLU-Pro","history":"initial import","indexing_mode":"index","subproblems":[],"grids":[]}