{"kind":"task","effective_mode":"full","benchmark":{"kind":"benchmark","effective_mode":"full","slug":"mmlu-pro","formal_name":"MMLU-Pro","introduction":"MMLU-Pro rebuilds MMLU with ten answer options instead of four and removes items that no longer separate models. Questions span fourteen academic and professional subjects.","introduction_ja":"","introduction_en":"","category":"Category not supplied","task_count":null,"acquisition_status":"Acquisition status not supplied","official_url":"https://huggingface.co/datasets/TIGER-Lab/MMLU-Pro","indexing_mode":"index","profile":{"resources":[],"task_format":"","scoring":"","metric":"","size":"","answer_access":"","license":"","citation":"","maintainer":"","released":"","why_hard":"","related":[],"history":"","construction":"","limitations":"","saturation":"","scores":[]}},"task_id":"1bf84e4b-53aa-5b0f-bfd9-badd28dfbc3f","task_key":"default--test--1162","task_revision_id":"1","upstream_id":"1162","short_description":"Mary, a wealthy St. Petersburg widow, executed her first and only will on May…","config":"default","split":"test","body":"{\"category\":\"law\",\"options\":\"['50% to Joan and the income from the remaining 50% to Joan for life, remainder to the Salvation Army, if Joan files a timely petition protesting the devise to the Salvation Army.', 'The estate is divided equally between Joan and the Salvation Army, but Joan has the option to reject her share.', '50% to Joan and the income from the remaining 50% to the Salvation Army for life, remainder to Joan.', '100% to Joan if she files a timely petition requesting that the devise to the Salvation Army be avoided.', 'The estate is held in trust with Joan as the beneficiary during her lifetime, then transferred to the Salvation Army upon her death.', '50% to Joan and 50% to the Salvation Army.', '100% to Joan.', '100% to the Salvation Army.', '75% to Joan and 25% to the Salvation Army.', '25% to Joan and 75% to the Salvation Army.']\",\"question\":\"Mary, a wealthy St. Petersburg widow, executed her first and only will on May 15, 1990 and died on August 18, 1990. Her will provided that her estate be divided equally between her only child, Joan, and the Salvation Army of Largo. How will Mary's estate actually be distributed?\",\"src\":\"ori_mmlu-professional_law\"}","display_format":"text","language":"","answer_status":"published","assets":[],"source_url":"https://huggingface.co/datasets/TIGER-Lab/MMLU-Pro","history":"initial import","indexing_mode":"index","subproblems":[],"grids":[]}