{"kind":"task","effective_mode":"full","benchmark":{"kind":"benchmark","effective_mode":"full","slug":"mmlu-pro","formal_name":"MMLU-Pro","introduction":"MMLU-Pro rebuilds MMLU with ten answer options instead of four and removes items that no longer separate models. Questions span fourteen academic and professional subjects.","introduction_ja":"","introduction_en":"","category":"Category not supplied","task_count":null,"acquisition_status":"Acquisition status not supplied","official_url":"https://huggingface.co/datasets/TIGER-Lab/MMLU-Pro","indexing_mode":"index","profile":{"resources":[],"task_format":"","scoring":"","metric":"","size":"","answer_access":"","license":"","citation":"","maintainer":"","released":"","why_hard":"","related":[],"history":"","construction":"","limitations":"","saturation":"","scores":[]}},"task_id":"6b9f2c32-79bf-520b-a9af-840141078fc8","task_key":"default--test--1453","task_revision_id":"1","upstream_id":"1453","short_description":"A defendant was charged with felony murder as a result of his setting fire to…","config":"default","split":"test","body":"{\"category\":\"law\",\"options\":\"['The defendant was coerced by another to set the fire.', 'The defendant did not intend to kill the victim.', 'The defendant thought the house was empty when he set the fire.', 'The defendant believed the victim had already evacuated the house.', 'The defendant was insane when he set the fire.', 'The victim died of a heart attack before the fire spread to his bedroom.']\",\"question\":\"A defendant was charged with felony murder as a result of his setting fire to the victim's house. The victim was upstairs sleeping when the fire was set, and he died in his bedroom during the fire. If the defendant can prove the facts to support his defense, which of the following assertions would LEAST likely remove liability for felony murder?\",\"src\":\"ori_mmlu-professional_law\"}","display_format":"text","language":"","answer_status":"published","assets":[],"source_url":"https://huggingface.co/datasets/TIGER-Lab/MMLU-Pro","history":"initial import","indexing_mode":"index","subproblems":[],"grids":[]}