{"kind":"task","effective_mode":"full","benchmark":{"kind":"benchmark","effective_mode":"full","slug":"mmlu-pro","formal_name":"MMLU-Pro","introduction":"MMLU-Pro rebuilds MMLU with ten answer options instead of four and removes items that no longer separate models. Questions span fourteen academic and professional subjects.","introduction_ja":"","introduction_en":"","category":"Category not supplied","task_count":null,"acquisition_status":"Acquisition status not supplied","official_url":"https://huggingface.co/datasets/TIGER-Lab/MMLU-Pro","indexing_mode":"index","profile":{"resources":[],"task_format":"","scoring":"","metric":"","size":"","answer_access":"","license":"","citation":"","maintainer":"","released":"","why_hard":"","related":[],"history":"","construction":"","limitations":"","saturation":"","scores":[]}},"task_id":"59b0f3f5-bc11-58a3-92b5-b4864a841559","task_key":"default--test--923","task_revision_id":"1","upstream_id":"923","short_description":"Alice owned Greyacre, a one-acre piece of property, in fee simple. Alice sold…","config":"default","split":"test","body":"{\"category\":\"law\",\"options\":\"['Alice, since her ownership was first in time and she was of record.', 'Barbara, because she was the first to record her deed.', 'Alice, since she made two valid sales of the same property.', 'Charlene, because she was the first to record her deed.', 'Barbara, due to her physical possession of the property.', 'Barbara, because Charlene had notice of the prior conveyance between Alice and Barbara.', \\\"Charlene, because she relied on Alice's representation of ownership.\\\", 'Alice, as she never actually transferred the property rights.', 'Charlene, as Barbara did not officially record her deed.', 'Charlene, because she was the last to receive the deed from Alice.']\",\"question\":\"Alice owned Greyacre, a one-acre piece of property, in fee simple. Alice sold the property to Barbara who moved into the only residence on the land. Barbara neglected to record the deed, and three years later, Alice conveyed the same property to Charlene who immediately filed the deed she received from Alice. Greyacre's jurisdiction has a race-notice statute and a tract index system. If an action is filed for title to the property, the likely outcome would be for\",\"src\":\"ori_mmlu-professional_law\"}","display_format":"text","language":"","answer_status":"published","assets":[],"source_url":"https://huggingface.co/datasets/TIGER-Lab/MMLU-Pro","history":"initial import","indexing_mode":"index","subproblems":[],"grids":[]}