{"kind":"task","effective_mode":"full","benchmark":{"kind":"benchmark","effective_mode":"full","slug":"mmlu-pro","formal_name":"MMLU-Pro","introduction":"MMLU-Pro rebuilds MMLU with ten answer options instead of four and removes items that no longer separate models. Questions span fourteen academic and professional subjects.","introduction_ja":"","introduction_en":"","category":"Category not supplied","task_count":null,"acquisition_status":"Acquisition status not supplied","official_url":"https://huggingface.co/datasets/TIGER-Lab/MMLU-Pro","indexing_mode":"index","profile":{"resources":[],"task_format":"","scoring":"","metric":"","size":"","answer_access":"","license":"","citation":"","maintainer":"","released":"","why_hard":"","related":[],"history":"","construction":"","limitations":"","saturation":"","scores":[]}},"task_id":"b7acd599-cf01-50e6-93d7-e9e1d07fb53d","task_key":"default--test--1030","task_revision_id":"1","upstream_id":"1030","short_description":"A passenger on a subway car placed her pocketbook on the seat next to her and…","config":"default","split":"test","body":"{\"category\":\"law\",\"options\":\"['assault.', 'burglary.', 'vandalism.', 'larceny.', 'robbery.', 'aggravated assault.', 'attempted larceny.', 'trespassing.', 'possession of stolen property.']\",\"question\":\"A passenger on a subway car placed her pocketbook on the seat next to her and began to zip up her jacket. Defendant was standing in front of the passenger and holding on to the hand rail. Seeing the pocketbook unattended, he suddenly grabbed it and tried to run away. As he did so, the passenger became very frightened and fell backward. She struck her head against the window and was knocked unconscious. In the commotion, Defendant dropped the pocketbook and hurriedly exited the subway car empty-handed. Defendant should be found guilty of\",\"src\":\"ori_mmlu-professional_law\"}","display_format":"text","language":"","answer_status":"published","assets":[],"source_url":"https://huggingface.co/datasets/TIGER-Lab/MMLU-Pro","history":"initial import","indexing_mode":"index","subproblems":[],"grids":[]}