{"kind":"task","effective_mode":"full","benchmark":{"kind":"benchmark","effective_mode":"full","slug":"mmlu-pro","formal_name":"MMLU-Pro","introduction":"MMLU-Pro rebuilds MMLU with ten answer options instead of four and removes items that no longer separate models. Questions span fourteen academic and professional subjects.","introduction_ja":"","introduction_en":"","category":"Category not supplied","task_count":null,"acquisition_status":"Acquisition status not supplied","official_url":"https://huggingface.co/datasets/TIGER-Lab/MMLU-Pro","indexing_mode":"index","profile":{"resources":[],"task_format":"","scoring":"","metric":"","size":"","answer_access":"","license":"","citation":"","maintainer":"","released":"","why_hard":"","related":[],"history":"","construction":"","limitations":"","saturation":"","scores":[]}},"task_id":"b7c7862b-90d7-580c-8e1f-bc838f0a9649","task_key":"default--test--9630","task_revision_id":"1","upstream_id":"9630","short_description":"A toy car and a toy truck collide. If the toy truck’s mass is double the toy…","config":"default","split":"test","body":"{\"category\":\"physics\",\"options\":\"['four times the magnitude and in the same direction', 'half the magnitude and in the opposite direction', 'double the magnitude and in the same direction', 'triple the magnitude and in the opposite direction', 'equal in magnitude and in the opposite direction', 'double the magnitude and in the opposite direction', 'four times the magnitude and in the opposite direction', 'equal in magnitude and in the same direction', 'half the magnitude and in the same direction', 'triple the magnitude and in the same direction']\",\"question\":\"A toy car and a toy truck collide. If the toy truck’s mass is double the toy car’s mass, then, compared to the acceleration of the truck, the acceleration of the car during the collision will be\",\"src\":\"ori_mmlu-high_school_physics\"}","display_format":"text","language":"","answer_status":"published","assets":[],"source_url":"https://huggingface.co/datasets/TIGER-Lab/MMLU-Pro","history":"initial import","indexing_mode":"index","subproblems":[],"grids":[]}