{"kind":"task","effective_mode":"full","benchmark":{"kind":"benchmark","effective_mode":"full","slug":"mmlu-pro","formal_name":"MMLU-Pro","introduction":"MMLU-Pro rebuilds MMLU with ten answer options instead of four and removes items that no longer separate models. Questions span fourteen academic and professional subjects.","introduction_ja":"","introduction_en":"","category":"Category not supplied","task_count":null,"acquisition_status":"Acquisition status not supplied","official_url":"https://huggingface.co/datasets/TIGER-Lab/MMLU-Pro","indexing_mode":"index","profile":{"resources":[],"task_format":"","scoring":"","metric":"","size":"","answer_access":"","license":"","citation":"","maintainer":"","released":"","why_hard":"","related":[],"history":"","construction":"","limitations":"","saturation":"","scores":[]}},"task_id":"339b4259-d9c9-513c-9a66-9b2a5129a6d5","task_key":"default--test--2666","task_revision_id":"1","upstream_id":"2666","short_description":"Which of the following correctly categorizes the components of Ivan Pavlov's…","config":"default","split":"test","body":"{\"category\":\"psychology\",\"options\":\"['The dogs (subjects) were presented with the sights and sounds of the presenter (UCR) while also being presented with food (CS). After a time, the dogs salivated in response to the presenter (CR).', 'The dogs (subjects) were presented with food (UCS) while also being presented with the sights and sounds of the presenter (CS). After a time, the dogs salivated in response to the presenter (CR).', 'The dogs (subjects) were presented with food (CR) while also being presented with the sights and sounds of the presenter (UCR). After a time, the dogs salivated in response to the presenter (UCS).', 'The dogs (subjects) were presented with food (UCR) while also being presented with the sights and sounds of the presenter (CR). After a time, the dogs salivated in response to the presenter (CS).', 'The dogs (subjects) were presented with the sights and sounds of the presenter (UCS) while also being presented with food (CR). After a time, the dogs salivated in response to the presenter (CS).', 'The dogs (subjects) were presented with the sights and sounds of the presenter (CR) while also being presented with food (UCR). After a time, the dogs salivated in response to the presenter (UCS).', 'The dogs (subjects) were presented with food (CS) while also being presented with the sights and sounds of the presenter (UCR). After a time, the dogs salivated in response to the food (UCS).', 'The dogs (subjects) were presented with the sights and sounds of the presenter (CS) while also being presented with food (UCS). After a time, the dogs salivated in response to the food (CR).', 'The dogs (subjects) were presented with the sights and sounds of the presenter (UCS) while also being presented with food (CS). After a time, the dogs salivated in response to the food (CR).', 'The dogs (subjects) were presented with food (CS) while also being presented with the sights and sounds of the presenter (UCS). After a time, the dogs salivated in response to the presenter (UCR).']\",\"question\":\"Which of the following correctly categorizes the components of Ivan Pavlov's research on conditioning?\",\"src\":\"ori_mmlu-high_school_psychology\"}","display_format":"text","language":"","answer_status":"published","assets":[],"source_url":"https://huggingface.co/datasets/TIGER-Lab/MMLU-Pro","history":"initial import","indexing_mode":"index","subproblems":[],"grids":[]}