{"kind":"task","effective_mode":"full","benchmark":{"kind":"benchmark","effective_mode":"full","slug":"mmlu-pro","formal_name":"MMLU-Pro","introduction":"MMLU-Pro rebuilds MMLU with ten answer options instead of four and removes items that no longer separate models. Questions span fourteen academic and professional subjects.","introduction_ja":"","introduction_en":"","category":"Category not supplied","task_count":null,"acquisition_status":"Acquisition status not supplied","official_url":"https://huggingface.co/datasets/TIGER-Lab/MMLU-Pro","indexing_mode":"index","profile":{"resources":[],"task_format":"","scoring":"","metric":"","size":"","answer_access":"","license":"","citation":"","maintainer":"","released":"","why_hard":"","related":[],"history":"","construction":"","limitations":"","saturation":"","scores":[]}},"task_id":"33a0a48b-e88e-5ad8-a9ee-affe59d0106e","task_key":"default--test--5947","task_revision_id":"1","upstream_id":"5947","short_description":"Warner and Myrdal claimed that black former slaves were not joining the 'ethnic…","config":"default","split":"test","body":"{\"category\":\"other\",\"options\":\"['the North American cities did not offer any advantages over the Southern states', 'the white population did not believe in liberty, equality, and democracy', 'they were not physically capable of moving to the North American cities', \\\"they were not allowed to join the 'ethnic melting pot' by the government\\\", 'they wanted to retain a strong sense of their original ethnic identity', 'they were not prepared to leave the Southern states and move to North America', 'they were not economically capable of moving to the North American cities', 'the promise of citizenship was contradicted by continued discrimination', \\\"they were afraid of losing their cultural heritage in the 'ethnic melting pot'\\\"]\",\"question\":\"Warner and Myrdal claimed that black former slaves were not joining the 'ethnic melting pot' of the North American cities because:\",\"src\":\"ori_mmlu-sociology\"}","display_format":"text","language":"","answer_status":"published","assets":[],"source_url":"https://huggingface.co/datasets/TIGER-Lab/MMLU-Pro","history":"initial import","indexing_mode":"index","subproblems":[],"grids":[]}