{"kind":"task","effective_mode":"full","benchmark":{"kind":"benchmark","effective_mode":"full","slug":"mmlu-pro","formal_name":"MMLU-Pro","introduction":"MMLU-Pro rebuilds MMLU with ten answer options instead of four and removes items that no longer separate models. Questions span fourteen academic and professional subjects.","introduction_ja":"","introduction_en":"","category":"Category not supplied","task_count":null,"acquisition_status":"Acquisition status not supplied","official_url":"https://huggingface.co/datasets/TIGER-Lab/MMLU-Pro","indexing_mode":"index","profile":{"resources":[],"task_format":"","scoring":"","metric":"","size":"","answer_access":"","license":"","citation":"","maintainer":"","released":"","why_hard":"","related":[],"history":"","construction":"","limitations":"","saturation":"","scores":[]}},"task_id":"4d80c970-ebd7-5e04-a15f-295f663c9747","task_key":"default--test--741","task_revision_id":"1","upstream_id":"741","short_description":"PaulineDiNardihas a choice between buying a stereo set in New Jersey for $698.50…","config":"default","split":"test","body":"{\"category\":\"business\",\"options\":\"['The New Jersey stereo, after accounting for travel expenses to New York that would increase the overall cost', 'The New York stereo, after negotiating a discount equal to the taxes', 'The New York stereo', 'The New Jersey stereo', 'Cannot be determined from the information given', 'Both are the same price', 'The New York stereo, if she uses a specific payment method that offers a cashback exceeding the taxes', 'The New Jersey stereo, but with a higher quality model that makes it a better value', 'The New York stereo, with an added warranty that justifies the extra cost', \\\"The New Jersey stereo, including an additional manufacturer's rebate not available in New York\\\"]\",\"question\":\"PaulineDiNardihas a choice between buying a stereo set in New Jersey for $698.50 and buying it in New York for $684.75 plus a 4% state tax and a 2% city tax. Which stereo should she buy?\",\"src\":\"stemez-Business\"}","display_format":"text","language":"","answer_status":"published","assets":[],"source_url":"https://huggingface.co/datasets/TIGER-Lab/MMLU-Pro","history":"initial import","indexing_mode":"index","subproblems":[],"grids":[]}