{"kind":"task","effective_mode":"full","benchmark":{"kind":"benchmark","effective_mode":"full","slug":"mmlu-pro","formal_name":"MMLU-Pro","introduction":"MMLU-Pro rebuilds MMLU with ten answer options instead of four and removes items that no longer separate models. Questions span fourteen academic and professional subjects.","introduction_ja":"","introduction_en":"","category":"Category not supplied","task_count":null,"acquisition_status":"Acquisition status not supplied","official_url":"https://huggingface.co/datasets/TIGER-Lab/MMLU-Pro","indexing_mode":"index","profile":{"resources":[],"task_format":"","scoring":"","metric":"","size":"","answer_access":"","license":"","citation":"","maintainer":"","released":"","why_hard":"","related":[],"history":"","construction":"","limitations":"","saturation":"","scores":[]}},"task_id":"b0a2bedc-5bb2-552b-a250-ea9c3d610ddb","task_key":"default--test--128","task_revision_id":"3","upstream_id":"128","short_description":"The extent to which a service envelops a product varies according to a number of…","config":"default","split":"test","body":"{\"category\":\"business\",\"options\":\"['The CEO of the company producing the product', 'The number of competitors in the market', 'Variations in supply and demand.', 'The time of year the product is sold', 'The retail location where the product is sold', 'The level of tangibility associated with the type of product.', 'The age of the product designer', 'The color of the product packaging', 'The way in which the service is delivered.', 'Performance-value.']\",\"question\":\"The extent to which a service envelops a product varies according to a number of factors. Which of the following is NOT one of the factors?\",\"src\":\"ori_mmlu-marketing\"}","display_format":"text","language":"","answer_status":"published","assets":[],"source_url":"https://huggingface.co/datasets/TIGER-Lab/MMLU-Pro","history":"initial import","indexing_mode":"index","subproblems":[],"grids":[]}