{"kind":"task","effective_mode":"full","benchmark":{"kind":"benchmark","effective_mode":"full","slug":"mmlu-pro","formal_name":"MMLU-Pro","introduction":"MMLU-Pro rebuilds MMLU with ten answer options instead of four and removes items that no longer separate models. Questions span fourteen academic and professional subjects.","introduction_ja":"","introduction_en":"","category":"Category not supplied","task_count":null,"acquisition_status":"Acquisition status not supplied","official_url":"https://huggingface.co/datasets/TIGER-Lab/MMLU-Pro","indexing_mode":"index","profile":{"resources":[],"task_format":"","scoring":"","metric":"","size":"","answer_access":"","license":"","citation":"","maintainer":"","released":"","why_hard":"","related":[],"history":"","construction":"","limitations":"","saturation":"","scores":[]}},"task_id":"1938d85f-7c6d-587e-9ef0-450ab363f9bc","task_key":"default--test--7176","task_revision_id":"1","upstream_id":"7176","short_description":"Suppose the price of beef rises by 10 percent and the quantity of beef demanded…","config":"default","split":"test","body":"{\"category\":\"economics\",\"options\":\"['demand for beef is price elastic and consumer spending on beef is rising.', 'demand for beef is price inelastic and consumer spending on beef is falling.', 'demand for beef is unit elastic and consumer spending on beef is rising.', 'demand for beef is price elastic and consumer spending on beef is falling.', 'demand for beef is perfectly inelastic and consumer spending on beef is rising.', 'demand for beef is perfectly elastic and consumer spending on beef is rising.', 'demand for beef is perfectly elastic and consumer spending on beef is falling.', 'demand for beef is unit elastic and consumer spending on beef is falling.', 'demand for beef is perfectly inelastic and consumer spending on beef is falling.', 'demand for beef is price inelastic and consumer spending on beef is rising.']\",\"question\":\"Suppose the price of beef rises by 10 percent and the quantity of beef demanded falls by 20 percent. We can conclude that\",\"src\":\"ori_mmlu-high_school_microeconomics\"}","display_format":"text","language":"","answer_status":"published","assets":[],"source_url":"https://huggingface.co/datasets/TIGER-Lab/MMLU-Pro","history":"initial import","indexing_mode":"index","subproblems":[],"grids":[]}