{"kind":"task","effective_mode":"full","benchmark":{"kind":"benchmark","effective_mode":"full","slug":"mmlu-pro","formal_name":"MMLU-Pro","introduction":"MMLU-Pro rebuilds MMLU with ten answer options instead of four and removes items that no longer separate models. Questions span fourteen academic and professional subjects.","introduction_ja":"","introduction_en":"","category":"Category not supplied","task_count":null,"acquisition_status":"Acquisition status not supplied","official_url":"https://huggingface.co/datasets/TIGER-Lab/MMLU-Pro","indexing_mode":"index","profile":{"resources":[],"task_format":"","scoring":"","metric":"","size":"","answer_access":"","license":"","citation":"","maintainer":"","released":"","why_hard":"","related":[],"history":"","construction":"","limitations":"","saturation":"","scores":[]}},"task_id":"d6c8efdc-7413-5ae8-8d3e-2175334cf3c0","task_key":"default--test--7132","task_revision_id":"1","upstream_id":"7132","short_description":"Mr. Jones is president of the First National Bank of St. Louis and wishes to…","config":"default","split":"test","body":"{\"category\":\"economics\",\"options\":\"['$340,000', '$600,000', '$425,000', '25%', '10%', '$510,000', '$1,700,000', '30%', '$255,000', '15%']\",\"question\":\"Mr. Jones is president of the First National Bank of St. Louis and wishes to determine if his bank is holding too much of its demand deposits as reserves. The bank's total deposits = $1,700,000 and the reserve ratio is 20%. If Mr. Jones finds that reserves = $850,000 what might he conclude about excess reserves?\",\"src\":\"stemez-Economics\"}","display_format":"text","language":"","answer_status":"published","assets":[],"source_url":"https://huggingface.co/datasets/TIGER-Lab/MMLU-Pro","history":"initial import","indexing_mode":"index","subproblems":[],"grids":[]}