{"kind":"task","effective_mode":"full","benchmark":{"kind":"benchmark","effective_mode":"full","slug":"mmlu-pro","formal_name":"MMLU-Pro","introduction":"MMLU-Pro rebuilds MMLU with ten answer options instead of four and removes items that no longer separate models. Questions span fourteen academic and professional subjects.","introduction_ja":"","introduction_en":"","category":"Category not supplied","task_count":null,"acquisition_status":"Acquisition status not supplied","official_url":"https://huggingface.co/datasets/TIGER-Lab/MMLU-Pro","indexing_mode":"index","profile":{"resources":[],"task_format":"","scoring":"","metric":"","size":"","answer_access":"","license":"","citation":"","maintainer":"","released":"","why_hard":"","related":[],"history":"","construction":"","limitations":"","saturation":"","scores":[]}},"task_id":"d581cc4d-bdaa-5610-88d5-d2042aef1053","task_key":"default--test--7635","task_revision_id":"1","upstream_id":"7635","short_description":"In the last 20 years firms that produce cameras have begun to produce fewer…","config":"default","split":"test","body":"{\"category\":\"economics\",\"options\":\"['the market system answering the question of \\\"what\\\" cameras should be produced.', 'the market system answering the question of \\\"when\\\" cameras should be produced.', 'the market system answering the question of \\\"why\\\" cameras should be produced.', 'the influence of government regulation on camera production.', 'the market system answering the question of \\\"where\\\" cameras should be produced.', 'how central planners dictate which cameras are produced.', 'the market system answering the question of \\\"who\\\" should consume the cameras that are produced.', 'the market system answering the question of \\\"how\\\" cameras should be produced.']\",\"question\":\"In the last 20 years firms that produce cameras have begun to produce fewer 35-mm cameras and more digital cameras. This trend is an example of\",\"src\":\"ori_mmlu-high_school_macroeconomics\"}","display_format":"text","language":"","answer_status":"published","assets":[],"source_url":"https://huggingface.co/datasets/TIGER-Lab/MMLU-Pro","history":"initial import","indexing_mode":"index","subproblems":[],"grids":[]}