{"kind":"task","effective_mode":"full","benchmark":{"kind":"benchmark","effective_mode":"full","slug":"mmlu-pro","formal_name":"MMLU-Pro","introduction":"MMLU-Pro rebuilds MMLU with ten answer options instead of four and removes items that no longer separate models. Questions span fourteen academic and professional subjects.","introduction_ja":"","introduction_en":"","category":"Category not supplied","task_count":null,"acquisition_status":"Acquisition status not supplied","official_url":"https://huggingface.co/datasets/TIGER-Lab/MMLU-Pro","indexing_mode":"index","profile":{"resources":[],"task_format":"","scoring":"","metric":"","size":"","answer_access":"","license":"","citation":"","maintainer":"","released":"","why_hard":"","related":[],"history":"","construction":"","limitations":"","saturation":"","scores":[]}},"task_id":"28f3806a-89f0-5cdb-8922-2251637ecd9c","task_key":"default--test--7566","task_revision_id":"1","upstream_id":"7566","short_description":"Ray and Dorothy can both cook and can both pull weeds in the garden on a…","config":"default","split":"test","body":"{\"category\":\"economics\",\"options\":\"['Ray cooks, since he has absolute advantage in pulling weeds.', 'Ray pulls weeds, since he has comparative advantage in pulling weeds.', 'Ray cooks, since he has comparative advantage in cooking.', 'Dorothy pulls weeds, since she has comparative advantage in pulling weeds.', 'Both Ray and Dorothy cook, since they both have absolute advantage in cooking.', 'Both Ray and Dorothy pull weeds, since they both have absolute advantage in pulling weeds.', 'Dorothy cooks, since she has absolute advantage in pulling weeds.', 'Dorothy pulls weeds, since she has absolute advantage in cooking.', 'Ray pulls weeds, since he has absolute advantage in cooking.', 'Dorothy cooks, since she has comparative advantage in cooking.']\",\"question\":\"Ray and Dorothy can both cook and can both pull weeds in the garden on a Saturday afternoon. For every hour of cooking, Ray can pull 50 weeds and Dorothy can pull 100 weeds. Based on this information,\",\"src\":\"ori_mmlu-high_school_microeconomics\"}","display_format":"text","language":"","answer_status":"published","assets":[],"source_url":"https://huggingface.co/datasets/TIGER-Lab/MMLU-Pro","history":"initial import","indexing_mode":"index","subproblems":[],"grids":[]}