{"kind":"task","effective_mode":"full","benchmark":{"kind":"benchmark","effective_mode":"full","slug":"mmlu-pro","formal_name":"MMLU-Pro","introduction":"MMLU-Pro rebuilds MMLU with ten answer options instead of four and removes items that no longer separate models. Questions span fourteen academic and professional subjects.","introduction_ja":"","introduction_en":"","category":"Category not supplied","task_count":null,"acquisition_status":"Acquisition status not supplied","official_url":"https://huggingface.co/datasets/TIGER-Lab/MMLU-Pro","indexing_mode":"index","profile":{"resources":[],"task_format":"","scoring":"","metric":"","size":"","answer_access":"","license":"","citation":"","maintainer":"","released":"","why_hard":"","related":[],"history":"","construction":"","limitations":"","saturation":"","scores":[]}},"task_id":"9e84ffe1-64e3-5ff4-95be-c08cc8dbaaad","task_key":"default--test--11123","task_revision_id":"1","upstream_id":"11123","short_description":"Use the following key to translate the given formula of PL to natural, English…","config":"default","split":"test","body":"{\"category\":\"philosophy\",\"options\":\"[\\\"Emily doesn't stop working unless Russell makes dinner.\\\", \\\"If Emily doesn't stop working then Russell makes dinner.\\\", \\\"Emily stops working if Russell doesn't make dinner.\\\", \\\"Russell doesn't make dinner if Emily stops working.\\\", \\\"If Emily stops working then Russell doesn't make dinner.\\\", \\\"If Russell makes dinner then Emily doesn't stop working.\\\", \\\"Russell doesn't make dinner unless Emily stops working.\\\", \\\"Emily stops working unless Russell doesn't make dinner.\\\", \\\"Russell makes dinner if Emily doesn't stop working.\\\"]\",\"question\":\" Use the following key to translate the given formula of PL to natural, English sentences.\\nA: Marina reads a Percy Jackson book.\\nB: Izzy plays Minecraft.\\nC: Emily stops working.\\nD: Russell makes dinner.\\nE: Ashleigh stops by.\\n~C ∨ D\",\"src\":\"ori_mmlu-formal_logic\"}","display_format":"text","language":"","answer_status":"published","assets":[],"source_url":"https://huggingface.co/datasets/TIGER-Lab/MMLU-Pro","history":"initial import","indexing_mode":"index","subproblems":[],"grids":[]}