{"kind":"task","effective_mode":"full","benchmark":{"kind":"benchmark","effective_mode":"full","slug":"mmlu-pro","formal_name":"MMLU-Pro","introduction":"MMLU-Pro rebuilds MMLU with ten answer options instead of four and removes items that no longer separate models. Questions span fourteen academic and professional subjects.","introduction_ja":"","introduction_en":"","category":"Category not supplied","task_count":null,"acquisition_status":"Acquisition status not supplied","official_url":"https://huggingface.co/datasets/TIGER-Lab/MMLU-Pro","indexing_mode":"index","profile":{"resources":[],"task_format":"","scoring":"","metric":"","size":"","answer_access":"","license":"","citation":"","maintainer":"","released":"","why_hard":"","related":[],"history":"","construction":"","limitations":"","saturation":"","scores":[]}},"task_id":"4a84e0bf-1c20-5e51-ac51-1edd77705f88","task_key":"default--test--11257","task_revision_id":"1","upstream_id":"11257","short_description":"Use the following key to translate the given formula of PL to natural, English…","config":"default","split":"test","body":"{\"category\":\"philosophy\",\"options\":\"[\\\"If it's not the case that both Izzy plays Minecraft and Ashleigh stops by then Russell makes dinner.\\\", \\\"If it's both not the case that Izzy plays Minecraft and Ashleigh stops by then Russell makes dinner.\\\", 'Izzy playing Minecraft and Ashleigh stopping by entails that Russell makes dinner.', \\\"Russell's making dinner is a sufficient condition for Izzy not playing Minecraft and Ashleigh stopping by.\\\"]\",\"question\":\" Use the following key to translate the given formula of PL to natural, English sentences.\\nA: Marina reads a Percy Jackson book.\\nB: Izzy plays Minecraft.\\nC: Emily stops working.\\nD: Russell makes dinner.\\nE: Ashleigh stops by.\\n(~B • E) ⊃ D\",\"src\":\"ori_mmlu-formal_logic\"}","display_format":"text","language":"","answer_status":"published","assets":[],"source_url":"https://huggingface.co/datasets/TIGER-Lab/MMLU-Pro","history":"initial import","indexing_mode":"index","subproblems":[],"grids":[]}