{"kind":"task","effective_mode":"full","benchmark":{"kind":"benchmark","effective_mode":"full","slug":"mmlu-pro","formal_name":"MMLU-Pro","introduction":"MMLU-Pro rebuilds MMLU with ten answer options instead of four and removes items that no longer separate models. Questions span fourteen academic and professional subjects.","introduction_ja":"","introduction_en":"","category":"Category not supplied","task_count":null,"acquisition_status":"Acquisition status not supplied","official_url":"https://huggingface.co/datasets/TIGER-Lab/MMLU-Pro","indexing_mode":"index","profile":{"resources":[],"task_format":"","scoring":"","metric":"","size":"","answer_access":"","license":"","citation":"","maintainer":"","released":"","why_hard":"","related":[],"history":"","construction":"","limitations":"","saturation":"","scores":[]}},"task_id":"18fa29c1-4128-5abe-b22a-bc60f93acebd","task_key":"default--test--10955","task_revision_id":"1","upstream_id":"10955","short_description":"Construct a complete truth table for the following pairs of propositions. Then,…","config":"default","split":"test","body":"{\"category\":\"philosophy\",\"options\":\"['Logically equivalent and inconsistent', 'Neither logically equivalent nor contradictory, but consistent', 'Logically equivalent', 'Inconsistent', 'Contradictory', 'Both contradictory and inconsistent', 'Neither contradictory nor consistent, but logically equivalent', 'Both logically equivalent and contradictory', 'Contradictory and consistent']\",\"question\":\" Construct a complete truth table for the following pairs of propositions. Then, using the truth tables, determine whether the statements are logically equivalent or contradictory. If neither, determine whether they are consistent or inconsistent. Justify your answers.\\n~(J ∨ K) · L and (L ⊃ J) · K\",\"src\":\"ori_mmlu-formal_logic\"}","display_format":"text","language":"","answer_status":"published","assets":[],"source_url":"https://huggingface.co/datasets/TIGER-Lab/MMLU-Pro","history":"initial import","indexing_mode":"index","subproblems":[],"grids":[]}