{"kind":"task","effective_mode":"full","benchmark":{"kind":"benchmark","effective_mode":"full","slug":"mmlu-pro","formal_name":"MMLU-Pro","introduction":"MMLU-Pro rebuilds MMLU with ten answer options instead of four and removes items that no longer separate models. Questions span fourteen academic and professional subjects.","introduction_ja":"","introduction_en":"","category":"Category not supplied","task_count":null,"acquisition_status":"Acquisition status not supplied","official_url":"https://huggingface.co/datasets/TIGER-Lab/MMLU-Pro","indexing_mode":"index","profile":{"resources":[],"task_format":"","scoring":"","metric":"","size":"","answer_access":"","license":"","citation":"","maintainer":"","released":"","why_hard":"","related":[],"history":"","construction":"","limitations":"","saturation":"","scores":[]}},"task_id":"1fdae9c8-3248-507f-a0a1-2f59b1e7330f","task_key":"default--test--10947","task_revision_id":"1","upstream_id":"10947","short_description":"Select the best English interpretation of the given proposition, using the…","config":"default","split":"test","body":"{\"category\":\"philosophy\",\"options\":\"['No large house is bigger than some apartments.', 'All apartments are not bigger than some large house.', 'No apartment is bigger than any large house.', 'No large house is bigger than all apartments.', 'All large houses are bigger than any apartment.', 'No apartment is smaller than any large house.', 'All large houses are not bigger than some apartment.', 'Some apartments are not bigger than all large houses.', 'Some large houses are not bigger than all apartments.', 'All apartments are bigger than any large house.']\",\"question\":\" Select the best English interpretation of the given proposition, using the following translation key: Ax: x is an apartment Hx: x is a house Lx: x is large Bxy: x is bigger than y (∀x){Ax ⊃ (∀y)[(Hy • Ly) ⊃ ∼Bxy]}\",\"src\":\"ori_mmlu-formal_logic\"}","display_format":"text","language":"","answer_status":"published","assets":[],"source_url":"https://huggingface.co/datasets/TIGER-Lab/MMLU-Pro","history":"initial import","indexing_mode":"index","subproblems":[],"grids":[]}