{"kind":"task","effective_mode":"full","benchmark":{"kind":"benchmark","effective_mode":"full","slug":"mmlu-pro","formal_name":"MMLU-Pro","introduction":"MMLU-Pro rebuilds MMLU with ten answer options instead of four and removes items that no longer separate models. Questions span fourteen academic and professional subjects.","introduction_ja":"","introduction_en":"","category":"Category not supplied","task_count":null,"acquisition_status":"Acquisition status not supplied","official_url":"https://huggingface.co/datasets/TIGER-Lab/MMLU-Pro","indexing_mode":"index","profile":{"resources":[],"task_format":"","scoring":"","metric":"","size":"","answer_access":"","license":"","citation":"","maintainer":"","released":"","why_hard":"","related":[],"history":"","construction":"","limitations":"","saturation":"","scores":[]}},"task_id":"a2b9ae3e-4744-5c4f-a8f3-0d72819b774f","task_key":"default--test--11224","task_revision_id":"1","upstream_id":"11224","short_description":"Select the best English interpretation of the given proposition, using the…","config":"default","split":"test","body":"{\"category\":\"philosophy\",\"options\":\"['Some house is smaller than every apartment.', 'Every house is bigger than some apartment.', 'Some apartments are bigger than all houses.', 'Some apartments are smaller than all houses.', 'Some house is bigger than every apartment.', 'Every apartment is bigger than some house.', 'All apartments are smaller than some house.', 'Every apartment is smaller than some house.', 'All houses are smaller than some apartment.', 'Every house is bigger than all apartments.']\",\"question\":\" Select the best English interpretation of the given proposition, using the following translation key: Ax: x is an apartment Hx: x is a house Lx: x is large Bxy: x is bigger than y (∀x)[Ax ⊃ (∃y)(Hy • Bxy)]\",\"src\":\"ori_mmlu-formal_logic\"}","display_format":"text","language":"","answer_status":"published","assets":[],"source_url":"https://huggingface.co/datasets/TIGER-Lab/MMLU-Pro","history":"initial import","indexing_mode":"index","subproblems":[],"grids":[]}