{"kind":"task","effective_mode":"full","benchmark":{"kind":"benchmark","effective_mode":"full","slug":"mmlu-pro","formal_name":"MMLU-Pro","introduction":"MMLU-Pro rebuilds MMLU with ten answer options instead of four and removes items that no longer separate models. Questions span fourteen academic and professional subjects.","introduction_ja":"","introduction_en":"","category":"Category not supplied","task_count":null,"acquisition_status":"Acquisition status not supplied","official_url":"https://huggingface.co/datasets/TIGER-Lab/MMLU-Pro","indexing_mode":"index","profile":{"resources":[],"task_format":"","scoring":"","metric":"","size":"","answer_access":"","license":"","citation":"","maintainer":"","released":"","why_hard":"","related":[],"history":"","construction":"","limitations":"","saturation":"","scores":[]}},"task_id":"1147974c-6454-5cfc-9b04-48384ccf8afe","task_key":"default--test--10948","task_revision_id":"1","upstream_id":"10948","short_description":"Identify the antecedent of the following conditional proposition: The Bees win…","config":"default","split":"test","body":"{\"category\":\"philosophy\",\"options\":\"['The Bees do not win their first game.', 'Both the Aardvarks and the Chipmunks do not win their first games.', 'The Chipmunks do not win their first game.', 'The Chipmunks win their first game.', 'Both the Aardvarks and the Chipmunks win their first games.', 'Either the Aardvarks or the Chipmunks do not win their first games.', 'The Aardvarks do not win their first game.', 'The Bees win their first game.', 'The Aardvarks win their first game.', 'Either the Aardvarks or the Chipmunks win their first games.']\",\"question\":\"Identify the antecedent of the following conditional proposition: The Bees win their first game only if either the Aardvarks or the Chipmunks do not win their first games.\",\"src\":\"ori_mmlu-formal_logic\"}","display_format":"text","language":"","answer_status":"published","assets":[],"source_url":"https://huggingface.co/datasets/TIGER-Lab/MMLU-Pro","history":"initial import","indexing_mode":"index","subproblems":[],"grids":[]}