{"kind":"task","effective_mode":"full","benchmark":{"kind":"benchmark","effective_mode":"full","slug":"mmlu-pro","formal_name":"MMLU-Pro","introduction":"MMLU-Pro rebuilds MMLU with ten answer options instead of four and removes items that no longer separate models. Questions span fourteen academic and professional subjects.","introduction_ja":"","introduction_en":"","category":"Category not supplied","task_count":null,"acquisition_status":"Acquisition status not supplied","official_url":"https://huggingface.co/datasets/TIGER-Lab/MMLU-Pro","indexing_mode":"index","profile":{"resources":[],"task_format":"","scoring":"","metric":"","size":"","answer_access":"","license":"","citation":"","maintainer":"","released":"","why_hard":"","related":[],"history":"","construction":"","limitations":"","saturation":"","scores":[]}},"task_id":"296f5784-b685-5c71-9b1c-269e6fa2d297","task_key":"default--test--10794","task_revision_id":"1","upstream_id":"10794","short_description":"Baron cites a summary of Ali Soufan's Senate testimony to suggest that","config":"default","split":"test","body":"{\"category\":\"philosophy\",\"options\":\"['torture is always wrong without exception.', 'torture is the most effective method of interrogation.', 'successful interrogation may not require torture.', 'all successful interrogations involve some form of torture.', 'Ali Soufan was subjected to torture during his testimony.', 'Ali Soufan denounced the use of torture entirely in his testimony.', 'endorsing a single act of torture requires endorsing torture as a practice.', 'Ali Soufan supports the use of torture in interrogation.', 'torture can be ethically justified in some situations.', 'torture works in only limited situations.']\",\"question\":\" Baron cites a summary of Ali Soufan's Senate testimony to suggest that\",\"src\":\"ori_mmlu-moral_disputes\"}","display_format":"text","language":"","answer_status":"published","assets":[],"source_url":"https://huggingface.co/datasets/TIGER-Lab/MMLU-Pro","history":"initial import","indexing_mode":"index","subproblems":[],"grids":[]}