{"kind":"task","effective_mode":"full","benchmark":{"kind":"benchmark","effective_mode":"full","slug":"mmlu-pro","formal_name":"MMLU-Pro","introduction":"MMLU-Pro rebuilds MMLU with ten answer options instead of four and removes items that no longer separate models. Questions span fourteen academic and professional subjects.","introduction_ja":"","introduction_en":"","category":"Category not supplied","task_count":null,"acquisition_status":"Acquisition status not supplied","official_url":"https://huggingface.co/datasets/TIGER-Lab/MMLU-Pro","indexing_mode":"index","profile":{"resources":[],"task_format":"","scoring":"","metric":"","size":"","answer_access":"","license":"","citation":"","maintainer":"","released":"","why_hard":"","related":[],"history":"","construction":"","limitations":"","saturation":"","scores":[]}},"task_id":"9bb5cd8d-59e9-5131-a7a4-d25aaae67856","task_key":"default--test--10495","task_revision_id":"1","upstream_id":"10495","short_description":"What does EXE do when the STP solver times out on a constraint query for a…","config":"default","split":"test","body":"{\"category\":\"computer science\",\"options\":\"['Assume that the query is not satisfiable and stop executing the path', 'Remove a subset of the constraints and retry the query.', 'Pause the execution until the query is resolved.', 'Skip the query and continue executing the path.', 'Assume that the query is satisfiable and continue executing the path.', 'Assume that the query is satisfiable and stop executing the path.', 'Restart STP and retry the query, up to a limited number of retries.', 'Execute the next path without resolving the query.', 'Retry the query until it is resolved, regardless of the number of retries.', 'Send an error message and terminate the program.']\",\"question\":\"What does EXE do when the STP solver times out on a constraint query for a particular path?\",\"src\":\"ori_mmlu-computer_security\"}","display_format":"text","language":"","answer_status":"published","assets":[],"source_url":"https://huggingface.co/datasets/TIGER-Lab/MMLU-Pro","history":"initial import","indexing_mode":"index","subproblems":[],"grids":[]}