{"kind":"task","effective_mode":"full","benchmark":{"kind":"benchmark","effective_mode":"full","slug":"mmlu-pro","formal_name":"MMLU-Pro","introduction":"MMLU-Pro rebuilds MMLU with ten answer options instead of four and removes items that no longer separate models. Questions span fourteen academic and professional subjects.","introduction_ja":"","introduction_en":"","category":"Category not supplied","task_count":null,"acquisition_status":"Acquisition status not supplied","official_url":"https://huggingface.co/datasets/TIGER-Lab/MMLU-Pro","indexing_mode":"index","profile":{"resources":[],"task_format":"","scoring":"","metric":"","size":"","answer_access":"","license":"","citation":"","maintainer":"","released":"","why_hard":"","related":[],"history":"","construction":"","limitations":"","saturation":"","scores":[]}},"task_id":"ccd3849e-d4a1-5d08-91d7-2d5f09015785","task_key":"default--test--1859","task_revision_id":"1","upstream_id":"1859","short_description":"An elderly woman entered a convenience store and tried to cash her Social…","config":"default","split":"test","body":"{\"category\":\"law\",\"options\":\"['Blackmail.', 'Fraud.', 'Conspiracy to commit robbery.', 'Robbery.', 'False pretenses.', 'No crime.', 'Assault.', 'Attempted robbery.', 'Threatening behavior.']\",\"question\":\"An elderly woman entered a convenience store and tried to cash her Social Security check. The owner told the woman that he was sorry but it was his policy not to cash Social Security checks. Terribly upset, the woman reached into her pocketbook and said, \\\"Listen, buddy, I've got a bomb in here, and if you don't give me the money, I'm going to blow up this joint. \\\" Actually, the woman did not have a bomb in her possession. The owner, who was not deceived by her threat, felt sorry for the woman and agreed to cash the check. The woman, who walked with the assistance of a cane, took the money and limped out of the store. After she left, the owner noticed that the woman had not endorsed the check. The woman has committed which, if any, of the following crimes?\",\"src\":\"ori_mmlu-professional_law\"}","display_format":"text","language":"","answer_status":"published","assets":[],"source_url":"https://huggingface.co/datasets/TIGER-Lab/MMLU-Pro","history":"initial import","indexing_mode":"index","subproblems":[],"grids":[]}