{"kind":"task","effective_mode":"full","benchmark":{"kind":"benchmark","effective_mode":"full","slug":"mmlu-pro","formal_name":"MMLU-Pro","introduction":"MMLU-Pro rebuilds MMLU with ten answer options instead of four and removes items that no longer separate models. Questions span fourteen academic and professional subjects.","introduction_ja":"","introduction_en":"","category":"Category not supplied","task_count":null,"acquisition_status":"Acquisition status not supplied","official_url":"https://huggingface.co/datasets/TIGER-Lab/MMLU-Pro","indexing_mode":"index","profile":{"resources":[],"task_format":"","scoring":"","metric":"","size":"","answer_access":"","license":"","citation":"","maintainer":"","released":"","why_hard":"","related":[],"history":"","construction":"","limitations":"","saturation":"","scores":[]}},"task_id":"ef9e8aab-dbaf-57b5-8321-e14d2c688f93","task_key":"default--test--1276","task_revision_id":"1","upstream_id":"1276","short_description":"A defendant was charged with illegally selling arms to a foreign country without…","config":"default","split":"test","body":"{\"category\":\"law\",\"options\":\"['admissible, because it shows the absence of an entry in the record of a public agency.', 'inadmissible, because it is an opinion rather than a fact.', 'inadmissible, because it is hearsay not within any recognized exception.', \\\"inadmissible, because the official's statement is speculative and not based on concrete evidence.\\\", \\\"admissible, because the official had firsthand knowledge of the information contained in the agency's employment records.\\\", 'admissible, because the official is an authoritative figure in the agency.', \\\"admissible, because it directly contradicts the defendant's testimony.\\\", \\\"inadmissible, because the official did not personally verify the absence of the defendant's employment records.\\\", 'inadmissible, because the agency employment records are the best evidence to substantiate whether the defendant was actually in the employ of the governmental intelligence agency.', \\\"admissible, as it provides evidence of the agency's standard operating procedures.\\\"]\",\"question\":\"A defendant was charged with illegally selling arms to a foreign country without a government license. During the trial, the defendant admitted selling the weapons, which included missiles, rocket launchers, bazookas, and ammunition, to agents of the foreign country. In his defense, however, the defendant testified that he did not sell the arms as a private citizen, but rather for his employer, a federal agency. In rebutting the defendant's testimony, the government called a high-ranking official from the federal agency to the stand. He testified that if the defendant were, in fact, a member of the agency, that information would ordinarily be contained in the agency's employment records. Asked whether the defendant's employment records were on file with the agency, the official then testified, \\\"I have searched the agency employment records diligently, but I haven't found any documentation that the defendant was employed by the agency. \\\"Upon objection by the defendant's attorney, the official's testimony is\",\"src\":\"ori_mmlu-professional_law\"}","display_format":"text","language":"","answer_status":"published","assets":[],"source_url":"https://huggingface.co/datasets/TIGER-Lab/MMLU-Pro","history":"initial import","indexing_mode":"index","subproblems":[],"grids":[]}