{"kind":"task","effective_mode":"full","benchmark":{"kind":"benchmark","effective_mode":"full","slug":"mmlu-pro","formal_name":"MMLU-Pro","introduction":"MMLU-Pro rebuilds MMLU with ten answer options instead of four and removes items that no longer separate models. Questions span fourteen academic and professional subjects.","introduction_ja":"","introduction_en":"","category":"Category not supplied","task_count":null,"acquisition_status":"Acquisition status not supplied","official_url":"https://huggingface.co/datasets/TIGER-Lab/MMLU-Pro","indexing_mode":"index","profile":{"resources":[],"task_format":"","scoring":"","metric":"","size":"","answer_access":"","license":"","citation":"","maintainer":"","released":"","why_hard":"","related":[],"history":"","construction":"","limitations":"","saturation":"","scores":[]}},"task_id":"72e6c931-6c3c-524e-8e90-6a581c256c8f","task_key":"default--test--1882","task_revision_id":"1","upstream_id":"1882","short_description":"In a prosecution of a defendant for receiving stolen property, an informer…","config":"default","split":"test","body":"{\"category\":\"law\",\"options\":\"[\\\"improper, as it infringes on the informer's personal privacy.\\\", 'improper, as it is unrelated to the crime being prosecuted.', \\\"proper, as tending to show the informer's bad character for truthfulness.\\\", 'improper, as it is speculative in nature.', \\\"proper, as relevant to the informer's possible bias.\\\", 'proper, as it is a form of character evidence.', 'improper, as relating to convictions of crimes not punishable by imprisonment in excess of one year.', 'improper, as evidence of conduct not related to truthfulness.']\",\"question\":\"In a prosecution of a defendant for receiving stolen property, an informer testified that the defendant sold him a stolen stereo, which the defendant knew had been stolen. During direct examination, the informer testified that he was unemployed but sometimes worked part-time as a substitute teacher. On cross-examination, the defendant's attorney asked the informer if he had recently filed an application to become a police officer. The informer responded affirmatively. The defendant's attorney then asked the informer the following question:\\\"Isn't it true that you hope that by acting as an undercover agent, the police department will overlook the fact that you had two misdemeanor convictions for possession of marijuana?\\\" The prosecuting attorney immediately objected. The trial judge should rule that the defendant's attorney's inquiry concerning the informer's hopes and misdemeanor convictions is\",\"src\":\"ori_mmlu-professional_law\"}","display_format":"text","language":"","answer_status":"published","assets":[],"source_url":"https://huggingface.co/datasets/TIGER-Lab/MMLU-Pro","history":"initial import","indexing_mode":"index","subproblems":[],"grids":[]}