{"kind":"task","effective_mode":"full","benchmark":{"kind":"benchmark","effective_mode":"full","slug":"mmlu-pro","formal_name":"MMLU-Pro","introduction":"MMLU-Pro rebuilds MMLU with ten answer options instead of four and removes items that no longer separate models. Questions span fourteen academic and professional subjects.","introduction_ja":"","introduction_en":"","category":"Category not supplied","task_count":null,"acquisition_status":"Acquisition status not supplied","official_url":"https://huggingface.co/datasets/TIGER-Lab/MMLU-Pro","indexing_mode":"index","profile":{"resources":[],"task_format":"","scoring":"","metric":"","size":"","answer_access":"","license":"","citation":"","maintainer":"","released":"","why_hard":"","related":[],"history":"","construction":"","limitations":"","saturation":"","scores":[]}},"task_id":"932e99e3-dff2-56e8-ba86-39aacd2556ba","task_key":"default--test--1430","task_revision_id":"1","upstream_id":"1430","short_description":"A heavyset man with long red hair robbed a liquor store. Thereafter, a man was…","config":"default","split":"test","body":"{\"category\":\"law\",\"options\":\"['admissible as a prior identification by the witness.', 'inadmissible, due to the potential for bias or suggestion.', 'admissible as past recollection recorded.', \\\"admissible, as it is the owner's personal account of the event.\\\", \\\"inadmissible, because it is a violation of the man's right of confrontation.\\\", 'admissible, because it is relevant to the case.', 'inadmissible, as it is based on a photograph and not a live lineup.', 'admissible, as it provides critical evidence for the case.', 'inadmissible, because the owner failed to identify the man in court.']\",\"question\":\"A heavyset man with long red hair robbed a liquor store. Thereafter, a man was arrested and charged with the armed robbery. At the man's trial, the owner of the liquor store was called to testify. He admitted that he was unable to identify the man, who now had a shaven head, as the robber. The prosecuting attorney then handed the owner six photographs. He proposed to testify, over defense objections, that he had previously told the prosecuting attorney that picture #4, admittedly a picture of the man before he shaved his head, was a picture of the person who robbed his store. The owner's proffered testimony should be adjudged\",\"src\":\"ori_mmlu-professional_law\"}","display_format":"text","language":"","answer_status":"published","assets":[],"source_url":"https://huggingface.co/datasets/TIGER-Lab/MMLU-Pro","history":"initial import","indexing_mode":"index","subproblems":[],"grids":[]}