{"kind":"task","effective_mode":"full","benchmark":{"kind":"benchmark","effective_mode":"full","slug":"mmlu-pro","formal_name":"MMLU-Pro","introduction":"MMLU-Pro rebuilds MMLU with ten answer options instead of four and removes items that no longer separate models. Questions span fourteen academic and professional subjects.","introduction_ja":"","introduction_en":"","category":"Category not supplied","task_count":null,"acquisition_status":"Acquisition status not supplied","official_url":"https://huggingface.co/datasets/TIGER-Lab/MMLU-Pro","indexing_mode":"index","profile":{"resources":[],"task_format":"","scoring":"","metric":"","size":"","answer_access":"","license":"","citation":"","maintainer":"","released":"","why_hard":"","related":[],"history":"","construction":"","limitations":"","saturation":"","scores":[]}},"task_id":"3356a6ac-759a-5c6d-b36c-5ad24dc0d1ff","task_key":"default--test--1652","task_revision_id":"1","upstream_id":"1652","short_description":"A resident lived in a house across the street from a small office building. One…","config":"default","split":"test","body":"{\"category\":\"law\",\"options\":\"[\\\"succeed, because the resident's remarks were offensive and damaging to the lawyer's reputation.\\\", 'succeed, because the lawyer found the remarks to be upsetting.', \\\"not succeed, because the resident's remarks were a matter of personal opinion rather than statements of fact.\\\", \\\"succeed, because the resident's remarks falsely accused the lawyer of committing a crime.\\\", \\\"not succeed, because the lawyer chose to continue his practice despite the resident's remarks.\\\", \\\"succeed, because the resident's remarks constituted slander per Se.\\\", \\\"not succeed, because the resident's remarks were made in a personal conversation, not in a public setting.\\\", \\\"succeed, because the resident's remarks caused emotional distress to the lawyer.\\\", \\\"not succeed, because the resident's remarks were not published or communicated to anyone but the plaintiff.\\\", \\\"not succeed, because the resident has a right to express her feelings about the lawyer's profession.\\\"]\",\"question\":\"A resident lived in a house across the street from a small office building. One of the offices in that building was that of a criminal defense lawyer. The resident hated the idea of the lawyer's office being close to her home. She felt that it meant dangerous criminals were coming and going in her neighborhood. The resident spent a lot of time looking out her window, trying to figure out which people going in and out of the office building were criminals, and imagining what sorts of crimes they had committed. The resident eventually decided that she needed to drive the lawyer out of the neighborhood. She made a telephone call to the lawyer and said, \\\"You are scum, and so are all of your clients. I know that you're conspiring to commit crimes with all those bad people you represent. The police should arrest you and put you in jail with your clients. \\\" The lawyer was very upset by what he heard. If the lawyer asserts a claim for defamation against the resident based on the telephone call, he will most likely\",\"src\":\"ori_mmlu-professional_law\"}","display_format":"text","language":"","answer_status":"published","assets":[],"source_url":"https://huggingface.co/datasets/TIGER-Lab/MMLU-Pro","history":"initial import","indexing_mode":"index","subproblems":[],"grids":[]}