{"kind":"task","effective_mode":"full","benchmark":{"kind":"benchmark","effective_mode":"full","slug":"mmlu-pro","formal_name":"MMLU-Pro","introduction":"MMLU-Pro rebuilds MMLU with ten answer options instead of four and removes items that no longer separate models. Questions span fourteen academic and professional subjects.","introduction_ja":"","introduction_en":"","category":"Category not supplied","task_count":null,"acquisition_status":"Acquisition status not supplied","official_url":"https://huggingface.co/datasets/TIGER-Lab/MMLU-Pro","indexing_mode":"index","profile":{"resources":[],"task_format":"","scoring":"","metric":"","size":"","answer_access":"","license":"","citation":"","maintainer":"","released":"","why_hard":"","related":[],"history":"","construction":"","limitations":"","saturation":"","scores":[]}},"task_id":"6db8aa3c-fed9-5753-8842-1d83b990e176","task_key":"default--test--1527","task_revision_id":"1","upstream_id":"1527","short_description":"A customer is suing a car dealer for selling him a salvaged vehicle that the car…","config":"default","split":"test","body":"{\"category\":\"law\",\"options\":\"['admissible as a statement of intent.', \\\"inadmissible as it violates the defendant's right to confront the witness.\\\", 'admissible as a declaration against interest.', 'admissible as a statement of present sense impression.', 'inadmissible as hearsay not within any recognized exception.', 'admissible under the excited utterance exception.', 'inadmissible as the statement was not made under oath.', 'inadmissible due to the absence of the defendant.', 'admissible as a statement of then-existing mental or emotional condition.', 'admissible as a spontaneous statement.']\",\"question\":\"A customer is suing a car dealer for selling him a salvaged vehicle that the car dealer had represented as being brand new. A few weeks before trial, the car dealer approached his sister and said, \\\"Sis, I need some sympathy. I sold a salvaged vehicle to a customer, and now he's suing me. I didn't mean any harm by it. I inspected the vehicle and everything. \\\" Unknown to either the car dealer or his sister, the sister's boyfriend was at the front door and overheard this conversation. When the time for trial came around, the car dealer left the country and refused to attend the trial, telling his attorney to handle it. The customer's attorney attempted several times to secure the car dealer's attendance at trial, but was unsuccessful. At trial, the sister's boyfriend is called to testify about the conversation he overheard. On objection by the car dealer's attorney, the court will most likely rule the testimony\",\"src\":\"ori_mmlu-professional_law\"}","display_format":"text","language":"","answer_status":"published","assets":[],"source_url":"https://huggingface.co/datasets/TIGER-Lab/MMLU-Pro","history":"initial import","indexing_mode":"index","subproblems":[],"grids":[]}