{"kind":"task","effective_mode":"full","benchmark":{"kind":"benchmark","effective_mode":"full","slug":"mmlu-pro","formal_name":"MMLU-Pro","introduction":"MMLU-Pro rebuilds MMLU with ten answer options instead of four and removes items that no longer separate models. Questions span fourteen academic and professional subjects.","introduction_ja":"","introduction_en":"","category":"Category not supplied","task_count":null,"acquisition_status":"Acquisition status not supplied","official_url":"https://huggingface.co/datasets/TIGER-Lab/MMLU-Pro","indexing_mode":"index","profile":{"resources":[],"task_format":"","scoring":"","metric":"","size":"","answer_access":"","license":"","citation":"","maintainer":"","released":"","why_hard":"","related":[],"history":"","construction":"","limitations":"","saturation":"","scores":[]}},"task_id":"70499d23-d731-5416-8b27-44cfc81ded98","task_key":"default--test--1309","task_revision_id":"1","upstream_id":"1309","short_description":"A customer at a fish market was leaving the store after purchasing an assortment…","config":"default","split":"test","body":"{\"category\":\"law\",\"options\":\"['admissible, as it is a statement that was made in the heat of the moment.', 'inadmissible, because the customer did not acknowledge hearing the warning.', 'inadmissible, because it was not a spontaneous utterance.', 'inadmissible, as hearsay not within any exception.', \\\"admissible, because it is relevant as to the customer's contributory negligence.\\\", 'admissible, because it provides context to the event.', 'inadmissible, because the witness was not directly involved in the incident.', 'admissible, because it indicates that the owner had no control over the situation.', \\\"admissible, as a statement of the declarant's present state of mind.\\\", 'inadmissible, as it is considered second-hand information.']\",\"question\":\"A customer at a fish market was leaving the store after purchasing an assortment of shrimp, oysters, and scallops. He was walking along the sidewalk in front of the store when he slipped on a piece of eel. He brought suit against the owner of the market claiming that he suffered leg and back injuries. The owner, although admitting that the customer was injured by slipping on the eel, denied negligence and claimed that the customer was contributorily negligent. At trial, the owner calls a witness to testify that before the fall he heard someone call out to the customer, \\\"Watch it, buddy, you're going to step on that piece of fish. \\\"The witness's testimony is\",\"src\":\"ori_mmlu-professional_law\"}","display_format":"text","language":"","answer_status":"published","assets":[],"source_url":"https://huggingface.co/datasets/TIGER-Lab/MMLU-Pro","history":"initial import","indexing_mode":"index","subproblems":[],"grids":[]}