{"kind":"task","effective_mode":"full","benchmark":{"kind":"benchmark","effective_mode":"full","slug":"mmlu-pro","formal_name":"MMLU-Pro","introduction":"MMLU-Pro rebuilds MMLU with ten answer options instead of four and removes items that no longer separate models. Questions span fourteen academic and professional subjects.","introduction_ja":"","introduction_en":"","category":"Category not supplied","task_count":null,"acquisition_status":"Acquisition status not supplied","official_url":"https://huggingface.co/datasets/TIGER-Lab/MMLU-Pro","indexing_mode":"index","profile":{"resources":[],"task_format":"","scoring":"","metric":"","size":"","answer_access":"","license":"","citation":"","maintainer":"","released":"","why_hard":"","related":[],"history":"","construction":"","limitations":"","saturation":"","scores":[]}},"task_id":"0622fc0b-101a-533c-abc8-e9aece55399d","task_key":"default--test--1261","task_revision_id":"1","upstream_id":"1261","short_description":"Doug Dogman had a domesticated golden retriever named Rover. Rover was a good,…","config":"default","split":"test","body":"{\"category\":\"law\",\"options\":\"['Doug, unless a statute or ordinance bans unleashed dogs on public streets.', 'Doug, because Retriever is a domesticated animal.', 'Ginny, because Rover was not on a leash and was not supervised.', 'Doug, because Ginny was responsible for maintaining control of her vehicle.', 'Ginny, because Rover was chasing cars, a potentially dangerous activity.', \\\"Ginny, because Doug's dog was a cause in fact of Ginny's injuries.\\\", 'Doug, because the strict liability rule for dogs would not include dogs not displaying vicious propensities.', 'Doug, because Rover was not known to be a dangerous dog.', 'Doug, because Ginny was negligent in hitting a parked car.', 'Ginny, because Doug was not present when the incident occurred.']\",\"question\":\"Doug Dogman had a domesticated golden retriever named Rover. Rover was a good, friendly dog who loved to run and chase cars but had never hurt anyone. One dark evening before the street lights turned on, Rover was chasing cars in the street, and Ginny Grandmother, aged 85, swerved to avoid hitting the dog. As she swerved, Ginny hit a parked car and suffered a head injury. If Ginny brings suit against Doug Dogman under a strict liability theory, the court will likely find for\",\"src\":\"ori_mmlu-professional_law\"}","display_format":"text","language":"","answer_status":"published","assets":[],"source_url":"https://huggingface.co/datasets/TIGER-Lab/MMLU-Pro","history":"initial import","indexing_mode":"index","subproblems":[],"grids":[]}