{"kind":"task","effective_mode":"full","benchmark":{"kind":"benchmark","effective_mode":"full","slug":"mmlu-pro","formal_name":"MMLU-Pro","introduction":"MMLU-Pro rebuilds MMLU with ten answer options instead of four and removes items that no longer separate models. Questions span fourteen academic and professional subjects.","introduction_ja":"","introduction_en":"","category":"Category not supplied","task_count":null,"acquisition_status":"Acquisition status not supplied","official_url":"https://huggingface.co/datasets/TIGER-Lab/MMLU-Pro","indexing_mode":"index","profile":{"resources":[],"task_format":"","scoring":"","metric":"","size":"","answer_access":"","license":"","citation":"","maintainer":"","released":"","why_hard":"","related":[],"history":"","construction":"","limitations":"","saturation":"","scores":[]}},"task_id":"8260278f-9f76-568b-8757-b5eeaf910cfc","task_key":"default--test--1672","task_revision_id":"1","upstream_id":"1672","short_description":"In which of the following situations would the actions of the defendant(s)…","config":"default","split":"test","body":"{\"category\":\"law\",\"options\":\"['A defendant, having lost his keys, broke a window to enter his own house.', 'A defendant, seeking shelter from a severe snowstorm, broke into a house. When he awoke the next morning, the defendant noticed a transistor radio, which he put into his pocket before fleeing the house.', 'A defendant broke into a museum at night, not to steal, but to add his own artwork to the exhibition.', \\\"A defendant, in an attempt to retrieve a ball that had accidentally been thrown into a neighbor's yard, climbed over the fence and entered the yard.\\\", 'A defendant was walking past a home when he saw the homeowner sitting in a chair near the window. Angered at the homeowner for stealing his former girlfriend, the defendant picked up a large brick that was lying nearby and hurled it through the window, intending to kill the homeowner.', 'A defendant, under the influence of alcohol, mistakenly entered the wrong house believing it was his own and fell asleep on the couch.']\",\"question\":\"In which of the following situations would the actions of the defendant(s) constitute a burglary?\",\"src\":\"ori_mmlu-professional_law\"}","display_format":"text","language":"","answer_status":"published","assets":[],"source_url":"https://huggingface.co/datasets/TIGER-Lab/MMLU-Pro","history":"initial import","indexing_mode":"index","subproblems":[],"grids":[]}