{"kind":"task","effective_mode":"full","benchmark":{"kind":"benchmark","effective_mode":"full","slug":"mmlu-pro","formal_name":"MMLU-Pro","introduction":"MMLU-Pro rebuilds MMLU with ten answer options instead of four and removes items that no longer separate models. Questions span fourteen academic and professional subjects.","introduction_ja":"","introduction_en":"","category":"Category not supplied","task_count":null,"acquisition_status":"Acquisition status not supplied","official_url":"https://huggingface.co/datasets/TIGER-Lab/MMLU-Pro","indexing_mode":"index","profile":{"resources":[],"task_format":"","scoring":"","metric":"","size":"","answer_access":"","license":"","citation":"","maintainer":"","released":"","why_hard":"","related":[],"history":"","construction":"","limitations":"","saturation":"","scores":[]}},"task_id":"be69dc2e-b53c-5a1c-bd64-790a23af69c6","task_key":"default--test--832","task_revision_id":"1","upstream_id":"832","short_description":"Clare manages a piano store. Her utility function is given by Utility = w - 100,…","config":"default","split":"test","body":"{\"category\":\"business\",\"options\":\"['300', '225', '100', '450', '350', '250', '150', '200', '500', '400']\",\"question\":\"Clare manages a piano store. Her utility function is given by Utility = w - 100, where w is the total of all monetary payments to her and 100 represents the monetary equivalent of the disutility of exerting effort to run the store. Her next best alternative to managing the store gives her zero utility. The store's revenue depends on random factors, with an equal chance of being $1,000 or $400. If shareholders offered to share half of the store's revenue with her, what would her expected utility be?\",\"src\":\"theoremQA-Finance\"}","display_format":"text","language":"","answer_status":"published","assets":[],"source_url":"https://huggingface.co/datasets/TIGER-Lab/MMLU-Pro","history":"initial import","indexing_mode":"index","subproblems":[],"grids":[]}