{"kind":"task","effective_mode":"full","benchmark":{"kind":"benchmark","effective_mode":"full","slug":"mmlu-pro","formal_name":"MMLU-Pro","introduction":"MMLU-Pro rebuilds MMLU with ten answer options instead of four and removes items that no longer separate models. Questions span fourteen academic and professional subjects.","introduction_ja":"","introduction_en":"","category":"Category not supplied","task_count":null,"acquisition_status":"Acquisition status not supplied","official_url":"https://huggingface.co/datasets/TIGER-Lab/MMLU-Pro","indexing_mode":"index","profile":{"resources":[],"task_format":"","scoring":"","metric":"","size":"","answer_access":"","license":"","citation":"","maintainer":"","released":"","why_hard":"","related":[],"history":"","construction":"","limitations":"","saturation":"","scores":[]}},"task_id":"21c57a57-efc1-540b-8c5b-4175d53fd920","task_key":"default--test--2407","task_revision_id":"1","upstream_id":"2407","short_description":"Shannon forgot her pillow when she went camping, so she complained about having…","config":"default","split":"test","body":"{\"category\":\"psychology\",\"options\":\"['functional fixedness', 'the availability heuristic', 'the overconfidence effect', 'the representativeness heuristic', 'confirmation bias', 'the framing effect', 'the hindsight bias', 'the illusion of control', 'cognitive dissonance', 'the anchoring effect']\",\"question\":\"Shannon forgot her pillow when she went camping, so she complained about having to sleep with her head flat on the ground the whole night. Her failure to fold up her jeans and sweater to use as a pillow-substitute best illustrates the effects of\",\"src\":\"ori_mmlu-high_school_psychology\"}","display_format":"text","language":"","answer_status":"published","assets":[],"source_url":"https://huggingface.co/datasets/TIGER-Lab/MMLU-Pro","history":"initial import","indexing_mode":"index","subproblems":[],"grids":[]}