{"kind":"task","effective_mode":"full","benchmark":{"kind":"benchmark","effective_mode":"full","slug":"mmlu-pro","formal_name":"MMLU-Pro","introduction":"MMLU-Pro rebuilds MMLU with ten answer options instead of four and removes items that no longer separate models. Questions span fourteen academic and professional subjects.","introduction_ja":"","introduction_en":"","category":"Category not supplied","task_count":null,"acquisition_status":"Acquisition status not supplied","official_url":"https://huggingface.co/datasets/TIGER-Lab/MMLU-Pro","indexing_mode":"index","profile":{"resources":[],"task_format":"","scoring":"","metric":"","size":"","answer_access":"","license":"","citation":"","maintainer":"","released":"","why_hard":"","related":[],"history":"","construction":"","limitations":"","saturation":"","scores":[]}},"task_id":"dc7fc56d-3ef1-57e4-a91c-eb9b010599ba","task_key":"default--test--1612","task_revision_id":"1","upstream_id":"1612","short_description":"A screenwriter had just finished a screenplay for a romantic comedy. After…","config":"default","split":"test","body":"{\"category\":\"law\",\"options\":\"['conditional assignment.', 'non-binding agreement.', 'legal gift.', 'contractual obligation.', 'gratuitous assignment of future rights.', 'binding promise.', 'revocable gift.', 'equitable lien.', 'informal contract.', 'irrevocable assignment.']\",\"question\":\"A screenwriter had just finished a screenplay for a romantic comedy. After reading the screenplay, a movie studio decided to option the screenplay for production. A lawyer for the movie studio negotiated an agreement with the screenwriter whereby the movie studio would pay the screenwriter $5,000 for the rights to the screenplay, and 10 percent of the gross receipts earned by the distribution of the movie, should the studio decide to film the screenplay. Both sides signed the agreement. Two weeks later, the screenwriter was at a birthday party for a close friend when she discovered she had forgotten to purchase a gift for her friend. She quickly grabbed a piece of paper and wrote on it:\\\"I promise that my friend is entitled to 5 percent of the proceeds I will receive for the distribution of the movie to be made from my screenplay. \\\"The screenwriter then signed the paper, placed it in an envelope, and handed it to the friend. The promise to pay 5 percent of the proceeds to the friend may best be interpreted as a (an)\",\"src\":\"ori_mmlu-professional_law\"}","display_format":"text","language":"","answer_status":"published","assets":[],"source_url":"https://huggingface.co/datasets/TIGER-Lab/MMLU-Pro","history":"initial import","indexing_mode":"index","subproblems":[],"grids":[]}