{"kind":"task","effective_mode":"full","benchmark":{"kind":"benchmark","effective_mode":"full","slug":"mmlu-pro","formal_name":"MMLU-Pro","introduction":"MMLU-Pro rebuilds MMLU with ten answer options instead of four and removes items that no longer separate models. Questions span fourteen academic and professional subjects.","introduction_ja":"","introduction_en":"","category":"Category not supplied","task_count":null,"acquisition_status":"Acquisition status not supplied","official_url":"https://huggingface.co/datasets/TIGER-Lab/MMLU-Pro","indexing_mode":"index","profile":{"resources":[],"task_format":"","scoring":"","metric":"","size":"","answer_access":"","license":"","citation":"","maintainer":"","released":"","why_hard":"","related":[],"history":"","construction":"","limitations":"","saturation":"","scores":[]}},"task_id":"6311b3a6-e569-5ef7-b0b8-9b3a6fee6e2d","task_key":"default--test--1764","task_revision_id":"1","upstream_id":"1764","short_description":"An employee worked for a hardware company in a city under an at-will-employment…","config":"default","split":"test","body":"{\"category\":\"law\",\"options\":\"['enforceable only with respect to the salary and termination provisions.', 'enforceable only with respect to the salary provision.', 'enforceable only with respect to the $1,500 penalty clause.', 'enforceable only with respect to the termination provision.', 'enforceable in all respects, except with regard to the $1,500 penalty clause.', 'enforceable only with respect to the non-competition clause.', 'enforceable in all respects, except with regard to the non-competition clause.', 'unenforceable in all respects.', 'enforceable in all respects, except with regard to the salary provision.', 'enforceable in all respects.']\",\"question\":\"An employee worked for a hardware company in a city under an at-will-employment arrangement that paid the employee a monthly salary of $2,000. After two years, the owner of the hardware store told the employee it was time for them to enter into a more formalized employment agreement. The owner and the employee then both signed a written agreement drafted by the owner, whereby the employee agreed to work for a salary of $3,000 per month. The agreement further specified that either party had a right to terminate the contract by giving 60- days' notice to the other. Moreover, the agreement stipulated that if the employee's employment at the hardware store should be terminated, he would not be able to engage in any aspect of the hardware business for a period of two years within a 50-mile radius of the city. Finally, the agreement stated that should the employee violate this agreement, he would be liable to the hardware company for $1,500. The written agreement between the owner and the employee would most likely be held\",\"src\":\"ori_mmlu-professional_law\"}","display_format":"text","language":"","answer_status":"published","assets":[],"source_url":"https://huggingface.co/datasets/TIGER-Lab/MMLU-Pro","history":"initial import","indexing_mode":"index","subproblems":[],"grids":[]}