{"kind":"task","effective_mode":"full","benchmark":{"kind":"benchmark","effective_mode":"full","slug":"mmlu-pro","formal_name":"MMLU-Pro","introduction":"MMLU-Pro rebuilds MMLU with ten answer options instead of four and removes items that no longer separate models. Questions span fourteen academic and professional subjects.","introduction_ja":"","introduction_en":"","category":"Category not supplied","task_count":null,"acquisition_status":"Acquisition status not supplied","official_url":"https://huggingface.co/datasets/TIGER-Lab/MMLU-Pro","indexing_mode":"index","profile":{"resources":[],"task_format":"","scoring":"","metric":"","size":"","answer_access":"","license":"","citation":"","maintainer":"","released":"","why_hard":"","related":[],"history":"","construction":"","limitations":"","saturation":"","scores":[]}},"task_id":"c4abf209-c512-5669-9b86-dd0c2f4bf564","task_key":"default--test--10863","task_revision_id":"1","upstream_id":"10863","short_description":"Select the best English interpretation of the given arguments in predicate…","config":"default","split":"test","body":"{\"category\":\"philosophy\",\"options\":\"['Either Nancy or Marvin are at work. All lawyers are not at work if they are out to dinner. Nancy is a lawyer and not out to dinner. So not everything is not at work.', 'Either Nancy or Marvin are at work. All lawyers are at work if they are out to dinner. Nancy is a lawyer and out to dinner. So not everything is at work.', 'Either Nancy or Marvin are at work. All lawyers are not at work if they are out to dinner. Nancy is not a lawyer and out to dinner. So not everything is not at work.', 'Either Nancy or Marvin are at work. All lawyers are not at work if they are out to dinner. Nancy is a lawyer and out to dinner. So not everything is at work.', 'Either Nancy or Marvin are at work. All lawyers are out to dinner if they are not at work. Nancy is a lawyer and out to dinner. So not everything is at work.', 'Either Nancy or Marvin are not at work. All lawyers are out to dinner if they are at work. Nancy is a lawyer and out to dinner. So not everything is at work.', 'Either Nancy or Marvin are at work. All lawyers are not at work if they are out to dinner. Nancy is a lawyer and out to dinner. So not everything is not at work.', 'Either Nancy or Marvin are not at work. All lawyers are not at work if they are out to dinner. Nancy is a lawyer and out to dinner. So not everything is at work.', 'Either Nancy or Marvin are at work. All lawyers are at work if they are not out to dinner. Nancy is a lawyer and out to dinner. So not everything is not at work.', 'Either Nancy or Marvin are at work. All lawyers are out to dinner if they are not at work. Nancy is a lawyer and out to dinner. So not everything is not at work.']\",\"question\":\" Select the best English interpretation of the given arguments in predicate logic.\\nWn ∨ Wm\\n(∀x)[Lx ⊃ (Dx ⊃ ~Wx)]\\nLn • Dn\\t/ ~(∀x)~Wx\",\"src\":\"ori_mmlu-formal_logic\"}","display_format":"text","language":"","answer_status":"published","assets":[],"source_url":"https://huggingface.co/datasets/TIGER-Lab/MMLU-Pro","history":"initial import","indexing_mode":"index","subproblems":[],"grids":[]}