{"kind":"task","effective_mode":"full","benchmark":{"kind":"benchmark","effective_mode":"full","slug":"simpleqa","formal_name":"SimpleQA","introduction":"SimpleQA collects short fact-seeking questions that have exactly one indisputable answer, measuring both how often a model is right and how often it declines. The format is deliberately narrow so that grading is not itself contestable.","introduction_ja":"","introduction_en":"","category":"Category not supplied","task_count":null,"acquisition_status":"Acquisition status not supplied","official_url":"https://github.com/openai/simple-evals","indexing_mode":"index","profile":{"resources":[],"task_format":"","scoring":"","metric":"","size":"","answer_access":"","license":"","citation":"","maintainer":"","released":"","why_hard":"","related":[],"history":"","construction":"","limitations":"","saturation":"","scores":[]}},"task_id":"cb24bf12-3cf2-555e-8aa7-bc9cfaa58e11","task_key":"default--test--cb24bf12-3cf2-555e-8aa7-bc9cfaa58e11","task_revision_id":"1","upstream_id":"","short_description":"In the 1900 report \"Clays of New York, Their Properties and Uses,\" in the…","config":"default","split":"test","body":"{\"problem\":\"In the 1900 report \\\"Clays of New York, Their Properties and Uses,\\\" in the Mineralogy of Clays section, it discusses the experiments of Vogt, which show that kaolinite is not the only substance that remains in suspension for a long time. Tests include potash mica, orthoclase from Norway, and what other mineral from what location?\"}","display_format":"text","language":"","answer_status":"published","assets":[],"source_url":"https://github.com/openai/simple-evals","history":"initial import","indexing_mode":"index","subproblems":[],"grids":[]}