{"kind":"task","effective_mode":"full","benchmark":{"kind":"benchmark","effective_mode":"full","slug":"arc-challenge","formal_name":"ARC-Challenge","introduction":"ARC-Challenge is the subset of grade-school science questions that both a retrieval baseline and a word co-occurrence baseline answered incorrectly. It tests reasoning with knowledge rather than recall alone.","introduction_ja":"","introduction_en":"","category":"Category not supplied","task_count":null,"acquisition_status":"Acquisition status not supplied","official_url":"https://huggingface.co/datasets/allenai/ai2_arc","indexing_mode":"index","profile":{"resources":[],"task_format":"","scoring":"","metric":"","size":"","answer_access":"","license":"","citation":"","maintainer":"","released":"","why_hard":"","related":[],"history":"","construction":"","limitations":"","saturation":"","scores":[]}},"task_id":"49e2760d-058e-5374-973f-60f933c8d2ac","task_key":"ARC~2dChallenge--test--Mercury~5fSC~5f414156","task_revision_id":"1","upstream_id":"Mercury_SC_414156","short_description":"A compass can be made by magnetizing a needle and setting it on a floating cork.…","config":"ARC-Challenge","split":"test","body":"{\"choices\":\"['The needle is measuring the air temperature.', 'The needle is showing the direction of the wind.', \\\"The needle is lining up with Earth's magnetic poles.\\\", 'The needle is pointing to the magnetic core of Earth.']\",\"question\":\"A compass can be made by magnetizing a needle and setting it on a floating cork. The cork will turn so that the needle is pointed north. Which statement explains why the needle points north?\"}","display_format":"text","language":"","answer_status":"published","assets":[],"source_url":"https://huggingface.co/datasets/allenai/ai2_arc","history":"initial import","indexing_mode":"index","subproblems":[],"grids":[]}