{"kind":"task","effective_mode":"full","benchmark":{"kind":"benchmark","effective_mode":"full","slug":"arc-challenge","formal_name":"ARC-Challenge","introduction":"ARC-Challenge is the subset of grade-school science questions that both a retrieval baseline and a word co-occurrence baseline answered incorrectly. It tests reasoning with knowledge rather than recall alone.","introduction_ja":"","introduction_en":"","category":"Category not supplied","task_count":null,"acquisition_status":"Acquisition status not supplied","official_url":"https://huggingface.co/datasets/allenai/ai2_arc","indexing_mode":"index","profile":{"resources":[],"task_format":"","scoring":"","metric":"","size":"","answer_access":"","license":"","citation":"","maintainer":"","released":"","why_hard":"","related":[],"history":"","construction":"","limitations":"","saturation":"","scores":[]}},"task_id":"921c2ba6-b5ad-5a2f-abe5-6f56b4b54004","task_key":"ARC~2dChallenge--test--MCAS~5f2004~5f9~5f10~2dv1","task_revision_id":"1","upstream_id":"MCAS_2004_9_10-v1","short_description":"When estimating amounts of building supplies, it is better to allow for some…","config":"ARC-Challenge","split":"test","body":"{\"choices\":\"['23', '43', '63', '83 331']\",\"question\":\"When estimating amounts of building supplies, it is better to allow for some waste and error rather than to cause delays by not ordering sufficient supplies. An engineer is estimating the number of bundles of shingles needed for a roof area of approximately 2000 sq. ft. In roofing, a square is 100 sq. ft., and each bundle covers one-third of a square. Which of the following is closest to the number of bundles that the engineer should order?\"}","display_format":"text","language":"","answer_status":"published","assets":[],"source_url":"https://huggingface.co/datasets/allenai/ai2_arc","history":"initial import","indexing_mode":"index","subproblems":[],"grids":[]}