{"kind":"task","effective_mode":"full","benchmark":{"kind":"benchmark","effective_mode":"full","slug":"drop","formal_name":"DROP","introduction":"DROP is a reading-comprehension benchmark whose questions require discrete operations over the paragraph — addition, subtraction, comparison, counting, sorting. It is built so that extracting a span cannot answer them.","introduction_ja":"","introduction_en":"","category":"Category not supplied","task_count":null,"acquisition_status":"Acquisition status not supplied","official_url":"https://huggingface.co/datasets/ucinlp/drop","indexing_mode":"index","profile":{"resources":[],"task_format":"","scoring":"","metric":"","size":"","answer_access":"","license":"","citation":"","maintainer":"","released":"","why_hard":"","related":[],"history":"","construction":"","limitations":"","saturation":"","scores":[]}},"task_id":"b9751469-20fc-54f8-958e-6b1fb4db5220","task_key":"default--validation--559d6785~2dd4fe~2d432c~2d8c2b~2df4e15a987579","task_revision_id":"1","upstream_id":"559d6785-d4fe-432c-8c2b-f4e15a987579","short_description":"How many touchdown passes did Roberts catch?","config":"default","split":"validation","body":"{\"passage\":\" The Raiders started well in Tennessee, taking a 13-6 lead at halftime over the Titans on a touchdown pass from Carr to Crabtree. The Raiders jumped out early in the third quarter on touchdown pass to Roberts. However, the Titans rallied scoring 15 points on two Marcus Mariota touchdown passes. Oakland drove late in the game looking for the winning score and appeared to have been stopped on an incomplete pass in the end zone on fourth down. However, the Titans were called for holding and Carr hit Roberts for the game-clinching touchdown two play later. The Raiders halted their three-game losing streak and moved to 5-6 on the season.\",\"question\":\"How many touchdown passes did Roberts catch?\"}","display_format":"text","language":"","answer_status":"published","assets":[],"source_url":"https://huggingface.co/datasets/ucinlp/drop","history":"initial import","indexing_mode":"index","subproblems":[],"grids":[]}