{"kind":"task","effective_mode":"full","benchmark":{"kind":"benchmark","effective_mode":"full","slug":"hellaswag","formal_name":"HellaSwag","introduction":"HellaSwag asks which of four continuations naturally follows a sentence or a video caption. The wrong options are produced by adversarial filtering, so items that are nearly trivial for people remain hard for models.","introduction_ja":"","introduction_en":"","category":"Category not supplied","task_count":null,"acquisition_status":"Acquisition status not supplied","official_url":"https://rowanzellers.com/hellaswag/","indexing_mode":"index","profile":{"resources":[],"task_format":"","scoring":"","metric":"","size":"","answer_access":"","license":"","citation":"","maintainer":"","released":"","why_hard":"","related":[],"history":"","construction":"","limitations":"","saturation":"","scores":[]}},"task_id":"ec4a1e47-ae90-551d-9ae3-4bd4f72bb705","task_key":"default--validation--20170","task_revision_id":"1","upstream_id":"20170","short_description":"HellaSwag validation 20170","config":"default","split":"validation","body":"{\"activity_label\":\"Family Life\",\"ctx\":\"[header] How to install a car seat [title] Separate the carrier from the base. [step] Most rear-facing seats consist of two parts: the base and the infant carrier. You will install the base, the part that remains in the car.\",\"endings\":\"['[substeps] Check to make sure the base remains inside the vehicle, and on the exterior of the vehicle, but not in any of the rear-facing seats. Lift the base out of the backseat and place it on a table in front of you.', 'Consider moving the base out of the vehicle once you have finished installing the seat. This piece will sit in the trunk and provide your infant with the seat.', \\\"Follow the manufacturer's instructions for removing the carrier from the base. Usually, there is simply a lever or button to activate.\\\", \\\"[title] Test for bias with the index finger of one hand on the infant carrier's base. [step] You will use your other thumb to separate the base and the upper corner.\\\"]\"}","display_format":"text","language":"","answer_status":"published","assets":[],"source_url":"https://rowanzellers.com/hellaswag/","history":"initial import","indexing_mode":"index","subproblems":[],"grids":[]}