{"kind":"task","effective_mode":"full","benchmark":{"kind":"benchmark","effective_mode":"full","slug":"hellaswag","formal_name":"HellaSwag","introduction":"HellaSwag asks which of four continuations naturally follows a sentence or a video caption. The wrong options are produced by adversarial filtering, so items that are nearly trivial for people remain hard for models.","introduction_ja":"","introduction_en":"","category":"Category not supplied","task_count":null,"acquisition_status":"Acquisition status not supplied","official_url":"https://rowanzellers.com/hellaswag/","indexing_mode":"index","profile":{"resources":[],"task_format":"","scoring":"","metric":"","size":"","answer_access":"","license":"","citation":"","maintainer":"","released":"","why_hard":"","related":[],"history":"","construction":"","limitations":"","saturation":"","scores":[]}},"task_id":"633b012b-8dab-5eb9-8429-5055bb327467","task_key":"default--validation--44890","task_revision_id":"1","upstream_id":"44890","short_description":"HellaSwag validation 44890","config":"default","split":"validation","body":"{\"activity_label\":\"Home and Garden\",\"ctx\":\"[header] How to build a seawall [title] Survey the area. [step] Walk along the area where you intend to build your seawall, carrying a tape measure, notepad, and pen. Use the tape measure to track the length you need to achieve.\",\"endings\":\"[\\\"Use the pad and pen to sketch a rough outline of your shore, taking special note of any changes in elevation or sharp curves. Finally, use spray paint to mark off 8' (2.43 m) intervals where you will place your poles.\\\", \\\"The length is the number of bars that you will need to excavate underground to create a tunnel. You don't need to be very precise with your measurements.\\\", '[title] Locate a concrete vertical weight of approximately 12 inches (30. 5 cm) and measurement 3 inches (7.6 cm) below that.', \\\"As you walk, you'll be observing how the thickness of the land changes over time. The distance between a seawall's thickest and smallest outcrops is two inches (5 cm).\\\"]\"}","display_format":"text","language":"","answer_status":"published","assets":[],"source_url":"https://rowanzellers.com/hellaswag/","history":"initial import","indexing_mode":"index","subproblems":[],"grids":[]}