{"kind":"task","effective_mode":"full","benchmark":{"kind":"benchmark","effective_mode":"full","slug":"hellaswag","formal_name":"HellaSwag","introduction":"HellaSwag asks which of four continuations naturally follows a sentence or a video caption. The wrong options are produced by adversarial filtering, so items that are nearly trivial for people remain hard for models.","introduction_ja":"","introduction_en":"","category":"Category not supplied","task_count":null,"acquisition_status":"Acquisition status not supplied","official_url":"https://rowanzellers.com/hellaswag/","indexing_mode":"index","profile":{"resources":[],"task_format":"","scoring":"","metric":"","size":"","answer_access":"","license":"","citation":"","maintainer":"","released":"","why_hard":"","related":[],"history":"","construction":"","limitations":"","saturation":"","scores":[]}},"task_id":"a524f332-5b4d-5089-be71-ef7179e99888","task_key":"default--validation--12274","task_revision_id":"1","upstream_id":"12274","short_description":"HellaSwag validation 12274","config":"default","split":"validation","body":"{\"activity_label\":\"Cars & Other Vehicles\",\"ctx\":\"[header] How to retrieve keys locked inside a car with a pull up lock [title] Get a long piece of string. [step] Find a long piece of string that's at least 3 feet (91.44 cm) long. You can use yarn or a thin rope.\",\"endings\":\"['Draw it around the edge of the front side of the car. Tie the two ends together on the end with tape.', \\\"String should be thick enough to hold the key. String often comes in various lengths, and it's important to gather the length well before making to minimize the chances of it breaking.\\\", \\\"If you don't have access to any of these materials, you can even use your shoelaces. [title] Make a slip knot [step] In the center of the string.\\\", 'There should be thick pieces on all sides and no slack. [substeps] The thicker the string, the more secure the lock is going to be.']\"}","display_format":"text","language":"","answer_status":"published","assets":[],"source_url":"https://rowanzellers.com/hellaswag/","history":"initial import","indexing_mode":"index","subproblems":[],"grids":[]}