{"kind":"task","effective_mode":"full","benchmark":{"kind":"benchmark","effective_mode":"full","slug":"hellaswag","formal_name":"HellaSwag","introduction":"HellaSwag asks which of four continuations naturally follows a sentence or a video caption. The wrong options are produced by adversarial filtering, so items that are nearly trivial for people remain hard for models.","introduction_ja":"","introduction_en":"","category":"Category not supplied","task_count":null,"acquisition_status":"Acquisition status not supplied","official_url":"https://rowanzellers.com/hellaswag/","indexing_mode":"index","profile":{"resources":[],"task_format":"","scoring":"","metric":"","size":"","answer_access":"","license":"","citation":"","maintainer":"","released":"","why_hard":"","related":[],"history":"","construction":"","limitations":"","saturation":"","scores":[]}},"task_id":"cc2537c1-8bdd-54ee-8855-310b4734b62f","task_key":"default--validation--12432","task_revision_id":"1","upstream_id":"12432","short_description":"HellaSwag validation 12432","config":"default","split":"validation","body":"{\"activity_label\":\"Computers and Electronics\",\"ctx\":\"[header] How to open a padlock [title] Look on the back of the lock to find the combination. [step] Most padlocks will have the combination listed either on the back of the lock or somewhere within the packaging. Look for a three digit code when you buy your lock.\",\"endings\":\"['[substeps] Also look for a colored one that indicates which way the lock would turn when open. The combination may also have an after a number or letters.', 'You will need to confirm the combination after you open the lock. [substeps] If you do not know the three digit combination, it may be difficult to read.', '[title] Turn the dial at least three times to the right. [step] Turning the dial three full rotations clockwise will reset the numbers and allow you to input the first number in your combination.', \\\"It should be written on the padlock itself, either on the back or directly under the combination pattern. It's normal to have a few different combinations when purchased with a lock.\\\"]\"}","display_format":"text","language":"","answer_status":"published","assets":[],"source_url":"https://rowanzellers.com/hellaswag/","history":"initial import","indexing_mode":"index","subproblems":[],"grids":[]}