{"kind":"task","effective_mode":"full","benchmark":{"kind":"benchmark","effective_mode":"full","slug":"hellaswag","formal_name":"HellaSwag","introduction":"HellaSwag asks which of four continuations naturally follows a sentence or a video caption. The wrong options are produced by adversarial filtering, so items that are nearly trivial for people remain hard for models.","introduction_ja":"","introduction_en":"","category":"Category not supplied","task_count":null,"acquisition_status":"Acquisition status not supplied","official_url":"https://rowanzellers.com/hellaswag/","indexing_mode":"index","profile":{"resources":[],"task_format":"","scoring":"","metric":"","size":"","answer_access":"","license":"","citation":"","maintainer":"","released":"","why_hard":"","related":[],"history":"","construction":"","limitations":"","saturation":"","scores":[]}},"task_id":"1e6b1eac-cbd5-502e-916d-beb43bcc0964","task_key":"default--validation--7953","task_revision_id":"1","upstream_id":"7953","short_description":"HellaSwag validation 7953","config":"default","split":"validation","body":"{\"activity_label\":\"Pets and Animals\",\"ctx\":\"[header] How to trap a robin bird [title] Turn the lights off and close the blinds. [step] Quiet, dark locations will help calm the bird and make him easier to capture. If there are no blinds, wait until it is dark outside to make the robin easier to catch.\",\"endings\":\"['[title] Watch the bird for a few minutes to determine her flight path. [step] Robins like to perch on raised surfaces and move frequently.', \\\"[title] Fill up the cage with the right amount of early morning sun. [step] Remember to always hold the cage slightly in front of the lights so that the bird doesn't realize that it is being captured and won't accidentally squeeze in in.\\\", 'The birds will feel more comfortable doing this than they would if you opened the windows or closed curtains after they are caught. [title] Catch the robin using a net or fishing net.', \\\"[substeps] If you want to catch a robin with your closed blinds, don't turn on any lights in your home. The robin will find your bright lights frightening and may give a rabbit phobia.\\\"]\"}","display_format":"text","language":"","answer_status":"published","assets":[],"source_url":"https://rowanzellers.com/hellaswag/","history":"initial import","indexing_mode":"index","subproblems":[],"grids":[]}