{"kind":"task","effective_mode":"full","benchmark":{"kind":"benchmark","effective_mode":"full","slug":"hellaswag","formal_name":"HellaSwag","introduction":"HellaSwag asks which of four continuations naturally follows a sentence or a video caption. The wrong options are produced by adversarial filtering, so items that are nearly trivial for people remain hard for models.","introduction_ja":"","introduction_en":"","category":"Category not supplied","task_count":null,"acquisition_status":"Acquisition status not supplied","official_url":"https://rowanzellers.com/hellaswag/","indexing_mode":"index","profile":{"resources":[],"task_format":"","scoring":"","metric":"","size":"","answer_access":"","license":"","citation":"","maintainer":"","released":"","why_hard":"","related":[],"history":"","construction":"","limitations":"","saturation":"","scores":[]}},"task_id":"92b3e3b9-e6cd-5558-9448-383ff97e80fb","task_key":"default--validation--12852","task_revision_id":"1","upstream_id":"12852","short_description":"HellaSwag validation 12852","config":"default","split":"validation","body":"{\"activity_label\":\"Home and Garden\",\"ctx\":\"[header] How to caulk a shower drain [title] Locate the screws on either side of the grid strainer above the drain. [title] Unscrew the grid drainer. [title] Inspect the grid drainer to see if it is clean.\",\"endings\":\"['[step] If not, remove both screws by rubbing them lightly over the part of the grid where you want to caulk. [title] Remove the bottom slat (s) that goes under the sides.', '[title] Loosen the screws by cutting them with a razor. [title] Apply a thin layer of spackle over the grid, allowing the spackle to dry before moving on.', '[step] If it is clean, you can set it aside for now. If it is covered in old caulk or debris, clean it before setting it aside for later.', '[title] Make sure there is no paint or debris on the grid, as it may affect the plumbing. [step] You can check for film, caked on debris, mold, wire fragments or any other damage.']\"}","display_format":"text","language":"","answer_status":"published","assets":[],"source_url":"https://rowanzellers.com/hellaswag/","history":"initial import","indexing_mode":"index","subproblems":[],"grids":[]}