{"kind":"task","effective_mode":"full","benchmark":{"kind":"benchmark","effective_mode":"full","slug":"hellaswag","formal_name":"HellaSwag","introduction":"HellaSwag asks which of four continuations naturally follows a sentence or a video caption. The wrong options are produced by adversarial filtering, so items that are nearly trivial for people remain hard for models.","introduction_ja":"","introduction_en":"","category":"Category not supplied","task_count":null,"acquisition_status":"Acquisition status not supplied","official_url":"https://rowanzellers.com/hellaswag/","indexing_mode":"index","profile":{"resources":[],"task_format":"","scoring":"","metric":"","size":"","answer_access":"","license":"","citation":"","maintainer":"","released":"","why_hard":"","related":[],"history":"","construction":"","limitations":"","saturation":"","scores":[]}},"task_id":"87b975eb-55ed-54e7-a306-57dc1fe5afb0","task_key":"default--validation--21389--variant-d30a5d2e4c8e","task_revision_id":"1","upstream_id":"21389--variant-d30a5d2e4c8e","short_description":"HellaSwag validation 21389","config":"default","split":"validation","body":"{\"activity_label\":\"Computers and Electronics\",\"ctx\":\"[header] How to use the mirror and pattern tools in siemans nx [title] Create a new document in nx. [substeps] When in the nx home screen, select \\\" new \\\" in the top left corner of the page. Once selected, a screen will appear.\",\"endings\":\"['From the \\\" create new document \\\" screen, select \\\" new. \\\" alternatively, change the color scale by sliding the \\\" color \\\" slider on the \\\" select \\\" screen.', '[title] Recognize what you want to use the mirror for. [step] Select the \\\" save as, \\\" or \\\" publish text as custom from this window \\\" box, to save your work.', 'Select \\\" model \\\" and select your desired unit of measurement. If you are following along with this example, select inches.', '[title] Create a new document in object settings. [step] Fill in the background of the screen before entering the appearance menu.']\"}","display_format":"text","language":"","answer_status":"published","assets":[],"source_url":"https://rowanzellers.com/hellaswag/","history":"initial import","indexing_mode":"index","subproblems":[],"grids":[]}