{"kind":"task","effective_mode":"full","benchmark":{"kind":"benchmark","effective_mode":"full","slug":"hellaswag","formal_name":"HellaSwag","introduction":"HellaSwag asks which of four continuations naturally follows a sentence or a video caption. The wrong options are produced by adversarial filtering, so items that are nearly trivial for people remain hard for models.","introduction_ja":"","introduction_en":"","category":"Category not supplied","task_count":null,"acquisition_status":"Acquisition status not supplied","official_url":"https://rowanzellers.com/hellaswag/","indexing_mode":"index","profile":{"resources":[],"task_format":"","scoring":"","metric":"","size":"","answer_access":"","license":"","citation":"","maintainer":"","released":"","why_hard":"","related":[],"history":"","construction":"","limitations":"","saturation":"","scores":[]}},"task_id":"464aacad-4a7a-5572-be67-19d6d3587a43","task_key":"default--validation--22618","task_revision_id":"1","upstream_id":"22618","short_description":"HellaSwag validation 22618","config":"default","split":"validation","body":"{\"activity_label\":\"Mowing the lawn\",\"ctx\":\"He uses a ruler to show more detailed and then moves the mower to the sidewalk to talk some more. he\",\"endings\":\"['then moves to the corner to talk again.', \\\"measures the mower's horsepower and bends down to finish his section of sidewalk.\\\", 'continues to talk about what he is doing and shows two hands stopping to write something on the board from time to time.', 'starts mowing his lawn finally going up and down the lawn.']\"}","display_format":"text","language":"","answer_status":"published","assets":[],"source_url":"https://rowanzellers.com/hellaswag/","history":"initial import","indexing_mode":"index","subproblems":[],"grids":[]}