{"kind":"task","effective_mode":"full","benchmark":{"kind":"benchmark","effective_mode":"full","slug":"hellaswag","formal_name":"HellaSwag","introduction":"HellaSwag asks which of four continuations naturally follows a sentence or a video caption. The wrong options are produced by adversarial filtering, so items that are nearly trivial for people remain hard for models.","introduction_ja":"","introduction_en":"","category":"Category not supplied","task_count":null,"acquisition_status":"Acquisition status not supplied","official_url":"https://rowanzellers.com/hellaswag/","indexing_mode":"index","profile":{"resources":[],"task_format":"","scoring":"","metric":"","size":"","answer_access":"","license":"","citation":"","maintainer":"","released":"","why_hard":"","related":[],"history":"","construction":"","limitations":"","saturation":"","scores":[]}},"task_id":"80ac8dc7-daf0-520c-9c50-b7619fe61287","task_key":"default--validation--6506","task_revision_id":"1","upstream_id":"6506","short_description":"HellaSwag validation 6506","config":"default","split":"validation","body":"{\"activity_label\":\"Philosophy and Religion\",\"ctx\":\"[header] How to teach a sunday school [title] Get to know the children. [step] Take a few minutes at the beginning of each class to make name tags and play a small game where you point to someone, and they give their name, age, and a hobby or two. [title] Have your lesson plan handy.\",\"endings\":\"['[step] Make sure you have your curriculum notes. A calculator works well, but a printed one is best.', '[step] Try doodling, drawing, whistling, singing, etc. There are many great ways to familiarize the children with day to day life.', '[step] Go in order, but improvise if needed. Go with the flow and remember that these are just little kids, so anything could happen.', '[step] You want a cue word for the lesson, something that the child can repeat and then mimic. You might be teaching the children the word shuffle that really sets the tone and builds the mood.']\"}","display_format":"text","language":"","answer_status":"published","assets":[],"source_url":"https://rowanzellers.com/hellaswag/","history":"initial import","indexing_mode":"index","subproblems":[],"grids":[]}