{"kind":"task","effective_mode":"full","benchmark":{"kind":"benchmark","effective_mode":"full","slug":"hellaswag","formal_name":"HellaSwag","introduction":"HellaSwag asks which of four continuations naturally follows a sentence or a video caption. The wrong options are produced by adversarial filtering, so items that are nearly trivial for people remain hard for models.","introduction_ja":"","introduction_en":"","category":"Category not supplied","task_count":null,"acquisition_status":"Acquisition status not supplied","official_url":"https://rowanzellers.com/hellaswag/","indexing_mode":"index","profile":{"resources":[],"task_format":"","scoring":"","metric":"","size":"","answer_access":"","license":"","citation":"","maintainer":"","released":"","why_hard":"","related":[],"history":"","construction":"","limitations":"","saturation":"","scores":[]}},"task_id":"5a9e3396-bfb0-5b17-838e-6c263599dfdc","task_key":"default--validation--42136--variant-e117eb83a136","task_revision_id":"1","upstream_id":"42136--variant-e117eb83a136","short_description":"HellaSwag validation 42136","config":"default","split":"validation","body":"{\"activity_label\":\"Sports and Fitness\",\"ctx\":\"[header] How to pitch a basha (tarp) tent [title] Spread out the basha where you intend to pitch it up. [step] Figure out where the poles need to go. [title] Set up the two gadget poles and tie a string across.\",\"endings\":\"[\\\"[title] You'll need a different length of string (most will be 0.62 cm long). [title] If you're using fresh or compressed balloons, you'll use a variable tie knot [title] Cut around this.\\\", '[step] To set up the two gadget poles, you could either mallet them down and/or secure them to pegs pulling in opposite directions. You could set the string across higher up or lower down depending on how waterproof you want you tent to be and how easy easy it will be to move in and out of it.', '[title] Draw a line the distances from the bottom of the tent apart from the pole line. [step] Use edge marking next to side lines.', '[title] Make sure that the ends are horizontal. [step] This means they should be on the side opposite each other.']\"}","display_format":"text","language":"","answer_status":"published","assets":[],"source_url":"https://rowanzellers.com/hellaswag/","history":"initial import","indexing_mode":"index","subproblems":[],"grids":[]}