{"kind":"task","effective_mode":"full","benchmark":{"kind":"benchmark","effective_mode":"full","slug":"hellaswag","formal_name":"HellaSwag","introduction":"HellaSwag asks which of four continuations naturally follows a sentence or a video caption. The wrong options are produced by adversarial filtering, so items that are nearly trivial for people remain hard for models.","introduction_ja":"","introduction_en":"","category":"Category not supplied","task_count":null,"acquisition_status":"Acquisition status not supplied","official_url":"https://rowanzellers.com/hellaswag/","indexing_mode":"index","profile":{"resources":[],"task_format":"","scoring":"","metric":"","size":"","answer_access":"","license":"","citation":"","maintainer":"","released":"","why_hard":"","related":[],"history":"","construction":"","limitations":"","saturation":"","scores":[]}},"task_id":"93c0ceba-c17d-583c-9d17-c05dcc38ca62","task_key":"default--validation--42442--variant-e2c650a44888","task_revision_id":"1","upstream_id":"42442--variant-e2c650a44888","short_description":"HellaSwag validation 42442","config":"default","split":"validation","body":"{\"activity_label\":\"Education and Communications\",\"ctx\":\"[header] How to add [title] Get comfortable with the concept of adding. [step] Take a handful of beans (or other small objects). Set one bean aside at a time into a pile and count each one as you go (1 , 2 , 3, etc).\",\"endings\":\"['Divide this pile up into columns to help add more dots at various points during the process. [substeps] Add the largest number from each column into a 0 and put it up again.', 'Stop after you have a few beans in the pile. How many did you put there? Write that number down.', \\\"Place the each pile onto your favorite plate. [substeps] If you've selected beans from your pantry, it will be pretty intuitive to put them into the pile, provided that they're reasonably digestible for eating.\\\", 'Do not fill each pile with potatoes or mushrooms. These shouldn\\\\'t be considered \\\" potatoes \\\" unless your crowd is small.']\"}","display_format":"text","language":"","answer_status":"published","assets":[],"source_url":"https://rowanzellers.com/hellaswag/","history":"initial import","indexing_mode":"index","subproblems":[],"grids":[]}