{"kind":"task","effective_mode":"full","benchmark":{"kind":"benchmark","effective_mode":"full","slug":"hellaswag","formal_name":"HellaSwag","introduction":"HellaSwag asks which of four continuations naturally follows a sentence or a video caption. The wrong options are produced by adversarial filtering, so items that are nearly trivial for people remain hard for models.","introduction_ja":"","introduction_en":"","category":"Category not supplied","task_count":null,"acquisition_status":"Acquisition status not supplied","official_url":"https://rowanzellers.com/hellaswag/","indexing_mode":"index","profile":{"resources":[],"task_format":"","scoring":"","metric":"","size":"","answer_access":"","license":"","citation":"","maintainer":"","released":"","why_hard":"","related":[],"history":"","construction":"","limitations":"","saturation":"","scores":[]}},"task_id":"93acc24a-34d0-543b-9490-de5895f09d8c","task_key":"default--validation--29379","task_revision_id":"1","upstream_id":"29379","short_description":"HellaSwag validation 29379","config":"default","split":"validation","body":"{\"activity_label\":\"Home and Garden\",\"ctx\":\"[header] How to prepare your home for the rainy season [title] Get your roof ready for the rains. [step] Inspect your roof twice per year to avoid costly problems that can escalate into tremendous cost. [substeps] Look for cracks along the ridge of your roof and where your shingles fold over to form the cap.\",\"endings\":\"[\\\"It's vital for your roof structure to be constructed appropriately for the upcoming rains. This can be accomplished with basic care, but should only be done once or twice a year, especially if there's a severe increase in rainfall.\\\", 'Inspect the valleys of your roof (the area of your roof with a downward slope). Make sure that the sheet metal flashing does not have any holes or rusty spots.', 'Make sure the roof is acrylic, known for its pointed corners and rough edges. [title] Make sure the water can drain from the eaves of your house.', 'Use the energy and materials that you have now taken in to remove plastic or stone caps that reinforce the roof (which can be put into reuse) or any other dramatic build-up. [title] Buy/rent or rent your house for the rainy season.']\"}","display_format":"text","language":"","answer_status":"published","assets":[],"source_url":"https://rowanzellers.com/hellaswag/","history":"initial import","indexing_mode":"index","subproblems":[],"grids":[]}