{"kind":"task","effective_mode":"full","benchmark":{"kind":"benchmark","effective_mode":"full","slug":"hellaswag","formal_name":"HellaSwag","introduction":"HellaSwag asks which of four continuations naturally follows a sentence or a video caption. The wrong options are produced by adversarial filtering, so items that are nearly trivial for people remain hard for models.","introduction_ja":"","introduction_en":"","category":"Category not supplied","task_count":null,"acquisition_status":"Acquisition status not supplied","official_url":"https://rowanzellers.com/hellaswag/","indexing_mode":"index","profile":{"resources":[],"task_format":"","scoring":"","metric":"","size":"","answer_access":"","license":"","citation":"","maintainer":"","released":"","why_hard":"","related":[],"history":"","construction":"","limitations":"","saturation":"","scores":[]}},"task_id":"11483781-4ca9-5919-a5aa-5fa6b141cf7d","task_key":"default--validation--49373","task_revision_id":"1","upstream_id":"49373","short_description":"HellaSwag validation 49373","config":"default","split":"validation","body":"{\"activity_label\":\"Food and Entertaining\",\"ctx\":\"[header] How to make coconut chocolate milk [title] Heat your water. [step] Heat your water until it is either very hot or boiling. You can do this easily by heating the water in a pot on your stovetop or in an electric kettle.\",\"endings\":\"[\\\"If you're happy with the heat, the water will take you two to three minutes to turn off. [substeps] A good rule of thumb is to vary the water based on your preference.\\\", 'If you have access to a stove, put it on the stove so that it is hot, but not too hot. Make sure that it is kept on the stove and boiled for up to two hours.', 'Water that is slightly below boiling will better preserve the nutritional value of the coconut. [title] Prepare your strainer while the water heats.', 'Make sure that the water is not too hot. If it is too cold, it will build up and cause scalding.']\"}","display_format":"text","language":"","answer_status":"published","assets":[],"source_url":"https://rowanzellers.com/hellaswag/","history":"initial import","indexing_mode":"index","subproblems":[],"grids":[]}