{"kind":"task","effective_mode":"full","benchmark":{"kind":"benchmark","effective_mode":"full","slug":"hellaswag","formal_name":"HellaSwag","introduction":"HellaSwag asks which of four continuations naturally follows a sentence or a video caption. The wrong options are produced by adversarial filtering, so items that are nearly trivial for people remain hard for models.","introduction_ja":"","introduction_en":"","category":"Category not supplied","task_count":null,"acquisition_status":"Acquisition status not supplied","official_url":"https://rowanzellers.com/hellaswag/","indexing_mode":"index","profile":{"resources":[],"task_format":"","scoring":"","metric":"","size":"","answer_access":"","license":"","citation":"","maintainer":"","released":"","why_hard":"","related":[],"history":"","construction":"","limitations":"","saturation":"","scores":[]}},"task_id":"0c18eebf-067a-5864-9f40-7be2ef424d09","task_key":"default--validation--13371","task_revision_id":"1","upstream_id":"13371","short_description":"HellaSwag validation 13371","config":"default","split":"validation","body":"{\"activity_label\":\"Family Life\",\"ctx\":\"[header] How to put a friend or relative out of your house [title] Determine why you want them to leave. [step] You need to be clear with your own reasoning before diving into the conversation. Review any agreements you made when they moved in, or any promises that have been made/broken.\",\"endings\":\"[\\\"Ask yourself why you want them to stay. [substeps] Resolving the issue isn't easy.\\\", 'Assess the situation and their current behavior, grounding your reasoning in fact. While \\\" i don\\\\'t like living with them \\\" is an acceptable reason to ask someone to move, you want concrete details, like \\\" they never do the dishes, \\\" \\\" they said they would leave months ago, \\\" etc.', \\\"[substeps] Think carefully about whether or not you want them to stop staying at your house. You don't always have to grant your friend or relative permission.\\\", '[substeps] Think about asking a favor or requesting that someone stay. Nothing is more amusing than sitting down with them (and maintaining eye contact) and asking them a favor that is completely unsolicited without talking to them first.']\"}","display_format":"text","language":"","answer_status":"published","assets":[],"source_url":"https://rowanzellers.com/hellaswag/","history":"initial import","indexing_mode":"index","subproblems":[],"grids":[]}