{"kind":"task","effective_mode":"full","benchmark":{"kind":"benchmark","effective_mode":"full","slug":"hellaswag","formal_name":"HellaSwag","introduction":"HellaSwag asks which of four continuations naturally follows a sentence or a video caption. The wrong options are produced by adversarial filtering, so items that are nearly trivial for people remain hard for models.","introduction_ja":"","introduction_en":"","category":"Category not supplied","task_count":null,"acquisition_status":"Acquisition status not supplied","official_url":"https://rowanzellers.com/hellaswag/","indexing_mode":"index","profile":{"resources":[],"task_format":"","scoring":"","metric":"","size":"","answer_access":"","license":"","citation":"","maintainer":"","released":"","why_hard":"","related":[],"history":"","construction":"","limitations":"","saturation":"","scores":[]}},"task_id":"a82ddde4-b821-5b97-af77-8b428c8a168d","task_key":"default--validation--33566","task_revision_id":"1","upstream_id":"33566","short_description":"HellaSwag validation 33566","config":"default","split":"validation","body":"{\"activity_label\":\"Personal Care and Style\",\"ctx\":\"[header] How to score in curling [title] Know how teams are set up. [step] Each game of curling is played with two teams, and each team has four members. [substeps] Each member of the team will throw two stones per end, and each of those stones is played one at a time, not back-to-back.\",\"endings\":\"['A player who strikes first gets to claim points. The player who scored first does not directly draw points.', \\\"The team that's less likely to get those stones in the end will have fewer rounds. The team with the most stones wins the weighted penalty.\\\", 'The team who throws the biggest stones the farthest wins. [title] Understand the order in which each team stands.', 'The other players will help direct the stone toward the house. Players will determine their order of rotation before the game and follow that rotation throughout all the ends.']\"}","display_format":"text","language":"","answer_status":"published","assets":[],"source_url":"https://rowanzellers.com/hellaswag/","history":"initial import","indexing_mode":"index","subproblems":[],"grids":[]}