{"kind":"task","effective_mode":"full","benchmark":{"kind":"benchmark","effective_mode":"full","slug":"hellaswag","formal_name":"HellaSwag","introduction":"HellaSwag asks which of four continuations naturally follows a sentence or a video caption. The wrong options are produced by adversarial filtering, so items that are nearly trivial for people remain hard for models.","introduction_ja":"","introduction_en":"","category":"Category not supplied","task_count":null,"acquisition_status":"Acquisition status not supplied","official_url":"https://rowanzellers.com/hellaswag/","indexing_mode":"index","profile":{"resources":[],"task_format":"","scoring":"","metric":"","size":"","answer_access":"","license":"","citation":"","maintainer":"","released":"","why_hard":"","related":[],"history":"","construction":"","limitations":"","saturation":"","scores":[]}},"task_id":"5f6e94c9-e8a8-5ae8-a6da-e724b3c5b1c7","task_key":"default--validation--35133","task_revision_id":"1","upstream_id":"35133","short_description":"HellaSwag validation 35133","config":"default","split":"validation","body":"{\"activity_label\":\"Philosophy and Religion\",\"ctx\":\"[header] How to start a church nursery [title] Start a committee of people interested in beginning a new church nursery. [step] Limit the maximum number of people for the committee to 8. Assign a committee chairperson.\",\"endings\":\"['[substeps] This committee will be responsible for making the decisions about the nursery and then presenting to the church council. The committee will need to present a budget for the nursery.', 'Leadership style and social habits tend to allow guilds more opportunities to play. [title] Ask about wedding themes in your space.', 'You might, for example, want a committee of 8 to 10 people. If you want to make the committee more person-like , [title] Write a blog.', \\\"[substeps] Have fun but don't recruit more than 1 committee. Don't let your friends have $10 , 000 to start a nursery.\\\"]\"}","display_format":"text","language":"","answer_status":"published","assets":[],"source_url":"https://rowanzellers.com/hellaswag/","history":"initial import","indexing_mode":"index","subproblems":[],"grids":[]}