{"kind":"task","effective_mode":"full","benchmark":{"kind":"benchmark","effective_mode":"full","slug":"hellaswag","formal_name":"HellaSwag","introduction":"HellaSwag asks which of four continuations naturally follows a sentence or a video caption. The wrong options are produced by adversarial filtering, so items that are nearly trivial for people remain hard for models.","introduction_ja":"","introduction_en":"","category":"Category not supplied","task_count":null,"acquisition_status":"Acquisition status not supplied","official_url":"https://rowanzellers.com/hellaswag/","indexing_mode":"index","profile":{"resources":[],"task_format":"","scoring":"","metric":"","size":"","answer_access":"","license":"","citation":"","maintainer":"","released":"","why_hard":"","related":[],"history":"","construction":"","limitations":"","saturation":"","scores":[]}},"task_id":"6e0755dd-5ab9-5c92-b4d2-3a2ede3cda67","task_key":"default--validation--48639--variant-44728973307c","task_revision_id":"1","upstream_id":"48639--variant-44728973307c","short_description":"HellaSwag validation 48639","config":"default","split":"validation","body":"{\"activity_label\":\"Family Life\",\"ctx\":\"[header] How to throw a baby shower [title] Make the guest list. [step] Who's coming to the shower? Invite the mother-to-be's mom, aunts, sisters, cousins, and her best friends. Once you figure out the final count, you'll be able to determine how many invitations to buy, the quantity of food you'll need, and what space will be big enough to accommodate everyone.\",\"endings\":\"['If you have two parents, invite both to chaperone. [title] Have the guest list ready to customize.', \\\"[substeps] Make sure to pick 5 people to invite. If the guest list isn't wide enough for everyone attending, try inviting more as well.\\\", '[title] Donate to charity or be a giant donor to raise money. [step] In many cases, pregnancy and baby-related fundraising events will require little extra to contribute.', \\\"You want to be organized for the event. [substeps] If you don't already have the guests' addresses, email them to ask.\\\"]\"}","display_format":"text","language":"","answer_status":"published","assets":[],"source_url":"https://rowanzellers.com/hellaswag/","history":"initial import","indexing_mode":"index","subproblems":[],"grids":[]}