{"kind":"task","effective_mode":"full","benchmark":{"kind":"benchmark","effective_mode":"full","slug":"hellaswag","formal_name":"HellaSwag","introduction":"HellaSwag asks which of four continuations naturally follows a sentence or a video caption. The wrong options are produced by adversarial filtering, so items that are nearly trivial for people remain hard for models.","introduction_ja":"","introduction_en":"","category":"Category not supplied","task_count":null,"acquisition_status":"Acquisition status not supplied","official_url":"https://rowanzellers.com/hellaswag/","indexing_mode":"index","profile":{"resources":[],"task_format":"","scoring":"","metric":"","size":"","answer_access":"","license":"","citation":"","maintainer":"","released":"","why_hard":"","related":[],"history":"","construction":"","limitations":"","saturation":"","scores":[]}},"task_id":"392b6aa3-9d67-5a43-9260-81bd0a940f84","task_key":"default--validation--41521","task_revision_id":"1","upstream_id":"41521","short_description":"HellaSwag validation 41521","config":"default","split":"validation","body":"{\"activity_label\":\"Finance and Business\",\"ctx\":\"[header] How to write an estimate [title] Review what needs to be done. [step] Before you can provide a customer with either an estimate or a quotation, you need to know exactly what they want you to do. Expect to ask a lot of questions, as people often can be vague about their needs.\",\"endings\":\"[\\\"Ask them what they want based off of facts, the strength of their argument (if relevant), and about the potential benefit or loss of the proposition. [substeps] Assess your company's needs based on that.\\\", \\\"[substeps] Keep in mind that sometimes a customer won't actually know what goes into a job-they just know what they want the final outcome to be. In many professions, this means you must actually visit the customer's work site to evaluate the job properly.\\\", 'This is a great start, but then do your homework. Ask what you can reasonably do, and get a certain amount of wiggle room.', 'This is especially true for works involving community service or inventory management. For example, if your company has a fundraiser for a local community center, for example, you may want to provide the following information.']\"}","display_format":"text","language":"","answer_status":"published","assets":[],"source_url":"https://rowanzellers.com/hellaswag/","history":"initial import","indexing_mode":"index","subproblems":[],"grids":[]}