{"kind":"task","effective_mode":"full","benchmark":{"kind":"benchmark","effective_mode":"full","slug":"hellaswag","formal_name":"HellaSwag","introduction":"HellaSwag asks which of four continuations naturally follows a sentence or a video caption. The wrong options are produced by adversarial filtering, so items that are nearly trivial for people remain hard for models.","introduction_ja":"","introduction_en":"","category":"Category not supplied","task_count":null,"acquisition_status":"Acquisition status not supplied","official_url":"https://rowanzellers.com/hellaswag/","indexing_mode":"index","profile":{"resources":[],"task_format":"","scoring":"","metric":"","size":"","answer_access":"","license":"","citation":"","maintainer":"","released":"","why_hard":"","related":[],"history":"","construction":"","limitations":"","saturation":"","scores":[]}},"task_id":"095c87ce-6182-59a4-b382-79c8149bc870","task_key":"default--validation--42344","task_revision_id":"1","upstream_id":"42344","short_description":"HellaSwag validation 42344","config":"default","split":"validation","body":"{\"activity_label\":\"Finance and Business\",\"ctx\":\"[header] How to evaluate suppliers [title] Take note of the quality of the merchandise you have ordered from the company. [step] Do the items match the description listed in the wholesale catalogue? Were items ever misrepresented? [title] Recall whether you have ever been displeased with the merchandise sent by the supplier. [step] Also, write down when you were pleased with the merchandise.\",\"endings\":\"['Have you ever felt unsatisfied or for unreasonable reasons? [title] Monitor reviews. [step] Were you satisfied with the product information you received? You must remain calm and find joy in the results of your analysis.', 'Assign a number to the quality of the merchandise shipped. 1 for being poor quality and 10 for being excellent quality.', 'Were customers ever expected to buy the merchandise before it became an import? [substeps] Conduct an inventory. Determine if you have ever stopped shopping for any items from the supplier.', 'Who did the product send? Did members of the company handle the bulk of the selling? [title] Document any problems related to the merchandise. [step] Were customers kicked out, frustrated, overly emotional, ripped-off or sexually inappropriate.']\"}","display_format":"text","language":"","answer_status":"published","assets":[],"source_url":"https://rowanzellers.com/hellaswag/","history":"initial import","indexing_mode":"index","subproblems":[],"grids":[]}