{"kind":"task","effective_mode":"full","benchmark":{"kind":"benchmark","effective_mode":"full","slug":"hellaswag","formal_name":"HellaSwag","introduction":"HellaSwag asks which of four continuations naturally follows a sentence or a video caption. The wrong options are produced by adversarial filtering, so items that are nearly trivial for people remain hard for models.","introduction_ja":"","introduction_en":"","category":"Category not supplied","task_count":null,"acquisition_status":"Acquisition status not supplied","official_url":"https://rowanzellers.com/hellaswag/","indexing_mode":"index","profile":{"resources":[],"task_format":"","scoring":"","metric":"","size":"","answer_access":"","license":"","citation":"","maintainer":"","released":"","why_hard":"","related":[],"history":"","construction":"","limitations":"","saturation":"","scores":[]}},"task_id":"c0f6c707-e2cd-5cd3-84d4-99a7ffed7665","task_key":"default--validation--42525","task_revision_id":"1","upstream_id":"42525","short_description":"HellaSwag validation 42525","config":"default","split":"validation","body":"{\"activity_label\":\"Food and Entertaining\",\"ctx\":\"[header] How to make tomato soup [title] Bring a large pot of water to a boil. [step] Fill the pot about halfway with water, then place it on a burner over high heat and allow it to come to a full boil. [title] Cut an \\\" x \\\" shape into the tomatoes.\",\"endings\":\"['[step] Use a sharp knife to cut an \\\" x \\\" on the tops or bottoms of the tomatoes. It\\\\'s not necessary to cut too deep-just pierce the skin.', '[step] About the diameter of a golf-ball. [title] Cook the tomato slices for 2-3 minutes or until cooked.', '[step] Use a sharp knife or a fork to cut the tomatoes lengthwise into thin pieces, about 1-inch (2.5-cm) long. The smaller pieces should start from the end of the \\\" x \\\" and go the other way.', '[step] Cut 8-12 slits in the tomato flesh about ¾ \\\" deep, about 1/2 \\\" (1.9 cm) apart. You want one of these slits raised on either side of the tomato.']\"}","display_format":"text","language":"","answer_status":"published","assets":[],"source_url":"https://rowanzellers.com/hellaswag/","history":"initial import","indexing_mode":"index","subproblems":[],"grids":[]}