{"kind":"task","effective_mode":"full","benchmark":{"kind":"benchmark","effective_mode":"full","slug":"hellaswag","formal_name":"HellaSwag","introduction":"HellaSwag asks which of four continuations naturally follows a sentence or a video caption. The wrong options are produced by adversarial filtering, so items that are nearly trivial for people remain hard for models.","introduction_ja":"","introduction_en":"","category":"Category not supplied","task_count":null,"acquisition_status":"Acquisition status not supplied","official_url":"https://rowanzellers.com/hellaswag/","indexing_mode":"index","profile":{"resources":[],"task_format":"","scoring":"","metric":"","size":"","answer_access":"","license":"","citation":"","maintainer":"","released":"","why_hard":"","related":[],"history":"","construction":"","limitations":"","saturation":"","scores":[]}},"task_id":"e9434a1d-ed77-5a41-8660-f5e03035cb4f","task_key":"default--validation--6537","task_revision_id":"1","upstream_id":"6537","short_description":"HellaSwag validation 6537","config":"default","split":"validation","body":"{\"activity_label\":\"Food and Entertaining\",\"ctx\":\"[header] How to eat lobster [title] Decide between a hard shell or a \\\" shedder. [step] \\\" if you go to a restaurant that offers you the chance to choose your own lobster, you may be asked whether you want a hard shelled lobster or one that has recently shedded. [substeps] Hard shell lobsters have matured enough that their shells are a little tough to break.\",\"endings\":\"['You can choose between a hard shell lobster and a firmer shell. The two types of hard shelling are really the difference between either hard shell lobsters or softer, fresher lobsters.', 'The meat inside is firm and flavorful. Shedders have softer shells, since they have recently moulted.', \\\"The hard shell lobsters are usually larger and more firm then those that are still in the shell. Or, the hard shell lobsters are the longer lasting and sturdier, which means they aren't as expensive as the hard shells.\\\", 'Either option would require you to work with a lobster that is still quite small to handle. Hard shelled lobsters are easier to hold between your fingers than hard shells.']\"}","display_format":"text","language":"","answer_status":"published","assets":[],"source_url":"https://rowanzellers.com/hellaswag/","history":"initial import","indexing_mode":"index","subproblems":[],"grids":[]}