{"kind":"task","effective_mode":"full","benchmark":{"kind":"benchmark","effective_mode":"full","slug":"hellaswag","formal_name":"HellaSwag","introduction":"HellaSwag asks which of four continuations naturally follows a sentence or a video caption. The wrong options are produced by adversarial filtering, so items that are nearly trivial for people remain hard for models.","introduction_ja":"","introduction_en":"","category":"Category not supplied","task_count":null,"acquisition_status":"Acquisition status not supplied","official_url":"https://rowanzellers.com/hellaswag/","indexing_mode":"index","profile":{"resources":[],"task_format":"","scoring":"","metric":"","size":"","answer_access":"","license":"","citation":"","maintainer":"","released":"","why_hard":"","related":[],"history":"","construction":"","limitations":"","saturation":"","scores":[]}},"task_id":"30ea2e4d-510a-5a00-bda2-02b8ac2fb70e","task_key":"default--validation--49586","task_revision_id":"1","upstream_id":"49586","short_description":"HellaSwag validation 49586","config":"default","split":"validation","body":"{\"activity_label\":\"Pets and Animals\",\"ctx\":\"[header] How to keep rabbit bottles from freezing [title] Fill the water bottle with warm water. [step] Do not use scalding hot water. This will cause it to take it longer for the water to freeze.\",\"endings\":\"['[substeps] Warm the water bottle in the microwave for 30 seconds at 75 degrees fahrenheit (75 degrees celsius) to thaw the water. Cold water can also kill rabbits with lesions such as stingers.', 'You can use either filtered or distilled water. Use the method described above if your rabbit will drink cooler water.', 'However, you will probably still need to change the water several times during the course of the day. [substeps] When doing this touch the stream of water with your hand.', 'Use enough water to cover the entire bottle. You should water only at below boiling point.']\"}","display_format":"text","language":"","answer_status":"published","assets":[],"source_url":"https://rowanzellers.com/hellaswag/","history":"initial import","indexing_mode":"index","subproblems":[],"grids":[]}