{"kind":"task","effective_mode":"full","benchmark":{"kind":"benchmark","effective_mode":"full","slug":"hellaswag","formal_name":"HellaSwag","introduction":"HellaSwag asks which of four continuations naturally follows a sentence or a video caption. The wrong options are produced by adversarial filtering, so items that are nearly trivial for people remain hard for models.","introduction_ja":"","introduction_en":"","category":"Category not supplied","task_count":null,"acquisition_status":"Acquisition status not supplied","official_url":"https://rowanzellers.com/hellaswag/","indexing_mode":"index","profile":{"resources":[],"task_format":"","scoring":"","metric":"","size":"","answer_access":"","license":"","citation":"","maintainer":"","released":"","why_hard":"","related":[],"history":"","construction":"","limitations":"","saturation":"","scores":[]}},"task_id":"fe1603cb-1555-5465-9555-379ad916972e","task_key":"default--validation--38184","task_revision_id":"1","upstream_id":"38184","short_description":"HellaSwag validation 38184","config":"default","split":"validation","body":"{\"activity_label\":\"Having an ice cream\",\"ctx\":\"A coffee mug sits on a counter then dry ingredients and water are added before the contents are stirred with a spoon. a graduated cup with milk\",\"endings\":\"['and sugar in it is now made to purify water and next poured into the pot to purify water, added salt, lots of water, and chopped chopped fruit juice in the cup.', 'has the mixture poured into it and is stirred.', 'sits on top of the a knife with a cup and proceeds to remove milk from the jug and pour it into the finished coffee in a shotput.', 'and sugar are added to the pot to form a gold color.']\"}","display_format":"text","language":"","answer_status":"published","assets":[],"source_url":"https://rowanzellers.com/hellaswag/","history":"initial import","indexing_mode":"index","subproblems":[],"grids":[]}