{"kind":"task","effective_mode":"full","benchmark":{"kind":"benchmark","effective_mode":"full","slug":"longbench-v2","formal_name":"LongBench v2","introduction":"LongBench v2 evaluates deep understanding and reasoning over long contexts through multiple-choice questions. Its official description lists 503 questions spanning tasks such as single-document and multi-document QA and code-repository understanding.","introduction_ja":"","introduction_en":"","category":"Category not supplied","task_count":null,"acquisition_status":"Acquisition status not supplied","official_url":"https://huggingface.co/datasets/zai-org/LongBench-v2","indexing_mode":"noindex","profile":{"resources":[],"task_format":"","scoring":"","metric":"","size":"","answer_access":"","license":"","citation":"","maintainer":"","released":"","why_hard":"","related":[]}},"task_id":"ff11c3ef-1a35-5af8-b2fe-d1fde867d948","task_key":"train--66ec0c4c821e116aacb1994a","task_revision_id":"3","upstream_id":"66ec0c4c821e116aacb1994a","short_description":"Which of the following statements is correct?","config":"","split":"train","body":"{\"choice_A\":\"Both contractor data and data crawled from the Internet are used to train VPT agents to model state-action pairs.\",\"choice_B\":\"All machine learning methods involved in the two articles are related to neural network deep learning.\",\"choice_C\":\"Both voyager and VPT control Minecraft agents by predicting the actions of simulated mouse and keyboard operations in each given state.\",\"choice_D\":\"VPT's modeling of action space is approximate rather than precise.\",\"context\":\"Video PreTraining (VPT): Learning to Act by\\nWatching Unlabeled Online Videos\\nAbstract\\nPretraining on noisy, internet-scale datasets has been heavily studied as a technique\\nfor training models with broad, general capabilities for text, images, and other\\nmodalities.1–6 However, for many sequential decision domains such as robotics,\\nvideo games, and computer use, publicly available data does not contain the labels\\nrequired to train behavioral priors in the same way. We extend the internet-scale\\npretraining paradigm to sequential decision domains through semi-supervised\\nimitation learning wherein agents learn to act by watching online unlabeled videos.\\nSpecifically, we show that with a small amount of labeled data we can train an\\ninverse dynamics model accurate enough to label a huge unlabeled source of online\\ndata – here, online videos of people playing Minecraft – from which we can then\\ntrain a general behavioral prior. Despite using the native human interface (mouse\\nand keyboard at 20Hz), we show that this behavioral prior has nontrivial zero-\\nshot capabilities and that it can be fine-tuned, with both imitation learning and\\nreinforcement learning, to hard-exploration tasks that are impossible to learn from\\nscratch via reinforcement learning. For many tasks our models exhibit human-\\nlevel performance, and we are the first to report computer agents that can craft\\ndiamond tools, which can take proficient humans upwards of 20 minutes (24,000\\nenvironment actions) of gameplay to accomplish.\\n1\\nIntroduction\\nWork in recent years has demonstrated the efficacy of pretraining large and general foundation\\nmodels7 on noisy internet-scale datasets for use in downstream tasks in natural language1–4 and\\ncomputer vision.5,6,8 For sequential decision domains (e.g. robotics, game playing, and computer\\nusage) where agents must repeatedly act within an environment, a wealth of data also exists on the\\nweb; however, most of this data is in the form of unlabeled video (i.e. without the actions taken\\nat each frame), making it much less straightforward to train a behavioral prior in these domains\\nthan it is in e.g. natural language. In a few rare settings, such as Chess, Go, and StarCraft, there\\n∗This was a large effort by a dedicated team. Each author made huge contributions on many fronts over long\\ntime periods. All members were full time on the project for over six months. BB, IA, PZ, and JC were on the\\noriginal VPT project team and were thus involved for even longer (over a year). Aside from those original team\\nmembers, author order is random. It was also randomized between IA and PZ.\\n†OpenAI\\n‡University of British Columbia\\narXiv:2206.11795v1  [cs.LG]  23 Jun 2022\\n\\n\\nalready exist large datasets with action labels from various online platforms that researchers have\\nused for imitation learning.9,10 When large labeled datasets do not exist, the canonical strategy\\nfor training capable agents is reinforcement learning (RL),11 which can be sample inefficient and\\nexpensive for hard-exploration problems.12–18 Many virtual tasks, e.g. navigating websites, using\\nPhotoshop, booking flights, etc., can be very hard to learn with RL and do not have large, commonly\\navailable sources of labeled data.19,20 In this paper, we seek to extend the paradigm of training\\nlarge, general-purpose foundation models to sequential decision domains by utilizing freely available\\ninternet-scale unlabeled video datasets with a simple semi-supervised imitation learning method. We\\ncall this method Video PreTraining (VPT) and demonstrate its efficacy in the domain of Minecraft.\\nExisting semi-supervised imitation learning methods aim to learn with few or no explicit action labels;\\nhowever, they generally rely on the policy’s ability to explore the environment throughout training,\\nmaking them susceptible to exploration bottlenecks.21–25 Furthermore, most prior semi-supervised\\nimitation learning work was tested in the relatively low data regime; because we experiment with far\\nmore data (∼70k hours of unlabeled video), we hypothesize that we can achieve good performance\\nwith a much simpler method, a trend that has proven true for pretraining in other modalities such\\nas text.1 In particular, given a large but unlabeled dataset, we propose generating pseudo-labels by\\ngathering a small amount of labeled data to train an inverse dynamics model (IDM) that predicts\\nthe action taken at each timestep in a video. Behavioral cloning (BC) can require a large amount\\nof data because the model must learn to infer intent and the distribution over future behaviors from\\nonly past observations. In contrast, the inverse dynamics modeling task is simpler because it is\\nnon-causal, meaning it can look at both past and future frames to infer actions. In most settings,\\nenvironment mechanics are far simpler than the breadth of human behavior that can take place within\\nthe environment, suggesting that non-causal IDMs could require far less data to train than causal BC\\nmodels. Using pseudo-labels generated from the IDM, we then train a model to mimic the distribution\\nof behavior in the previously unlabeled dataset with standard behavioral cloning at scale, which does\\nnot require any model rollouts and thus does not suffer from any potential exploration bottlenecks\\nin the environment. Finally, we show we can fine-tune this model to downstream tasks with either\\nbehavioral cloning or reinforcement learning.\\nFigure 1:\\nExample Minecraft\\ncrafting GUI. Agents use the\\nmouse and keyboard to navigate\\nmenus and drag and drop items.\\nWe chose to test our method in Minecraft because (a) it is one\\nof the most actively played games in the world26 and thus has\\na wealth of commonly available video data online, (b) it is a\\nfairly open-ended sandbox game with an extremely wide variety\\nof potential things to do, build, and collect, making our results\\nmore applicable to real-world applications such as computer us-\\nage, which also tends to be varied and open-ended, and (c) it\\nhas already garnered interest by the RL community as a research\\ndomain due to its complexity and correspondingly difficult ex-\\nploration challenges.27–31 In this work we use the native human\\ninterface for Minecraft so that we can (1) most accurately model\\nthe human behavior distribution and reduce domain shift between\\nvideo data and the environment, (2) make data collection easier by allowing our human contractors to\\nplay the game without modification, and (3) eliminate the need to hand-engineer a custom interface\\nfor models to interact with the environment. This choice means that our models play at 20 frames\\nper second and must use a mouse and keyboard interface to interact with human GUIs for crafting,\\nsmelting, trading, etc., including dragging items to specific slots or navigating the recipe book with\\nthe mouse cursor (Fig. 1). Compared to prior work in Minecraft that uses a lower frame rate and\\nconstructs crafting and attacking macros,30,32–34 using the native human interface drastically increases\\nthe environment’s exploration difficulty, making most simple tasks near impossible with RL from\\nscratch. Even the simple task of gathering a single wooden log while already facing a tree takes 60\\nconsecutive attack actions with the human interface, meaning the chance for a naive random policy to\\nsucceed is 1\\n2\\n60. While this paper shows results in Minecraft only, the VPT method is general and\\ncould be applied to any domain.\\nIn Section 4 we show that the VPT foundation model has nontrivial zero-shot performance, accom-\\nplishing tasks impossible to learn with RL alone, such as crafting planks and crafting tables (tasks\\nrequiring a human proficient in Minecraft a median of 50 seconds or ∼970 consecutive actions).\\nThrough fine-tuning with behavioral cloning to smaller datasets that target more specific behavior\\ndistributions, our agent is able to push even further into the technology tree, crafting stone tools\\n2\\n\\n\\n(taking a human a median of 2.3 minutes or ∼2790 actions). Finally, fine-tuning via RL produces\\nthe most dramatic improvements: our agent is able to craft diamond tools, an unprecedented result\\nin Minecraft made even more challenging by using the native human interface. This task requires\\na proficient human a median upwards of 20 minutes or ∼24000 actions. The main contributions\\nof this work are (1) we are the first to show promising results applying semi-supervised imitation\\nlearning to extremely large, noisy, and freely available video datasets for sequential decision domains,\\n(2) we show that such pretraining plus fine-tuning enables agents to solve tasks that were otherwise\\nimpossible to learn, (3) we show that labeled contractor data is far more efficiently used within\\nthe VPT method than it would be by directly training a foundation model from it and (4) we open\\nsource our contractor data, trained model weights, and Minecraft environment for future research\\ninto learning to act via semi-supervised imitation learning at scale.\\n2\\nPreliminaries and Related Work\\nImitation learning methods35–38 seek to construct a policy that accurately models the distribution of\\nbehavior in some dataset D = {(oi, ai)}, i ∈{1...N} of action-observation pairs. In order to roll\\nout these policies in an environment, they must be causal, meaning they condition on observations\\nfrom the current timestep t and past timesteps only, i.e. π ∼p(at|o1...ot). Imitation learning is\\nsimplest when demonstrations are labeled with corresponding actions. Imitating labeled trajectories\\nhas seen success in aerial vehicles,39,40 self-driving cars,41,42 board games,9,43 and video games.10,44\\nWhen labeled demonstrations are not available, standard behavioral cloning will not work; however,\\nthere is a large body of work in imitating behavior from unlabeled demonstrations.22 For instance,\\nGAIL23 constructs an adversarial objective incentivizing the trained policy to exhibit behaviors\\nindistinguishable from those in the target dataset. Edwards et al. 45 propose to first learn a latent\\npolicy using unlabeled demonstrations and then map the learned latent actions to real actions with\\na small amount of environment interaction. Peng et al. 46 first use motion-capture methods to track\\nagent positions in videos and then train RL agents to match these waypoints. Similarly, Behbahani\\net al. 47 and Aytar et al. 48 task a RL agent to match waypoints; however, they construct waypoints that\\nare embeddings from unsupervised feature learning models. Pathak et al. 49 and Nair et al. 50 train\\ngoal conditioned policies to take actions that advance the current state towards expert-provided goal\\nstates expressed as high dimensional visual waypoints. Most similar to our own work, Torabi et al. 24\\nsimultaneously train (1) an inverse dynamics model (IDM),51 which aims to uncover the underlying\\naction between timesteps given observations of past and future timesteps, e.g. pIDM(at|ot, ot+1), and\\n(2) a behavioral cloning (BC) model on trajectories of observations labeled with the IDM. Data to\\ntrain the IDM is collected by rolling out the BC model in the target environment such that both\\nmodels improve in tandem. However, at any point in training if there are sequences in the dataset that\\nthe IDM performs poorly on, it requires that the BC model perform those or similar sequences in\\norder for the IDM to improve and correctly label them. Therefore, if the BC model does not explore\\nefficiently, it could severely slow down learning. In order to avoid this potential issue we opted for a\\nsimpler two-stage approach: we first train an IDM on a small number of labeled trajectories collected\\nfrom human contractors (they play the game as would normally as we record their keypresses and\\nmouse movements). Because human contractors reach most relevant parts of the state space, we can\\nhold the IDM fixed throughout BC training.\\nCompared to most previous work in semi-supervised imitation learning, we experiment in the much\\nmore complex and open-ended environment of Minecraft. Minecraft is a voxel-based 3D video\\ngame that, due its popularity and wide variety of mechanics, has attracted a vast amount of RL\\nresearch.27,28,30–34,52–60 A large body of work focuses on small, custom-made Minecraft worlds\\nwith tasks such as navigation,53,60 block placing,54,55 instruction following,58,59 combat,56 and\\nothers.28,31,57 Work operating in the massive, randomly generated environments of Minecraft itself\\nhas included hill climbing,52 automated curriculum learning30 and, most closely related to the RL\\nexperiments presented in Sec. 4.4, diamond mining.27,32–34 However, to the best of our knowledge,\\nthere is no published work that operates in the full, unmodified human action space, which includes\\ndrag-and-drop inventory management and item crafting.\\n3\\n\\n\\nCollecting “Clean” Data\\nTraining the VPT Foundation Model\\nvia Behavioral Cloning\\nTraining the Inverse Dynamics Model (IDM)\\n~270k hours\\nunlabeled\\nvideo\\n~70k hours\\nunlabeled\\nvideo\\n~2k hours\\nvideo\\nlabeled with\\nactions\\nFilter for “clean”\\nvideo segments\\nSearch for relevant\\nMinecraft videos\\nvia keywords\\nContractors\\ncollect data \\nLabel videos\\nwith IDM \\n~70k hours\\nvideo\\nIDM-labeled\\nwith actions\\nTrain non-causal IDM\\nTrain causal\\nVPT Foundation Model\\na\\nd\\nspace\\nw\\na\\nd\\nspace\\nw\\nFigure 2: Video Pretraining (VPT) Method Overview.\\n3\\nMethods\\nInverse Dynamics Models (IDM)\\nVPT, illustrated in Figure 2, requires we first collect a small\\namount of labeled contractor data with which to train an inverse dynamics model pIDM(at|o1...T ),\\nwhich seeks to minimize the negative log-likelihood of an action at timestep t given a trajectory of T\\nobservations ot : t ∈[1...T]. In contrast to an imitation learning policy, the IDM can be non-causal,\\nmeaning its prediction for at can be a function of both past and future events, i.e. ot′>t. Compared to\\nthe behavioral cloning objective of modeling the distribution of human intent given past frames only,\\nwe hypothesize that inverting environment dynamics is easier and more data efficient to learn. Indeed,\\nSec. 4.1 will show that the IDM objective is much easier to learn, and furthermore Sec. 4.6 will show\\nthat with very little labeled data (as few as 100 hours) we can train a fairly accurate IDM. This IDM\\ncan be used to label online videos, providing the large amount of data required for the harder task of\\nbehavioral cloning. See appendices D and B for IDM training and data collection details.\\nData Filtering\\nWe gather a large dataset of Minecraft videos by searching the web for related\\nkeywords (Appendix A). Online videos often (1) include overlaid artifacts, such as a video feed\\nof the player’s face, channel logos, watermarks, etc., (2) are collected from platforms other than\\na computer with different gameplay, or (3) are from different game modes, e.g. in Minecraft we\\nonly want \\\"survival mode\\\" where players start from scratch and must gather or craft all their items.\\nWe call data “clean” if it does not contain visual artifacts and is from survival mode, and call all\\nother data “unclean.” With enough data, a large enough model, and enough training compute, a BC\\nmodel trained on both unclean and clean videos would likely still perform well in a clean Minecraft\\nenvironment. However, for simplicity and training compute efficiency, we choose to filter out unclean\\nsegments of video (note that a video may contain both clean and unclean segments). We do this by\\ntraining a model to filter out unclean segments using a small dataset (8800) of images sampled from\\nonline videos labeled by contractors as clean or unclean (Appendix A.2).\\nVPT Foundation Model\\nWe train a foundation model with standard behavioral cloning, i.e. mini-\\nmizing the negative log-likelihood of actions predicted by the IDM on clean data. For a particular\\ntrajectory of length T we minimize\\nmin\\nθ\\nX\\nt∈[1...T ]\\n−log πθ(at|o1, . . . , ot), where at ∼pIDM(at|o1, . . . , ot, . . . , oT )\\n(1)\\nAs we will see in the following sections, this model exhibits nontrivial zero-shot behavior and can be\\nfine-tuned with both imitation learning and RL to perform even more complex skills.\\n4\\nResults\\n4.1\\nPerformance of the Inverse Dynamics Model\\nThe IDM architecture is comprised primarily of a temporal convolution layer, a ResNet62 image\\nprocessing stack, and residual unmasked attention layers, from which the IDM simultaneously\\npredicts keypresses and mouse movements (see Appendix D for IDM architecture and training\\ndetails). A key hypothesis behind our work is that IDMs can be trained with a relatively small amount\\nof labeled data. While more data improves both mouse movement and keypress predictions, our best\\n4\\n\\n\\nFigure 3: (Left) IDM keypress accuracy and mouse movement R2 (explained variance61) as a\\nfunction of dataset size. (Right) IDM vs. behavioral cloning data efficiency.\\nIDM trains on only 1962 hours of data (compared to the ∼70k hours of clean data we collected from\\nthe internet) and achieves 90.6% keypress accuracy and a 0.97 R2 for mouse movements evaluated\\non a held-out validation set of contractor-labeled data (Figure 3 left).\\nFigure 3 (right) validates our hypothesis that IDMs are far more data efficient than BC models, likely\\nbecause inverting environment mechanics is far easier than modeling the entire distribution of human\\nbehavior. The IDM is two orders of magnitude more data efficient than a BC model trained on the\\nsame data and improves more quickly with more data. This evidence supports the hypothesis that it is\\nmore effective to use contractor data within the VPT pipeline by training an IDM than it is to train a\\nfoundation model from contractor data directly (Sections 4.5 and 4.6 provide additional evidence).\\n4.2\\nVPT Foundation Model Training and Zero-Shot Performance\\nFigure 4: (Left) Training and validation loss on the web_clean internet dataset with IDM pseudo-\\nlabels, and loss on the main IDM contractor dataset, which has ground-truth labels but is out-of-\\ndistribution (see text). (Right) Amount a given item was collected per episode averaged over 2500\\n60-minute survival episodes as a function of training epoch, shaded with the standard error of the\\nmean. Basic mining refers to collection of dirt, gravel, or sand (all materials that can be gathered\\nwithout tools). Logs are obtained by repeatedly hitting trees for three seconds, a difficult feat for an\\nRL agent to achieve as we show in Sec. 4.4. Planks can be crafted from logs, and crafting tables\\ncrafted from planks. Crafting requires using in-game crafting GUIs, and proficient humans take a\\nmedian of 50 seconds (970 consecutive actions) to make a crafting table.\\nWe now explore the emergent behavior learned by a behavioral cloning policy trained on an extremely\\nlarge, but noisy, internet dataset labeled with our IDM. To collect the unlabeled internet dataset,\\nwe searched for publicly available videos of Minecraft play with search terms such as “minecraft\\nsurvival for beginners.” These searches resulted in ∼270k hours of video, which we filtered down to\\n“clean” video segments yielding an unlabeled dataset of ∼70k hours, which we refer to as web_clean\\n(Appendix A has further details on data scraping and filtering). We then generated pseudo-labels\\nfor web_clean with our best IDM (Section 3) and then trained the VPT foundation model with\\nbehavioral cloning. Preliminary experiments suggested that our model could benefit from 30 epochs\\nof training and that a 0.5 billion parameter model was required to stay in the efficient learning\\nregime63 for that training duration (Appendix H), which took ∼9 days on 720 V100 GPUs.\\nWe evaluate our models by measuring validation loss (Fig. 4, left) and rolling them out in the\\nMinecraft environment. Unless otherwise noted, in all environment evaluations we spawn agents in a\\nstandard survival mode game where they play for 60 minutes, i.e. 72000 consecutive actions, and we\\nplot the mean and shade the standard error of the mean for various game statistics such as crafting\\nand collection rates (Fig. 4, right). The VPT foundation model quickly learns to chop down trees\\nto collect logs, a task we found near impossible for an RL agent to achieve with the native human\\ninterface (Sec. 4.4). It also learns to craft those logs into wooden planks and then use those planks\\n5\\n\\n\\nto craft a crafting table, which are required to unlock most other technology in the game and take a\\nhuman proficient in Minecraft approximately 50 seconds (970 consecutive actions) to collect. While\\nthese behaviors are fairly complex in the native human action space, the VPT foundation model crafts\\nthese items at a rate far below that of our proficient contractors, e.g. on average our contractors craft\\n5.44 crafting tables in 60 minutes of play versus 0.19 for the foundation model. The model also crafts\\na non-negligible amount of wooden sticks, which are required to make wooden tools; collects various\\nflowers and crafts dyes from them; kills zombies that appear during the night; hunts wild animals;\\ncollects various berries and mushrooms and eats them; and finds game-generated villages from which\\nto collect various rare items from chests. The model also learned to navigate uneven terrain, swim,\\nand pillar jump, which involves the agent repeatedly jumping and quickly placing a block below itself\\nsuch that it climbs upward by making a pillar.(iv)\\nWhile training and validation loss decrease healthily over training (Fig. 4, left), loss on our contractor\\ndataset (which the VPT model does not train on) begins increasing after 7 epochs. Contractor data\\ncould be out-of-distribution because our contractors may have a different distribution of play or\\nbecause there is some impactful visual domain shift compared to videos from the web. While one\\ncould have expected this would be predictive of declining evaluation performance, we do not see\\nnotable game statistics from the VPT foundation model rollouts (Figure 4, right) decrease over\\ntraining, and in the next section we show that BC fine-tuning performance continually improves as the\\nVPT foundation model trains. We provide more insight into this curious phenomenon in Appendix H.\\n4.3\\nFine-Tuning with Behavioral Cloning\\nFoundation models are designed to have a broad behavior profile and be generally capable across a\\nwide variety of tasks. To incorporate new knowledge or allow them to specialize on a narrower task\\ndistribution, it is common practice to fine-tune these models to smaller, more specific datasets.1 The\\nVPT foundation model trained on the broad web_clean dataset had nontrivial zero-shot performance;\\nit was able to craft a crafting table yet unable to go past this in the technology tree. As a case\\nstudy into BC fine-tuning, we attempt to improve the VPT foundation model’s ability to collect\\nand craft these “early game” items by fine-tuning to two narrower datasets targeted at Minecraft\\nbehavior within the first few minutes of players starting in a fresh world. In the first dataset,\\ncontractor_house, contractors have 10 minutes to build a basic house from scratch using primarily\\nwood, sand, and dirt. Collecting contractor data can be difficult and expensive, so we also construct a\\ndataset earlygame_keyword by searching for videos online with descriptions that match keywords\\nsuch as “new world”, “let’s play episode 1”, etc.; this is a subset of web_clean and is labeled with\\nthe IDM. See Appendix B.4 and A.3 for full descriptions of both datasets.\\nEffect of Foundation Model Quality on BC Fine-Tuning\\n59x\\n213x\\n59x\\nFigure 5:\\n(Left) Collection and crafting rates for three policies:\\nthe zero-shot VPT foun-\\ndation model, and the VPT foundation model BC fine-tuned to the earlygame_keyword or\\ncontractor_house datasets. BC fine-tuning to either dataset improves performance, including (for\\nthe contractor_house dataset) yielding wooden and stone tools. Proficient Minecraft players take\\na median of 1.2 minutes (1390 actions) to construct wooden tools and 2.3 minutes (2790 actions)\\nto construct stone tools. (Right) Collection and crafting rates for VPT foundation model snapshots\\nthroughout training after they are BC fine-tuned to the contractor_house dataset. In general,\\ncrafting-related behaviors increase throughout foundation model training. Fig. 4 defines the other\\ntask terms (logs, planks, crafting tables, and total crafting).\\n(iv)Sample videos: https://www.youtube.com/playlist?list=PLNAOIb_agjf3U3rSvG_BCWqJ869NdBhcP\\n6\\n\\n\\nFine-tuning to earlygame_keyword results in a large boost compared to the zero-shot foundation\\nmodel: 2.5x more crafting tables, 6.1x more planks, 4.3x more logs, and 5.5x more crafting overall\\n(Fig. 5). However, when fine-tuning to this dataset we did not see any new behaviors emerge,\\nonly a refinement of existing skills. We saw an even bigger improvement when fine-tuning to the\\ncontractor_house dataset: 213x more crafting tables, 59x more wooden planks, 7x more logs,\\nand 59x more crafting over all. In addition, we saw the emergence of crafting wooden tools, which\\nrequires placing a crafting table on the ground, opening it to reveal a new crafting interface, and then\\nusing it to craft wooden tools. This entire sequence takes a proficient human player a median of 1.2\\nminutes (1390 consecutive actions) to accomplish. The model goes further and collects cobblestone,\\nwhich requires a wooden pickaxe to mine, and crafts stone tools, requiring it to again use a crafting\\ntable; this takes a proficient human player a median of 2.3 minutes (2790 consecutive actions). We\\nalso saw this model more frequently raiding villages that randomly spawn in the game, hunting\\nanimals for food, in addition to many behaviors we saw performed by the foundation model.(v)\\nDespite the foundation model’s zero-shot rollout performance plateauing 1/3 into training (Fig. 4,\\nright), fine-tuning performance does continue to increase throughout foundation model training\\n(Fig. 5, right). Additionally, there is a stark difference in performance when training from scratch vs.\\nfine-tuning from the VPT foundation model (Fig. 5 right, comparing the left and rightmost points).\\n4.4\\nFine-Tuning with Reinforcement Learning\\nFigure 6: Typical sequence of items for obtaining a diamond pickaxe. Below each item is the median\\ntime and number of actions contractors required to obtain that item and the percentage of contractors\\nthat got the item within 10 minutes. The median time to obtain a diamond pickaxe is unknown (except\\nthat it is > 20m) because contractors obtained this item in less than 50% of 20-minute episodes.\\nTo demonstrate the efficacy of RL fine-tuning, we chose the challenging goal of obtaining a diamond\\npickaxe within 10 minutes starting from a fresh Minecraft survival world. Doing so involves acquiring\\na sequence of difficult-to-obtain items that require complex skills like mining, inventory management,\\ncrafting with and without a crafting table, tool use, operating a furnace, and mining at the lowest\\ndepths, where many hazards like enemies and lava exist (Fig. 6). Adding to the difficulty, progress\\ncan be easily lost by dropping items, destroying items, or dying. Obtaining a diamond pickaxe more\\noften than not takes a proficient human over 20 minutes (24,000 actions).\\nAgents are rewarded for each item obtained in the sequence, with lower rewards for items that have to\\nbe collected in bulk and higher rewards for items near the end of the sequence. Agents are optimized\\nwith the phasic policy gradient64 RL algorithm for ∼1.3 million episodes (roughly 1.4×1010 frames).\\nEpisodes last for 10 minutes. See Appendix G.1 for reward function and RL training details. Due to\\ncomputational constraints, RL experiments use a ∼248 million parameter VPT model (Appendix H).\\nA major problem when fine-tuning with RL is catastrophic forgetting65,66 because previously learned\\nskills can be lost before their value is realized. For instance, while our VPT foundation model never\\nexhibits the entire sequence of behaviors required to smelt iron zero-shot, it did train on examples of\\nplayers smelting with furnaces. It therefore may have some latent ability to smelt iron once the many\\nprerequisites to do so have been performed. To combat the catastrophic forgetting of latent skills\\nsuch that they can continually improve exploration throughout RL fine-tuning, we add an auxiliary\\nKullback-Leibler (KL) divergence loss between the RL model and the frozen pretrained policy.10\\nTraining from a randomly initialized policy fails to achieve almost any reward, underscoring how\\nhard an exploration challenge the diamond pickaxe task is for RL in the native human action space\\n(Fig. 7a). The model never learns to reliably collect logs, typically the first of many steps to obtaining\\na diamond pickaxe (Fig. 7b). RL fine-tuning from the VPT foundation model does substantially\\nbetter (Fig. 7a), learning everything up to mining iron ore and crafting furnaces. (Fig. 7c). However,\\nthis agent fails at smelting an iron ingot, the next item required to get further into the tech tree, likely\\n(v)Sample Videos: https://www.youtube.com/playlist?list=PLNAOIb_agjf2yDSs4AqcoyPv4z_eWUiKm\\n7\\n\\n\\n0\\n5\\n10\\n15\\n20\\n25\\nReward\\nReward over episodes\\nRL from Rand. Init. model\\nRL from VPT Found. model\\nRL from Early-Game model\\nNo KL-loss\\n(a)\\n0\\n20\\n40\\n60\\n80\\n100\\n% episodes\\nRL from Rand. Init. model\\nLogs\\nPlanks\\nSticks\\nCrafting T\\nables\\nWooden Pickaxe\\nCobblestone\\nStone Pickaxe\\nCoal\\nT\\norch\\nFurnace\\nIron Ore\\nIron Ingot\\nIron Pickaxe\\nDiamonds\\nDiamond Pickaxe\\n(b)\\n0.0\\n0.2\\n0.4\\n0.6\\n0.8\\n1.0\\n1.2\\n1.4\\nEpisodes\\n1e6\\n0\\n20\\n40\\n60\\n80\\n100\\n% episodes\\nRL from VPT Found. model\\n(c)\\n0.0\\n0.2\\n0.4\\n0.6\\n0.8\\n1.0\\n1.2\\n1.4\\nEpisodes\\n1e6\\n0\\n20\\n40\\n60\\n80\\n100\\n% episodes\\n2.5%\\nRL from Early-Game model\\n(d)\\nFigure 7: RL Fine-tuning results. (a) RL from a randomly initialized model fails to get almost\\nany reward, RL fine-tuning from the VPT foundation model performs substantially better with a\\nreward near 13, and RL fine-tuning from the early-game model performs best with a reward of 25.\\nWhen training the early-game model without a KL loss to the original policy (No KL-loss) progress\\nstalls after 100,000 episodes, suggesting that the skills necessary to make further progress have been\\ncatastrophically forgotten. (b) RL from a randomly initialized model occasionally collects sticks by\\nbreaking leaves (an easy but inefficient method of getting sticks that does not require logs or planks)\\nand never learns to reliably collect logs. (c) RL fine-tuning from the VPT Foundation model learns\\neverything in the curriculum up to iron ore and making furnaces, but fails to learn to use the furnace to\\nsmelt iron ingots. (d) RL fine-tuning from the early-game model learns to obtain (at human-level) all\\nitems in the sequence towards a diamond pickaxe and crafts a diamond pickaxe in 2.5% of episodes.\\nbecause the zero-shot probability that the VPT foundation model smelts an iron ingot is too low, even\\nwhen given the prerequisite materials.\\nResults further improve by first BC fine-tuning the VPT Foundation Model to the\\nearlygame_keyword dataset (the early-game model, Sec. 4.3) and then fine-tuning with RL\\n(Fig. 7a), which in preliminary experiments we found to perform better than first fine-tuning to\\ncontractor_house followed by fine-tuning with RL (Appendix G.2). The three-phase training\\n(pretraining, BC fine-tuning, and then RL fine-tuning) succeeds in learning extremely difficult tasks:\\nit achieves over 80% reliability on iron pickaxes, almost 20% reliability on collecting diamonds, and\\n2.5% reliability on obtaining a diamond pickaxe (Fig. 7d). For comparison, human players given\\nthe objective of obtaining a diamond pickaxe collect these items in 57%, 15%, and 12% of episodes,\\nrespectively, meaning our model is human-level for crafting iron pickaxes and mining diamonds.\\nOthers have managed to obtain diamonds with ∼0.1% reliability in 15 minutes32,33 but always with a\\nsimplified action space designed to ease exploration. To the best of our knowledge, we are the first to\\nreport non-zero success rates on crafting a diamond pickaxe. Qualitatively, the model developed\\nuseful skills for diamond mining, such as efficient mining patterns, cave exploration, returning to\\npreviously placed objects like crafting tables, and advanced techniques like using wooden pickaxes\\nas fuel when moving on to iron tools.(vi)\\nFinally, we validated the importance of the KL loss to the pretrained model during RL fine-tuning.\\nThe treatment without a KL loss obtains only items early in the sequence (logs, planks, sticks, and\\ncrafting tables) limiting its reward (Fig. 7a). This failure to progress further into the sequence is\\nlikely because, while the initial skills of chopping logs and crafting planks are being learned with RL,\\nsubsequent skills like crafting a wooden pickaxe are lost due to catastrophic forgetting.\\n4.5\\nData Scaling Properties of the Foundation Model\\nIn this section we validate a core hypothesis behind this work: that it is far more effective to use\\nlabeled contractor data to train an IDM within the VPT method than it is to directly train a BC\\nfoundation model from that same small contractor dataset. If we could cheaply collect a labeled\\ncontractor dataset of a similar order of magnitude as web_clean, then this would not be important;\\nhowever, collecting that scale of data would have cost millions of dollars. Figure 8 compares\\nfoundation models trained on increasing orders of magnitude of data from 1 hour up to the full ∼70k\\nweb_clean dataset. Foundation models trained up to and including 1k hours are trained on the IDM\\n(vi)Videos found at https://www.youtube.com/playlist?list=PLNAOIb_agjf3e_UKweM5pQUSfTw8r-Wfc\\n8\\n\\n\\nTrained on\\nContractor Data\\nTrained on IDM\\nLabeled Web Data\\nFigure 8: (Left) Zero-shot rollout performance of foundation models trained on varying amounts\\nof data. Models to the left of the dashed black line (points ≤1k hours) were trained on contractor\\ndata (ground-truth labels), and models to the right were trained on IDM pseudo-labeled subsets\\nof web_clean. Due to compute limitations, this analysis was performed with smaller (71 million\\nparameter) models except for the final point, which is the 0.5 billion parameter VPT foundation\\nmodel. (Right) The corresponding performance of each model after BC fine-tuning each model to\\nthe contractor_house dataset.\\ncontractor data, and those trained on 5k hours and above are trained on subsets of web_clean, which\\ndoes not contain any IDM contractor data. Scaling training data increases log collection, mining, and\\ncrafting capabilities. The zero-shot model only begins to start crafting crafting tables at over 5000\\nhours of training data. When fine-tuning each foundation model to contractor_house, we see that\\ncrafting rates for crafting tables and wooden tools increase by orders of magnitude when using the\\nentire ∼70k hour web_clean dataset. We furthermore only see the emergence of crafting stone tools\\nat the largest data scale.\\n4.6\\nEffect of Inverse Dynamics Model Quality on Behavioral Cloning\\nFigure 9: Zero-shot performance of BC models\\ntrained from scratch on the earlygame_keyword\\ndataset labeled with IDMs that were trained on\\nincreasing amounts of contractor data.\\nThis section investigates how downstream\\nBC performance is affected by IDM qual-\\nity.\\nWe train IDMs on increasingly larger\\ndatasets and use each to independently label\\nthe earlygame_keyword dataset (this smaller\\ndataset was chosen due to a limited compute bud-\\nget). We then train a BC model from scratch on\\neach dataset and report game statistics for each\\nmodel as a function of IDM contractor dataset\\nsize (Fig. 9).\\nIDMs trained on at least 10 hours of data are\\nrequired for any crafting, and the crafting rate\\nincreases quickly up until 100 hours of data, after which there are few to no gains and differences are\\nlikely due to noise. Similarly, crafting tables are only crafted after 50 or more hours of IDM data, and\\nagain gains plateau after 100 hours. While in all previous experiments we use our best IDM trained\\non 1962 hours of data, these results suggest we could reduce that number to as low as 100 hours.\\n5\\nDiscussion and Conclusion\\nThe results presented in this paper help pave the path to utilizing the wealth of unlabeled data on the\\nweb for sequential decision domains. Compared to generative video modeling or contrastive methods\\nthat would only yield representational priors, VPT offers the exciting possibility of directly learning\\nto act during pretraining and using these learned behavioral priors as extremely effective exploration\\npriors for RL. VPT could even be a better general representation learning method even when the\\ndownstream task is not learning to act in that domain—for example, fine-tuning to explain what is\\nhappening in a video—because arguably the most important information in any given scene would be\\npresent in features trained to correctly predict the distribution over future human actions. We leave\\nthis intriguing direction to future work.\\nFuture work could improve results with more data (we estimate we could collect >1M hours) and\\nlarger, better-tuned models. Furthermore, all the models in this work condition on past observations\\nonly; we cannot ask the model to perform specific tasks. Appendix I presents preliminary experiments\\non conditioning our models on closed captions (text transcripts of speech in videos), showing they\\n9\\n\\n\\nbecome weakly steerable; we believe this a rich direction for future research. Also, loss was not\\nconsistently correlated with downstream evaluation metrics (Sec. 4.2), which often made progress\\nslow and hard-won. Another fruitful future direction would be to investigate the correlation between\\nvarious training metrics and downstream evaluations. Finally, while we do not anticipate any direct\\nnegative societal impacts from the models trained in this work, as VPT improves and expands to other\\ndomains it will be important to assess and mitigate harms that emerge with other forms of pretraining\\non internet datasets, such as emulating inappropriate behavior.67\\nIn conclusion, VPT extends the paradigm of training large and general purpose behavioral priors from\\nfreely available internet-scale data to sequential decision domains. Our models exhibited impressive\\nzero-shot behavior and, when fine-tuned with RL, achieved an unprecedented result of crafting a\\ndiamond pickaxe in Minecraft (all the more difficult given the human interface). We further showed\\nthat contractor data is far better used within the VPT pipeline than to train a foundation model directly\\nand that only a small amount of contractor data (about $2000 USD) was required to unlock massive\\namounts of unlabeled online data for use in BC. Finally, learning with the human keyboard and mouse\\ninterface is highly general and allows losslessly modeling the entire distribution of human behavior.\\nWhile we only experiment in Minecraft, we believe that VPT provides a general recipe for training\\nbehavioral priors in hard, yet generic, action spaces in any domain that has a large amount of freely\\navailable unlabeled data, such as computer usage.\\nReferences\\n[1] Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal,\\nArvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language models are\\nfew-shot learners. Advances in neural information processing systems, 33:1877–1901, 2020.\\n[2] Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. Bert: Pre-training of\\ndeep bidirectional transformers for language understanding. arXiv preprint arXiv:1810.04805,\\n2018.\\n[3] Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike\\nLewis, Luke Zettlemoyer, and Veselin Stoyanov. Roberta: A robustly optimized bert pretraining\\napproach. arXiv preprint arXiv:1907.11692, 2019.\\n[4] Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena,\\nYanqi Zhou, Wei Li, and Peter J Liu. Exploring the limits of transfer learning with a unified\\ntext-to-text transformer. arXiv preprint arXiv:1910.10683, 2019.\\n[5] Dhruv Mahajan, Ross Girshick, Vignesh Ramanathan, Kaiming He, Manohar Paluri, Yixuan Li,\\nAshwin Bharambe, and Laurens Van Der Maaten. Exploring the limits of weakly supervised\\npretraining. In Proceedings of the European conference on computer vision (ECCV), pages\\n181–196, 2018.\\n[6] Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal,\\nGirish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual\\nmodels from natural language supervision. In International Conference on Machine Learning,\\npages 8748–8763. PMLR, 2021.\\n[7] Rishi Bommasani, Drew A Hudson, Ehsan Adeli, Russ Altman, Simran Arora, Sydney von\\nArx, Michael S Bernstein, Jeannette Bohg, Antoine Bosselut, Emma Brunskill, et al. On the\\nopportunities and risks of foundation models. arXiv preprint arXiv:2108.07258, 2021.\\n[8] Xiaohua Zhai, Alexander Kolesnikov, Neil Houlsby, and Lucas Beyer. Scaling vision transform-\\ners. CoRR, abs/2106.04560, 2021. URL https://arxiv.org/abs/2106.04560.\\n[9] David Silver, Aja Huang, Chris J Maddison, Arthur Guez, Laurent Sifre, George Van Den Driess-\\nche, Julian Schrittwieser, Ioannis Antonoglou, Veda Panneershelvam, Marc Lanctot, et al. Mas-\\ntering the game of go with deep neural networks and tree search. Nature, 529(7587):484–489,\\n2016.\\n[10] Oriol Vinyals, Igor Babuschkin, Wojciech M Czarnecki, Michaël Mathieu, Andrew Dudzik, Jun-\\nyoung Chung, David H Choi, Richard Powell, Timo Ewalds, Petko Georgiev, et al. Grandmaster\\nlevel in starcraft ii using multi-agent reinforcement learning. Nature, 575(7782):350–354, 2019.\\n10\\n\\n\\n[11] Richard S Sutton and Andrew G Barto. Reinforcement learning: An introduction. MIT press,\\n2018.\\n[12] Christopher Berner, Greg Brockman, Brooke Chan, Vicki Cheung, Przemysław D˛\\nebiak, Christy\\nDennison, David Farhi, Quirin Fischer, Shariq Hashme, Chris Hesse, et al. Dota 2 with large\\nscale deep reinforcement learning. arXiv preprint arXiv:1912.06680, 2019.\\n[13] Bowen Baker, Ingmar Kanitscheider, Todor Markov, Yi Wu, Glenn Powell, Bob McGrew,\\nand Igor Mordatch.\\nEmergent tool use from multi-agent autocurricula.\\narXiv preprint\\narXiv:1909.07528, 2019.\\n[14] Max Jaderberg, Wojciech M Czarnecki, Iain Dunning, Luke Marris, Guy Lever, Antonio Garcia\\nCastaneda, Charles Beattie, Neil C Rabinowitz, Ari S Morcos, Avraham Ruderman, et al.\\nHuman-level performance in 3d multiplayer games with population-based reinforcement learn-\\ning. Science, 364(6443):859–865, 2019.\\n[15] Adrià Puigdomènech Badia, Bilal Piot, Steven Kapturowski, Pablo Sprechmann, Alex Vitvit-\\nskyi, Zhaohan Daniel Guo, and Charles Blundell. Agent57: Outperforming the atari human\\nbenchmark. In International Conference on Machine Learning, pages 507–517. PMLR, 2020.\\n[16] Marc Bellemare, Sriram Srinivasan, Georg Ostrovski, Tom Schaul, David Saxton, and Remi\\nMunos.\\nUnifying count-based exploration and intrinsic motivation.\\nAdvances in neural\\ninformation processing systems, 29, 2016.\\n[17] Yuri Burda, Harrison Edwards, Amos Storkey, and Oleg Klimov. Exploration by random\\nnetwork distillation. arXiv preprint arXiv:1810.12894, 2018.\\n[18] Adrien Ecoffet, Joost Huizinga, Joel Lehman, Kenneth O Stanley, and Jeff Clune. First return,\\nthen explore. Nature, 590(7847):580–586, 2021.\\n[19] Peter C Humphreys, David Raposo, Toby Pohlen, Gregory Thornton, Rachita Chhaparia, Alistair\\nMuldal, Josh Abramson, Petko Georgiev, Alex Goldin, Adam Santoro, et al. A data-driven\\napproach for learning to control computers. arXiv preprint arXiv:2202.08137, 2022.\\n[20] Tianlin Shi, Andrej Karpathy, Linxi Fan, Jonathan Hernandez, and Percy Liang. World of\\nbits: An open-domain platform for web-based agents. In Doina Precup and Yee Whye Teh,\\neditors, Proceedings of the 34th International Conference on Machine Learning, volume 70 of\\nProceedings of Machine Learning Research, pages 3135–3144. PMLR, 06–11 Aug 2017. URL\\nhttps://proceedings.mlr.press/v70/shi17a.html.\\n[21] Andrew Y Ng, Stuart J Russell, et al. Algorithms for inverse reinforcement learning. In Icml,\\nvolume 1, page 2, 2000.\\n[22] Faraz Torabi, Garrett Warnell, and Peter Stone. Recent advances in imitation learning from\\nobservation. arXiv preprint arXiv:1905.13566, 2019.\\n[23] Jonathan Ho and Stefano Ermon. Generative adversarial imitation learning. Advances in neural\\ninformation processing systems, 29, 2016.\\n[24] Faraz Torabi, Garrett Warnell, and Peter Stone. Behavioral cloning from observation. arXiv\\npreprint arXiv:1805.01954, 2018.\\n[25] YuXuan Liu, Abhishek Gupta, Pieter Abbeel, and Sergey Levine. Imitation from observation:\\nLearning to imitate behaviors from raw video via context translation. In 2018 IEEE International\\nConference on Robotics and Automation (ICRA), pages 1118–1125. IEEE, 2018.\\n[26] Twinfinite\\nStaff.\\nMost\\nplayed\\ngames\\nin\\n2021,\\nranked\\nby\\npeak\\nconcur-\\nrent\\nplayers.\\nTwinfinite.\\nURL\\nhttps://twinfinite.net/2021/12/\\nmost-played-games-in-2020-ranked-by-peak-concurrent-players/.\\n[27] William H Guss, Brandon Houghton, Nicholay Topin, Phillip Wang, Cayden Codel, Manuela\\nVeloso, and Ruslan Salakhutdinov. Minerl: A large-scale dataset of minecraft demonstrations.\\narXiv preprint arXiv:1907.13440, 2019.\\n11\\n\\n\\n[28] Chen Tessler, Shahar Givony, Tom Zahavy, Daniel Mankowitz, and Shie Mannor. A deep\\nhierarchical approach to lifelong learning in minecraft. In Proceedings of the AAAI Conference\\non Artificial Intelligence, volume 31, 2017.\\n[29] Christian Scheller, Yanick Schraner, and Manfred Vogel. Sample efficient reinforcement\\nlearning through learning from demonstrations in minecraft. In NeurIPS 2019 Competition and\\nDemonstration Track, pages 67–76. PMLR, 2020.\\n[30] Ingmar Kanitscheider, Joost Huizinga, David Farhi, William Hebgen Guss, Brandon Houghton,\\nRaul Sampedro, Peter Zhokhov, Bowen Baker, Adrien Ecoffet, Jie Tang, et al. Multi-task\\ncurriculum learning in a complex, visual, hard-exploration domain: Minecraft. arXiv preprint\\narXiv:2106.14876, 2021.\\n[31] Junhyuk Oh, Valliappa Chockalingam, Honglak Lee, et al. Control of memory, active perception,\\nand action in minecraft. In International Conference on Machine Learning, pages 2790–2799.\\nPMLR, 2016.\\n[32] Vihang P Patil, Markus Hofmarcher, Marius-Constantin Dinu, Matthias Dorfer, Patrick M Blies,\\nJohannes Brandstetter, Jose A Arjona-Medina, and Sepp Hochreiter. Align-rudder: Learning\\nfrom few demonstrations by reward redistribution. arXiv preprint arXiv:2009.14108, 2020.\\n[33] Alexey Skrynnik, Aleksey Staroverov, Ermek Aitygulov, Kirill Aksenov, Vasilii Davydov, and\\nAleksandr I Panov. Forgetful experience replay in hierarchical reinforcement learning from\\ndemonstrations. arXiv preprint arXiv:2006.09939, 2020.\\n[34] Zichuan Lin, Junyou Li, Jianing Shi, Deheng Ye, Qiang Fu, and Wei Yang.\\nJuewu-mc:\\nPlaying minecraft with sample-efficient hierarchical reinforcement learning. arXiv preprint\\narXiv:2112.04907, 2021.\\n[35] Dean A Pomerleau. Alvinn: An autonomous land vehicle in a neural network. Advances in\\nneural information processing systems, 1, 1988.\\n[36] Stefan Schaal. Is imitation learning the route to humanoid robots? Trends in cognitive sciences,\\n3(6):233–242, 1999.\\n[37] Brenna D Argall, Sonia Chernova, Manuela Veloso, and Brett Browning. A survey of robot\\nlearning from demonstration. Robotics and autonomous systems, 57(5):469–483, 2009.\\n[38] Ahmed Hussein, Mohamed Medhat Gaber, Eyad Elyan, and Chrisina Jayne. Imitation learning:\\nA survey of learning methods. ACM Computing Surveys (CSUR), 50(2):1–35, 2017.\\n[39] Claude Sammut, Scott Hurst, Dana Kedzier, and Donald Michie. Learning to fly. In Machine\\nLearning Proceedings 1992, pages 385–393. Elsevier, 1992.\\n[40] Alessandro Giusti, Jérôme Guzzi, Dan C Cire¸\\nsan, Fang-Lin He, Juan P Rodríguez, Flavio\\nFontana, Matthias Faessler, Christian Forster, Jürgen Schmidhuber, Gianni Di Caro, et al. A\\nmachine learning approach to visual perception of forest trails for mobile robots. IEEE Robotics\\nand Automation Letters, 1(2):661–667, 2015.\\n[41] Mariusz Bojarski, Davide Del Testa, Daniel Dworakowski, Bernhard Firner, Beat Flepp, Prasoon\\nGoyal, Lawrence D Jackel, Mathew Monfort, Urs Muller, Jiakai Zhang, et al. End to end learning\\nfor self-driving cars. arXiv preprint arXiv:1604.07316, 2016.\\n[42] Felipe Codevilla, Matthias Müller, Antonio López, Vladlen Koltun, and Alexey Dosovitskiy.\\nEnd-to-end driving via conditional imitation learning. In 2018 IEEE international conference\\non robotics and automation (ICRA), pages 4693–4700. IEEE, 2018.\\n[43] Rémi Coulom. Computing “elo ratings” of move patterns in the game of go. ICGA journal, 30\\n(4):198–208, 2007.\\n[44] Todd Hester, Matej Vecerik, Olivier Pietquin, Marc Lanctot, Tom Schaul, Bilal Piot, Dan Horgan,\\nJohn Quan, Andrew Sendonaris, Ian Osband, et al. Deep q-learning from demonstrations. In\\nProceedings of the AAAI Conference on Artificial Intelligence, volume 32, 2018.\\n12\\n\\n\\n[45] Ashley Edwards, Himanshu Sahni, Yannick Schroecker, and Charles Isbell. Imitating latent\\npolicies from observation. In International conference on machine learning, pages 1755–1763.\\nPMLR, 2019.\\n[46] Xue Bin Peng, Angjoo Kanazawa, Jitendra Malik, Pieter Abbeel, and Sergey Levine. Sfv:\\nReinforcement learning of physical skills from videos. ACM Transactions On Graphics (TOG),\\n37(6):1–14, 2018.\\n[47] Feryal Behbahani, Kyriacos Shiarlis, Xi Chen, Vitaly Kurin, Sudhanshu Kasewa, Ciprian Stirbu,\\nJoao Gomes, Supratik Paul, Frans A Oliehoek, Joao Messias, et al. Learning from demonstration\\nin the wild. In 2019 International Conference on Robotics and Automation (ICRA), pages\\n775–781. IEEE, 2019.\\n[48] Yusuf Aytar, Tobias Pfaff, David Budden, Thomas Paine, Ziyu Wang, and Nando De Fre-\\nitas. Playing hard exploration games by watching youtube. Advances in neural information\\nprocessing systems, 31, 2018.\\n[49] Deepak Pathak, Parsa Mahmoudieh, Guanghao Luo, Pulkit Agrawal, Dian Chen, Yide Shentu,\\nEvan Shelhamer, Jitendra Malik, Alexei A. Efros, and Trevor Darrell. Zero-shot visual imitation.\\nIn ICLR, 2018.\\n[50] Ashvin Nair, Dian Chen, Pulkit Agrawal, Phillip Isola, Pieter Abbeel, Jitendra Malik, and Sergey\\nLevine. Combining self-supervised learning and imitation for vision-based rope manipulation.\\npages 2146–2153, 05 2017. doi: 10.1109/ICRA.2017.7989247.\\n[51] Duy Nguyen-Tuong, Jan Peters, Matthias Seeger, and Bernhard Schölkopf. Learning inverse\\ndynamics: a comparison. In European symposium on artificial neural networks, number CONF,\\n2008.\\n[52] David Abel, Alekh Agarwal, Fernando Diaz, Akshay Krishnamurthy, and Robert E Schapire.\\nExploratory gradient boosting for reinforcement learning in complex domains. arXiv preprint\\narXiv:1603.04119, 2016.\\n[53] Dilip Arumugam, Jun Ki Lee, Sophie Saskin, and Michael L Littman. Deep reinforcement\\nlearning from policy-dependent human feedback. arXiv preprint arXiv:1902.04257, 2019.\\n[54] Alexander Trott, Stephan Zheng, Caiming Xiong, and Richard Socher. Keeping your dis-\\ntance: Solving sparse reward tasks using self-balancing shaped rewards. Advances in Neural\\nInformation Processing Systems, 32, 2019.\\n[55] Stephan Alaniz. Deep reinforcement learning with model learning and monte carlo tree search\\nin minecraft. arXiv preprint arXiv:1803.08456, 2018.\\n[56] Hiroto Udagawa, Tarun Narasimhan, and Shim-Young Lee. Fighting zombies in minecraft with\\ndeep reinforcement learning. Technical report, Technical report, Technical report, Stanford\\nUniversity, 2016.\\n[57] Tianmin Shu, Caiming Xiong, and Richard Socher. Hierarchical and interpretable skill acquisi-\\ntion in multi-task reinforcement learning. arXiv preprint arXiv:1712.07294, 2017.\\n[58] Junhyuk Oh, Satinder Singh, Honglak Lee, and Pushmeet Kohli. Zero-shot task generalization\\nwith multi-task deep reinforcement learning. In International Conference on Machine Learning,\\npages 2661–2670. PMLR, 2017.\\n[59] Zhengxiang Shi, Yue Feng, and Aldo Lipani. Learning to execute or ask clarification questions.\\narXiv preprint arXiv:2204.08373, 2022.\\n[60] Tambet Matiisen, Avital Oliver, Taco Cohen, and John Schulman. Teacher–student curriculum\\nlearning. IEEE transactions on neural networks and learning systems, 31(9):3732–3740, 2019.\\n[61] Robert George Douglas Steel, James Hiram Torrie, et al. Principles and procedures of statistics.\\nPrinciples and procedures of statistics., 1960.\\n13\\n\\n\\n[62] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image\\nrecognition. In Proceedings of the IEEE conference on computer vision and pattern recognition,\\npages 770–778, 2016.\\n[63] Jared Kaplan, Sam McCandlish, Tom Henighan, Tom B Brown, Benjamin Chess, Rewon Child,\\nScott Gray, Alec Radford, Jeffrey Wu, and Dario Amodei. Scaling laws for neural language\\nmodels. arXiv preprint arXiv:2001.08361, 2020.\\n[64] Karl W Cobbe, Jacob Hilton, Oleg Klimov, and John Schulman. Phasic policy gradient. In\\nMarina Meila and Tong Zhang, editors, Proceedings of the 38th International Conference on\\nMachine Learning, volume 139 of Proceedings of Machine Learning Research, pages 2020–\\n2027. PMLR, 18–24 Jul 2021. URL https://proceedings.mlr.press/v139/cobbe21a.\\nhtml.\\n[65] Dhireesha Kudithipudi, Mario Aguilar-Simon, Jonathan Babb, Maxim Bazhenov, Douglas\\nBlackiston, Josh Bongard, Andrew P Brna, Suraj Chakravarthi Raja, Nick Cheney, Jeff Clune,\\net al. Biological underpinnings for lifelong learning machines. Nature Machine Intelligence, 4\\n(3):196–210, 2022.\\n[66] James Kirkpatrick, Razvan Pascanu, Neil Rabinowitz, Joel Veness, Guillaume Desjardins,\\nAndrei A Rusu, Kieran Milan, John Quan, Tiago Ramalho, Agnieszka Grabska-Barwinska, et al.\\nOvercoming catastrophic forgetting in neural networks. Proceedings of the national academy of\\nsciences, 114(13):3521–3526, 2017.\\n[67] Emily M Bender, Timnit Gebru, Angelina McMillan-Major, and Shmargaret Shmitchell. On\\nthe dangers of stochastic parrots: Can language models be too big???. In Proceedings of the\\n2021 ACM Conference on Fairness, Accountability, and Transparency, pages 610–623, 2021.\\n[68] Fabian Pedregosa, Gaël Varoquaux, Alexandre Gramfort, Vincent Michel, Bertrand Thirion,\\nOlivier Grisel, Mathieu Blondel, Peter Prettenhofer, Ron Weiss, Vincent Dubourg, et al. Scikit-\\nlearn: Machine learning in python. the Journal of machine Learning research, 12:2825–2830,\\n2011.\\n[69] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez,\\nŁukasz Kaiser, and Illia Polosukhin. Attention is all you need. Advances in neural information\\nprocessing systems, 30, 2017.\\n[70] Jimmy Lei Ba, Jamie Ryan Kiros, and Geoffrey E Hinton. Layer normalization. arXiv preprint\\narXiv:1607.06450, 2016.\\n[71] Yann A LeCun, Léon Bottou, Genevieve B Orr, and Klaus-Robert Müller. Efficient backprop.\\nIn Neural networks: Tricks of the trade, pages 9–48. Springer, 2012.\\n[72] Diederik P Kingma and Jimmy Ba. Adam: A method for stochastic optimization. arXiv preprint\\narXiv:1412.6980, 2014.\\n[73] Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan,\\nTrevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, Alban Desmaison, Andreas\\nKopf, Edward Yang, Zachary DeVito, Martin Raison, Alykhan Tejani, Sasank Chilamkurthy,\\nBenoit Steiner, Lu Fang, Junjie Bai, and Soumith Chintala. Pytorch: An imperative style, high-\\nperformance deep learning library. In H. Wallach, H. Larochelle, A. Beygelzimer, F. d'Alché-\\nBuc, E. Fox, and R. Garnett, editors, Advances in Neural Information Processing Systems 32,\\npages 8024–8035. Curran Associates, Inc., 2019. URL http://papers.neurips.cc/paper/\\n9015-pytorch-an-imperative-style-high-performance-deep-learning-library.\\npdf.\\n[74] Yann N Dauphin, Razvan Pascanu, Caglar Gulcehre, Kyunghyun Cho, Surya Ganguli, and\\nYoshua Bengio. Identifying and attacking the saddle point problem in high-dimensional non-\\nconvex optimization. Advances in neural information processing systems, 27, 2014.\\n[75] Jason Yosinski, Jeff Clune, Yoshua Bengio, and Hod Lipson. How transferable are features in\\ndeep neural networks? Advances in neural information processing systems, 27, 2014.\\n14\\n\\n\\n[76] Zihang Dai, Zhilin Yang, Yiming Yang, Jaime Carbonell, Quoc V Le, and Ruslan Salakhutdinov.\\nTransformer-xl: Attentive language models beyond a fixed-length context. arXiv preprint\\narXiv:1901.02860, 2019.\\n[77] John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal\\npolicy optimization algorithms. arXiv preprint arXiv:1707.06347, 2017.\\n[78] John Schulman, Philipp Moritz, Sergey Levine, Michael Jordan, and Pieter Abbeel. High-\\ndimensional continuous control using generalized advantage estimation.\\narXiv preprint\\narXiv:1506.02438, 2015.\\n[79] Ronald J Williams. Simple statistical gradient-following algorithms for connectionist reinforce-\\nment learning. Machine learning, 8(3):229–256, 1992.\\n[80] Volodymyr Mnih, Adria Puigdomenech Badia, Mehdi Mirza, Alex Graves, Timothy Lillicrap,\\nTim Harley, David Silver, and Koray Kavukcuoglu. Asynchronous methods for deep reinforce-\\nment learning. In International conference on machine learning, pages 1928–1937. PMLR,\\n2016.\\n[81] Marcin Andrychowicz, Filip Wolski, Alex Ray, Jonas Schneider, Rachel Fong, Peter Welinder,\\nBob McGrew, Josh Tobin, OpenAI Pieter Abbeel, and Wojciech Zaremba. Hindsight experience\\nreplay. Advances in neural information processing systems, 30, 2017.\\n[82] Tom Schaul, Daniel Horgan, Karol Gregor, and David Silver. Universal value function ap-\\nproximators. In International conference on machine learning, pages 1312–1320. PMLR,\\n2015.\\n[83] Open Ended Learning Team, Adam Stooke, Anuj Mahajan, Catarina Barros, Charlie Deck,\\nJakob Bauer, Jakub Sygnowski, Maja Trebacz, Max Jaderberg, Michael Mathieu, et al. Open-\\nended learning leads to generally capable agents. arXiv preprint arXiv:2107.12808, 2021.\\n[84] Jelena Luketina, Nantas Nardelli, Gregory Farquhar, Jakob Foerster, Jacob Andreas, Edward\\nGrefenstette, Shimon Whiteson, and Tim Rocktäschel.\\nA survey of reinforcement learn-\\ning informed by natural language. In Proceedings of the Twenty-Eighth International Joint\\nConference on Artificial Intelligence, IJCAI-19, pages 6309–6317. International Joint Confer-\\nences on Artificial Intelligence Organization, 7 2019. doi: 10.24963/ijcai.2019/880. URL\\nhttps://doi.org/10.24963/ijcai.2019/880.\\n[85] DeepMind Interactive Agents Team, Josh Abramson, Arun Ahuja, Arthur Brussee, Federico\\nCarnevale, Mary Cassin, Felix Fischer, Petko Georgiev, Alex Goldin, Tim Harley, et al. Creating\\nmultimodal interactive agents with imitation and self-supervised learning. arXiv preprint\\narXiv:2112.03763, 2021.\\n[86] Aditya Ramesh, Prafulla Dhariwal, Alex Nichol, Casey Chu, and Mark Chen. Hierarchical\\ntext-conditional image generation with clip latents. arXiv preprint arXiv:2204.06125, 2022.\\n[87] Dong Yu and Li Deng. Automatic speech recognition, volume 1. Springer, 2016.\\n[88] Daulet Nurmanbetov. rpunct, May 25 2021. URL https://github.com/Felflare/rpunct.\\naccessed 2022-04-22.\\n[89] Arvind Neelakantan, Tao Xu, Raul Puri, Alec Radford, Jesse Michael Han, Jerry Tworek,\\nQiming Yuan, Nikolas Tezak, Jong Wook Kim, Chris Hallacy, et al. Text and code embeddings\\nby contrastive pre-training. arXiv preprint arXiv:2201.10005, 2022.\\nAcknowledgements\\nWe thank the following people for helpful discussions and support: Bob McGrew, Ken Stanley,\\nJoel Lehman, Ilya Sutskever, Wojciech Zaremba, Ingmar Kanitscheider, David Farhi, Glenn Powell,\\nJonathan Gordon, and the OpenAI supercomputing team, especially Christian Gibson, Ben Chess,\\nand Christopher Berner.\\n15\\n\\n\\nSupplementary Information\\nA\\nCollecting Internet Data\\nA.1\\nInitial Unclean Dataset Curation\\nOur goal was to curate a video dataset of Minecraft gameplay from the survival game mode. Addition-\\nally, we prefer the data come from game modes as close as possible to our evaluation environment,\\nmeaning preferably coming from Minecraft version 1.16, being on a computer (which uses a mouse\\nand keyboard vs. video game controllers with keypads and other buttons), being single- (vs. multi-)\\nplayer, and having the default look of the game (vs. modifications that alter that style, such as to\\nmake it look realistic). To try to accomplish these goals, we collect a dataset by performing keyword\\nsearches of publicly available videos on the internet. A list of search queries we used are given in\\nTable 1.\\nminecraft survival longplay\\nminecraft gameplay no webcam\\nminecraft gameplay survival mode\\nminecraft survival tutorial\\nminecraft survival guide\\nminecraft survival let’s play\\nminecraft survival for beginners\\nminecraft beginners guide\\nultimate minecraft starter guide\\nminecraft survival guide 1.16\\nminecraft how to start a new survival world\\nminecraft survival fresh start\\nminecraft survival let’s play episode 1\\nlet’s play minecraft episode 1\\nminecraft survival 101\\nminecraft survival learning to play\\nhow to play minecraft survival\\nhow to play minecraft\\nminecraft survival basic\\nminecraft survival for noobs\\nminecraft survival for dummies\\nhow to play minecraft for beginners\\nminecraft survival tutorial series\\nminecraft survival new world\\nminecraft survival a new beginning\\nminecraft survival episodio 1\\nminecraft survival epizod 1\\nminecraft survival 1. bölüm\\ni made a new minecraft survival world\\nTable 1: Search terms used for generating the initial web dataset.\\nFor videos that have metadata available, we perform an additional step of metadata-based filtering\\nto eliminate videos that do not fit our target distribution. In this step, we look for a list of blacklist\\nkeywords in the video title and description and reject videos that contain these terms. The blacklist\\nkeywords we use are: {ps3, ps4, ps5, xbox 360, playstation, timelapse, multiplayer, minecraft pe,\\npocket edition, skyblock, realistic minecraft, how to install, how to download, realmcraft, animation}.\\nThis process yielded us ∼270k hours of unlabeled data, which we filter down to only a “clean” subset\\nas described in the next section.\\nA.2\\nTraining a Model to Filter out Unclean Video Segments\\nWe restrict the scope of this work to the Minecraft Survival game mode and therefore limit our\\ntraining dataset to clips that are obtained from this mode that are relatively free from visual artifacts.\\n16\\n\\n\\nTo do so, we asked contractors to label a set of random video frames (images) from Minecraft videos\\n(N=8800). These images were from a random subset of the videos we collected toward the beginning\\nof the project (Section A.1).\\nA.2.1\\nLabel Collection\\nWe asked 5 workers on Amazon Mechanical Turk (mTurk) that we selected with a sample qualification\\ntask to label random screen capture images to be used in training the classifier. A sample worker\\ninterface that the workers saw on mTurk is given in Figure 10.\\nWe asked workers to label videos as being in one of the following three categories (see Figure 11 for\\nvisual examples of each class):\\n1. Minecraft Survival Mode - No Artifacts: Video frames (images) that correspond\\nto the Minecraft Survival game mode that do not contain any non-game visual artifacts (e.g.\\nsubscribe buttons, channel logos, advertisements, picture-in-picture of the narrator, etc.).\\n2. Minecraft Survival Mode - with Artifacts:\\nVideo frames (images) of the\\nMinecraft Survival game mode that include such visual artifacts.\\n3. None of the Above: Video frames (images) that are not from the Minecraft survival\\ngame mode, including those from other Minecraft game modes such as creative mode or\\neven other games/topics entirely.\\nThe full set of instructions workers received are as follows (note that we also included multiple image\\nexamples from each category in the worker instructions, similar to the sample subset provided in\\nFigure 11):\\nPlease help us identify screenshots that belong only to the survival mode in Minecraft. Everything\\nelse (Minecraft creative mode, other games, music videos, etc.) should be marked as None of the\\nabove. Survival mode is identified by the info at the bottom of the screen:\\n• a health bar (row of hearts)\\n• a hunger bar (row of chicken drumsticks)\\n• a bar showing items held\\nSurvival Mode\\nValid survival mode videos have health/hunger bars and an item hotbar at the bottom of the screen.\\nCreative Mode\\nCreative mode only has an item hotbar and should be classified as None of the Above.\\nLabel Descriptions\\n• Minecraft Survival Mode - No Artifacts: These images will be clean screenshots\\nfrom the Minecraft survival mode gameplay without any noticeable artifacts.\\n• Minecraft Survival Mode - with Artifacts: These images will be valid survival\\nmode screenshots, but with some added artifacts. Typical artifacts may include image\\noverlays (a logo/brand), text annotations, a picture-in-picture of the player, etc.\\n• None of the Above: Use this category when the image is not a valid Minecraft survival\\nscreenshot. It may be a non-Minecraft frame or from a different game mode. In non-survival\\ngame modes such as the creative mode, the health/hunger bars will be missing from the\\nimage, the item hotbar may or may not be still present.\\nIn total, we spent $319.96 on human labeling experiments on mTurk, of which $159.98 was directly\\npaid to workers. The remaining amount was spent towards Amazon platform fees. The workers\\nreceived $0.01 per labeled image, at an hourly compensation of $7.20 (based on an estimated labeling\\ntime of 5 seconds/image – in our internal sample run of the same task, we found the average labeling\\ntime to be < 3 seconds).\\nSince we perform rigorous keyword and metadata based filtering of videos (as described in A.1) from\\nwhich we served sample images to be labeled, serving offensive content to workers was extremely\\n17\\n\\n\\nlow risk and no such images were detected during our manual checks. We only collected labels\\nduring our experiment, and the workers were fully anonymized via the mTurk platform, therefore no\\npersonally identifiable information (PII) was collected.\\nFigure 10: Amazon Mechanical Turk worker interface showing an example labeling task\\nFigure 11: (Left) Sample image for Class 1: Minecraft Survival Mode - No Artifacts.\\n(Middle) Sample image for Class 2: Minecraft Survival Mode - with Artifacts – Image\\ncontains annotations and picture-in-picture of the narrator. (Right) Sample image for Class 3: None\\nof the Above – Image is missing the hotbar as well as health and armor bars, indicating that it was\\nnot captured during survival mode gameplay\\nA.2.2\\nSVM Training\\nWith the image labels collected as described in the previous section, we trained a classifier to extract\\nvideo segments that consist of frames from the Minecraft Survival Mode - No Artifacts\\ncategory. Given a set of labeled images, we obtain embeddings for each image using the RN50x64\\nResNet CLIP Model.6 This is a ResNet-based CLIP model that is scaled up to have approximately\\n64x the compute of a ResNet-50. We then train a Support Vector Machine (SVM) using the RBF\\nkernel to obtain a frame classifier. We use the Scikit-learn68 SVM implementation with the parameter\\nconfiguration given in Table 2.\\nFinally, we apply the classifier to frames of raw video sequences at a rate of 3 frames/second. We filter\\nfor videos that consist of at least 80% \\\"clean\\\" frames at this stage (Classes Minecraft Survival\\nMode - with Artifacts and None of the Above are both considered not clean). From this set,\\nwe apply a median filter (with a kernel size of 7) to the labels and segment videos by splitting the\\n\\\"clean\\\" segments that are at least 5s in duration. The result of this is our final web_clean dataset.\\nA.3\\nearly_game Dataset\\nThe early_game dataset is a ∼3000 hour subset of web_clean targeted at “early game” Minecraft\\nbehavior, i.e. instances where players start in a fresh world with no items. We obtain the metadata\\ntext that accompanies the videos in web_clean and determine whether any of the following regular\\nexpressions match:\\n18\\n\\n\\nCLIP Model Specification\\nRN50x64 (see text)\\nCLIP Input Image Resolution\\n448x448x3\\nCLIP Embedding Feature Length\\n1024\\nSVM Parameters\\nKernel\\nrbf\\nC\\n20\\nGamma\\nscale\\nSample Size\\nClass 1\\n2200\\nClass 2\\n2200\\nClass 3\\n4400\\nTable 2: Feature Extraction Details and SVM Configuration. The parameters are for the SVM\\nimplementation in Scikit-learn68.\\n• (ep|episode|eps|day|session|sesh|chapter|chap\\n.|series|part|parte|pt|round|day|tâ\\n.p|bölüm|episodio|epizod|\\u000bpizod)(\\n)*(\\\\.1|#1|1|\\\\.01|#01|01|one[ˆ0-9]|$)\\n• start\\n• beginning\\n• (new|fresh|clean).*(world|game|play)\\n• from scratch\\nFrom this set of videos, we take only the first 5 minutes of each video.\\nB\\nContractor Data\\nB.1\\nRecording Contractor Play\\nOur contractors use a custom Minecraft recorder that we built that records their actions and game video\\nfeeds as they play. The recorder is implemented using the MCP-Reborn (github.com/Hexeption/MCP-\\nReborn) modding package. To ensure that the recorder environment is as close as possible to\\nthe Minecraft environment used for RL rollouts and evaluations (Appendix C), we use the same\\nunderlying game engine for both. The recorder is a Java app that runs in a window mode, with\\nconstant resolution of 1280x760. Brightness is set to 0 (the \\\"gloomy\\\" setting in Minecraft), which\\nis the default setting. Other graphics settings (field of view, GUI scale) are fixed to the values used\\nin the Minecraft environment (C.1); we explicitly prevented users from changing graphics settings.\\nUnlike the environment, the recorder allows all keyboard key presses and continuous (as opposed to\\nbinned) mouse actions. On every game step (or “tick”) the frame buffer used to display the game\\nwindow is downsized to 640x360 and written into a video file. In-game actions are recorded in a\\nseparate JSONL file (a text file where each line is a JSON-formatted string). All recordings are\\nchunked into 5 minute clips: after each 5 minute segment of contractor game play the recorder\\nautomatically uploads the video file, the JSONL file with actions, as well as a Minecraft state file.\\nTo ensure that contractors cannot corrupt each other’s data, we provided every contractor with an\\nindividual cloud bucket, as well as with credentials giving write access only to that bucket. Credentials\\nalso included adjective-adjective-noun names (e.g. grumpy-amethyst-chipmunk), generated with the\\nnamegenerator python package to ensure contractor anonymity when we publish the data.\\nB.2\\nContractor Contract\\nWe recruited contractors by posting the following offer on the UpWork freelancing platform.\\n“We are collecting data for training AI models in Minecraft. You’ll need to install\\njava, download the modified version of Minecraft (that collects and uploads your\\nplay data), and play Minecraft survival mode! Paid per hour of gameplay. Prior\\nexperience in Minecraft not necessary. We do not collect any data that is unrelated\\nto Minecraft from your computer.”\\n19\\n\\n\\nWe had the applications open for a day, and then randomly selected 10 applicants for the first round of\\ncontractors. Later in the project, as we needed more data and as some contractors asked to terminate\\ntheir contracts, we added more applicants from the original pool as well as referrals from the currently\\nworking contractors. The contractors were paid $20 per hour (minus Upwork platform fees and\\napplicable taxes). All of the results presented in this paper are based on about 4,500 hours of data\\n(including data recorded to gather statistics of human play that was not used for training), which cost\\nus around $90,000. Over the course of the project, we collected some data we did not use due to\\nbugs in the recorder and for some ideas we ultimately did not pursue. In total, we spent about $160k\\nfor contractor compensation over the course of the project. However, as we discuss in Sec. 4.6, we\\ncould likely obtain most of our results with an IDM trained using only $2000 worth of data, i.e. the\\nfoundation VPT model, BC fine-tuning to the earlygame_keyword dataset, and the RL fine-tuning\\nresults. Collecting the contractor_house dataset cost about $8000. Because we used the IDM\\ntrained on about 2000 hours of contractor data, the actual cost of contractor data for those results was\\naround $40,000.\\nIn early stages of the project, we were planning to use contractor data solely for the purpose of training\\nthe IDM. As such, no specific tasks were given, other than “play the survival mode of Minecraft like\\nyou normally would.” Later in the project, we requested that contractors perform specific tasks in\\nMinecraft, such as:\\n• Collect as many units of wood as possible, using only wooden or stone tools (treechop)\\n• Start a new world every 30 minutes of game play\\n• Build a basic house in 10 minutes using only dirt, wood, sand, and either wooden or stone\\ntools (contractor_house, more details below in Appendix B.4).\\n• Starting from a new world and an empty inventory, find resources and craft a diamond\\npickaxe in 20 minutes (obtain_diamond_pickaxe). This dataset was used to obtain\\nstatistics for how long it takes humans on average to complete this task (and the subtasks\\nrequired to complete it) when obtaining a diamond pickaxe is their goal.\\nSince we only recorded in-game events and videos, the data does not include personally identifiable\\ninformation. That being said, the contractors could theoretically use Minecraft’s open-world property\\nto generate personally identifiable information and/or offensive content (e.g. by using Minecraft\\nblocks to write their name or offensive messages, then finding a spot from which the message would\\nbe visible). In practice, we have not seen any attempts to do so in the contractor videos that we\\nwatched. Of course, we train our BC models on videos from the internet of people playing Minecraft,\\nand if such behavior is in those videos our model could also potentially learn it, although we expect\\nsuch behavior is rare enough that our model would not be likely to reproduce it.\\nB.3\\nData for the Inverse Dynamics Model.\\nSince the IDM’s task is to infer actions given the video, any labelled data is appropriate for IDM\\ntraining. In practice, we included general gameplay as well as the treechop task data described\\nin the previous section, which amounted to a total of 1962 hours. Due to collecting datasets like\\ncontractor_house only at late stages of the project, they were not included in IDM training.\\nB.4\\ncontractor_house.\\nThe contractor_house contains about 420 hours of data. We asked contractors to build a basic\\nhouse in 10 minutes, using only basic dirt, wood, and sand, blocks. Each trajectory starts in a newly\\ngenerated world and a timer forcibly ends a trajectory after a 20 minute time limit. For this task, many\\ncontractors chose to begin their trajectories by crafting basic tools and building blocks, specifically\\nit was common for the first 2 minutes to be spent crafting a wooden pickaxe and then mining stone\\nfor an assortment of stone tools before gathering more building blocks and beginning to create their\\nstructure.\\nC\\nMinecraft environment details\\nOur Minecraft training environment is a hybrid between MineRL27 and the MCP-Reborn\\n(github.com/Hexeption/MCP-Reborn) Minecraft modding package. Unlike the regular Minecraft\\n20\\n\\n\\nFigure 12: (Left) Sample of a Minecraft frame in the original resolution (640x360) with an in-game\\nGUI open. The mouse cursor can be seen in the center of the image. This particular GUI shows the\\nplayer’s inventory and can be used to craft very basic items. (Middle) We downsample images to\\n128x128 for computational reasons. Shown is a downsampled observation with an in-game GUI for\\ncrafting. This is the resolution used by our models. (Right) A 128x128 observation as seen by our\\nmodels without in-game GUI. The health, hunger, hotbar overlays, and agent hand can be seen in the\\nlower part of the image.\\ngame, in which the server (or the \\\"world\\\") always runs at 20Hz and the client runs as fast as rendering\\ncan complete (typically at 60-100Hz), in our version the client and server run in the same thread\\nat the same frequency. This allows us to run the environment slower or faster than real time, while\\navoiding artifacts like missing chunks of the world. The action and observation spaces are similar\\nto those of MineRL environments and are described in more detail in the following subsections.\\nThe environment also returns diagnostic information, such as in-game stats, contents of the agent’s\\ninventory, whether any in-game GUI is open, etc., which we use for tracking and recording but not as\\ninputs to the models. The episode length is 10 minutes for RL experiments and 60 minutes for BC\\nmodel evaluations. The agent can \\\"die\\\" in a number of ways, such as staying under water for too long\\nand drowning, being killed by hostile mobs, or falling from a tall structure. We do not terminate the\\nepisode on agent \\\"death\\\". Instead, just as for humans in the regular Minecraft game, the agent drops\\nall its items when it dies and respawns at a random spot close to the initial spawning spot in the same\\nMinecraft world. The policy state is not masked on death, so the model can remember the fact that it\\nhas died and act accordingly.\\nC.1\\nObservation space\\nThe environment observations are simply the raw pixels from the Minecraft game that a human\\nwould see. Unlike MineRL, we do not remove overlays like the hotbar, health indicators, and the\\nanimation of a moving hand shown in response to the attack or “use” actions. The field of view is\\n70 degrees, which corresponds to the Minecraft default. GUI scale (a parameter controlling the size\\nof the in-game GUI) is set to 2, and brightness is set to 2 (which is not a Minecraft default, but is\\nvery frequently used in online videos). The rendering resolution is 640x360, which is downsampled\\nto 128x128 before being input to the models. We empirically found 128x128 to be the smallest\\nresolution for which in-game GUI elements are still discernible, and then chose that to minimize\\ncompute costs. Whenever an in-game GUI is open, we additionally render an image of a mouse\\ncursor at the appropriate mouse position to match what a human player’s operating system does (Fig.\\n12).\\nC.2\\nAction space\\nOur action space includes almost all actions directly available to human players, such as keypresses,\\nmouse movements, and clicks. The specific binary actions we include are shown in Table 3.\\nOne difference between the human action space and our agent’s is that we disallow typing arbitrary\\nletters, which is only useful for entering text into the search bar of the crafting recipe book. Humans\\ncan either do that or browse the recipe book with the mouse, the latter of which our agent can still\\ndo. However, because we do allow the agent to press letters that are also shortcuts for actions (e.g.\\noutside of the GUI, the \\\"W\\\" key triggers the forward action) agents are able to press a few keys\\nwithin the GUI (W, A, S, D, E, Q) that produce letters if the recipe book search bar is selected. We\\nhave not seen agents attempt to search the recipe book with these letters. Instead, our agents navigate\\nthe recipe book with the mouse or craft by dragging items around the crafting window.\\n21\\n\\n\\nAction\\nHuman action\\nDescription\\nforward\\nW key\\nMove forward.\\nback\\nS key\\nMove backward.\\nleft\\nA key\\nStrafe left.\\nright\\nD key\\nStrafe right.\\njump\\nspace key\\nJump.\\ninventory\\nE key\\nOpen or close inventory and the 2x2 crafting grid.\\nsneak\\nshift key\\nMove carefully in current direction of motion. In the\\nGUI it acts as a modifier key: when used with attack\\nit moves item from/to the inventory to/from the hot-\\nbar, and when used with craft it crafts the maximum\\nnumber of items possible instead of just 1.\\nsprint\\nctrl key\\nMove fast in the current direction of motion.\\nattack\\nleft mouse button\\nAttack; In GUI, pick up the stack of items or place the\\nstack of items in a GUI cell; when used as a double\\nclick (attack - no attack - attack sequence), collect all\\nitems of the same kind present in inventory as a single\\nstack.\\nuse\\nright mouse button\\nPlace the item currently held or use the block the player\\nis looking at. In GUI, pick up the stack of items or place\\na single item from a stack held by mouse.\\ndrop\\nQ key\\nDrop a single item from the stack of items the player\\nis currently holding. If the player presses ctrl-Q then\\nit drops the entire stack. In the GUI, the same thing\\nhappens except to the item the mouse is hovering over.\\nhotbar.[1-9]\\nkeys 1 – 9\\nSwitch active item to the one in a given hotbar cell.\\nTable 3: Binary actions included in the action space. https://minecraft.fandom.com/wiki/\\nControls has more detailed descriptions of each action.\\nIn addition to the binary (on/off) keypress actions, our action space also includes mouse movements.\\nAs with human gameplay, when in-game GUIs are not open, mouse X and Y actions change the\\nagent’s yaw and pitch, respectively. When a GUI is open, camera actions move the mouse cursor.\\nMouse movements are relative (i.e. they move the mouse or camera relative to the current position,\\nand thus their effect depends on the current position).\\nInventory interaction in Minecraft requires fine-grained mouse movements to achieve tasks such as\\ncrafting and smelting, while mining and navigating the world can be achieved with coarser mouse\\naction. To be able to achieve both with the same action space, we implemented mouse movements\\nas a set of discrete actions with foveated binning along each axis (Fig. 13), which in preliminary\\nexperiments we found to improve crafting performance.\\nD\\nInverse Dynamics Model Training Details\\nD.1\\nIDM Architecture\\nThe IDM model has approximately 0.5 billion trainable weights. The input to the IDM is 128\\nconsecutive image frames (128 frames of video), each of which has dimensions 128 × 128 × 3. The\\nIDM is tasked with predicting the action at each frame. All image pixel values are first divided\\nby 255.0 such that they lie within the range [0, 1]. The first layer of the IDM is a 3-D convolution\\nwith 128 learnable filters with a temporal kernel width of 5 and spatial kernel widths of 1. This\\nconvolution is non-causal, meaning that embeddings at time index t are functions of pixel values at\\ntimes t −2, t −1, t, t + 1, and t + 2. We found this layer to be extremely important in IDM training\\n22\\n\\n\\n−80 −60 −40 −20\\n0\\n20\\n40\\n60\\n80\\nMouse movement (pixels)\\n−10\\n−5\\n0\\n5\\n10\\nCamera angle (deg)\\n0\\n2\\n4\\n6\\n8\\n10\\nCamera bin\\nFigure 13: Relative camera angle or mouse movement in pixels vs. action bin. The same binning is\\nused for both X and Y coordinates. The binning is foveated, meaning that binning is more fine-grained\\nfor smaller movements and more coarse-grained for larger movements. There are 11 bins for each\\naxis (X and Y). The center of each bin (indicated with green circles) is used when un-discretizing\\nmovements (that is, when converting from an action expressed as a bin to a camera angle or mouse\\nmovement).\\nas it incorporates neighboring temporal information immediately, and we show results comparing\\nIDM performance with and without this layer in Figure 14. This comparison was made on the default\\n(1962-hour) IDM dataset.\\n0\\n5\\n10\\n15\\n20\\n25\\n30\\n35\\nTraining Progress (Epoch)\\n0.5\\n1.0\\n1.5\\n2.0\\n2.5\\n3.0\\nLoss\\nWith 3D Conv\\nNo 3D Conv\\n0\\n5\\n10\\n15\\n20\\n25\\n30\\n35\\nTraining Progress (Epoch)\\n0.0\\n0.2\\n0.4\\n0.6\\n0.8\\nKeypress Accuracy\\nWith 3D Conv\\nNo 3D Conv\\n0\\n5\\n10\\n15\\n20\\n25\\n30\\n35\\nTraining Progress (Epoch)\\n6\\n5\\n4\\n3\\n2\\n1\\n0\\n1\\nMouse R2\\nWith 3D Conv\\nNo 3D Conv\\nFigure 14: Effect of 3-D Convolution in the IDM Architecture.\\nThis initial temporal convolutional layer is followed by a ResNet62 image processing network. In this\\npart of the model, no extra temporal information is shared between neighboring frames; however,\\nsince each frame was first processed with the temporal convolution, some temporal information\\nis present at this stage. The ResNet image processing network is comprised of three subsequent\\nstacks with widths W = {64, 128, 128}. Each stack is comprised of, in order, (1) an initial 3x3\\nconvolutional layer with 1-pixel zero padding at the embedding boundary (such that the outgoing\\nembedding dimensions are the same as the incoming embedding dimension) with W output channels,\\n(2) a 3x3 max pooling with stride 2 and padding 1 such that the embedding width and height are\\nhalved, and (3) two classic ResNet blocks as defined in He et al. 62 with each layer also having W\\noutput channels.\\nThe output of the ResNet stack is flattened into a 1-dimensional vector of size 217 = 131072 (one\\nvector for each frame in the video) such that at this stage there are 128 vectors of size 131072. Each\\nvector is independently processed with two frame-wise dense layers with 256 output activations and\\nthen 4096 output activations, respectively. The result is then fed through 4 subsequent non-causal\\n(umasked) residual transformer69 blocks. Each block first has an unmasked attention layer, i.e. frames\\nmay attend to future frames, with 32 attention heads of dimension 128 each and a surrounding residual\\nconnection that skips this layer. The embedding is then passed through a frame-wise dense layer\\nwith output dimension 16384 and another with output dimension returning to 4096; a single residual\\nconnection skips past this pair of frame-wise dense layers (not skipping past each layer separately,\\nbut skipping the pair). All dense layers have their weights tied through time, so each frame in the\\nvideo is processed with the same weights.\\nFinally, independent dense layer heads for each action are pulled from the final embedding – a 2\\nclass on/off categorical parameterized with a softmax for each available key as well as a 11-way\\n23\\n\\n\\ncategorical for both the discretized horizontal and vertical mouse movements (See Appendix C.2 for\\ndetails on the action space).\\nEach dense layer or convolutional layer in the network is preceded by a layernorm70 and followed by\\na ReLU non-linearity. Weights are initialized with Fan-In initialization71 and biases are initialized to\\nzero.\\nD.2\\nIDM Training\\nThe total loss for the network is the sum of each independent action prediction loss (one for each\\nkey and one for both mouse directions). Each independent loss is the negative log-likelihood of the\\ncorrect action. We use the ADAM72 optimizer with a linear learning rate decay. We use an initial\\nlearning rate of 0.003, a batch size of 128 (where each item in the batch is a video sequence of 128\\nframes), and a weight decay of 0.01. Hyperparameters were tuned in preliminary experiments. The\\nIDM is trained on our contractor collected dataset for 20 epochs. This took 4 days on 32 A100 GPUs.\\nWe add data augmentation to each video segment; augmentations are randomly sampled once per\\nsegment such they are temporally consistent. Using the Pytorch73 transforms library, we adjust the\\nhue by a random factor between -0.2 and 0.2, saturation between 0.8 and 1.2, brightness between\\n0.8 and 1.2, and contrast between 0.8 and 1.2. We also randomly rotate the image between -2 and 2\\ndegrees, scale it by a random factor between 0.98 and 1.02, shear it between -2 and 2 degrees, and\\ntranslate it between -2 and 2 pixels in both the x and y dimensions.\\nDue the large computational cost of running all of the experiments in this paper, training results\\nare from one run of training (for IDM, BC, and RL training): this non-ideal situation is mitigated\\nbecause deep learning training tends to be low variance74,75 and because we often have data points\\nfrom sweeps (e.g. on dataset size) that suggest overall trends.\\nD.3\\nGenerating Pseudo Labels with the IDM\\nSection 4.1 shows that inverse dynamics modeling is a much easier task than behavioral cloning\\nbecause IDMs can be non-causal. The IDM is trained to simultaneously predict all 128 actions for\\neach video sequence, so the IDM will effectively be causal for frames at the end of the video clip\\nbecause future frames are not included in the sequence. For this reason, we apply the IDM over a\\nvideo using a sliding window with stride 64 frames and only use the pseudo-label prediction for\\nframes 32 to 96 (the center 64 frames). By doing this, the IDM prediction at the boundary of the\\nvideo clip is never used except for the first and last frames of a full video.\\nE\\nFoundation Model Behavioral Cloning\\nE.1\\nFoundation Model Architecture\\nThe behavioral cloning model architecture is the same as the IDM architecture described in Appendix\\nD.1 except that we modify the architecture so that it is causal (i.e. cannot see the future when making\\npredictions). This means the BC architecture does not have the initial non-causal convolution the\\nIDM has (this layer is omitted completely). Furthermore, the residual transformer layers are now\\ncausally masked (as is standard in language modeling) and we do Transformer-XL-style76 training\\nwhere frames can attend to keys and values from past batches within the same video. We also use a\\nTransformer-XL-style relative attention position embedding.\\nE.2\\nNull Action Filtering\\nThe most common action humans take is the null action (no keypresses or mouse movements), which\\naccounts for 35% of all actions they take. Among other reasons, a player may take the null action to\\nwait for something in the game to finish, to pause between actions, or to take a break to grab a glass\\nof water. Early on in the project we found that the BC model would take a much larger fraction than\\n35% of null actions, often upwards of 95%. In order to prevent this behavior we removed frames with\\nnull actions from the dataset. We compare a few different treatments: we filter nulls if there have\\nbeen 1, 3, or 21 frames of consecutive null actions, and include a treatment that does not perform\\nany null filtering. Null action filtering generally helps, increasing all crafting rates (Figure 15 left).\\n24\\n\\n\\nbasic mining\\nlogs\\nplanks\\ncrafting tables\\ntotal crafting\\nnull actions\\n10 2\\n100\\n102\\n104\\nEffect of Null Action Filtering\\nNone\\nAll\\nGroups of 3\\nGroups of 21\\nbasic mining\\nlogs\\nplanks\\ncrafting tables\\ntotal crafting\\nnull actions\\n10 3\\n10 2\\n10 1\\n100\\n101\\n102\\n103\\nJoint Hierarchical vs Factored Action Spaces\\nFactored\\nHierarchical Joint\\nFigure 15: (Left) Effect of Null Action Filtering during training. We compare environment metrics\\nand number of sampled null action during rollouts (rightmost group of columns) for the following\\ntreatments: no null action filtering (blue), filtering all null actions (green), filtering only groups of\\n3 or more null actions (red), and filtering only groups of 21 or more null actions (purple). (Right)\\nHierarchical versus Factored Action Spaces.\\nFiltering only groups of 3 performed slightly better than filtering all null action or groups of 21. Initial\\nexperiments indicated that filtering all null actions was better; however, after further model tuning\\nand after we had already trained our largest models, we found that filtering only groups of 3 or more\\nnull actions performed best. Due to compute constraints we were not able to redo all experiments\\nwith this setting, but doing so would be a reasonable choice for any future work.\\nE.3\\nJoint Hierarchical Action Space\\nWe originally worked with a factored action space, where each keypress could be independently on\\nor off, and this choice was independent of whether the mouse was being moved. This could cause\\nissues for modeling the human behavior distribution exactly. Say for a given state, humans either with\\n50% probability (a) move forward and attack or with 50% probability (b) move left and drop their\\nitem. The best a factored distribution can do is to assign 50% probability to each of the 4 constituent\\nactions because it chooses to press each button simultaneously and independently. See Appendix C.2\\nfor details on the entire action space.\\nFor this reason, we implemented a joint distribution over actions; however, the full joint distribution\\nover 20 binary buttons and two mouse movement dimensions discretized into 11 bins each would\\nresult in in 220 × 112 ≈1.2 × 108 possible combinations. This is far too large for many reasons, e.g.\\nthe final layer from the transformer stack with a dimension of 4096 would need to be mapped to each\\ncombination resulting in 4096 × 1.2 × 108 ≈5.2 × 1011 parameters for this final layer alone. In\\norder to reduce this we noted that many buttons in Minecraft have no effect when simultaneously\\npressed; for example, if a player tries to move forward and backward at the same time, they remain in\\nplace. Below we list the the sets of mutually exclusive actions. Furthermore, the inventory button is\\nexclusive with all other buttons and mouse movement.\\nMutually Exclusive Actions\\nforward, back\\nleft, right\\nsprint, sneak\\nhotbar.[1-9]\\nEven reducing the joint action space to reflect these mutually exclusive combinations still results in a\\nhuge action space when combined with the discretized mouse movements, i.e. 33×10×24×112+1 ≈\\n5.2 × 105. This calculation results from 33 for the 3 sets of 2 mutually exclusive keys above where\\ntaking neither in the set is an option, ×10 for the 9 hotbar keys or no hotbar keypress, ×24 for the\\nremaining binary 4 keys: use, drop, attack, and jump, ×112 for mouse movements, and finally\\n+1 for the inventory button which is mutually exclusive with all other actions. ∼5.2 × 105 is still\\nquite large so we chose to implement a hierarchical binary action for camera being moved or not. If\\nthis action is on, then there is a secondary discrete action head with 121 classes (the joint distribution\\nof mouse movements because each discretized mouse direction has 11 bins) that determines where\\n25\\n\\n\\nto move the mouse. If the hierarchical action is off, then there is no mouse movement, loss for\\nthe secondary mouse movement action is masked during training, and the secondary action head\\nneed not be sampled during evaluations. While this no longer models the full joint distribution, it is\\nquite a bit better than the factored action space since dependencies between keypresses as well as\\nwhether or not to move the mouse (although not which mouse movement) are modeled jointly. The\\nresulting action space has dimension 33 × 10 × 24 × 2 + 1 = 8461 (the 112 dimensional multiplier\\nfor camera movement has been replaced by a multiplier of 2 here, corresponding to a binary action\\nfor whether or not to move the mouse) with an additional 121-dimension head for the joint camera\\nmovements. In the future it would be interesting to implement sequential conditional action spaces to\\nmore completely model the joint distribution.\\nIn Figure 15 (right) we compare environment rollout performance between BC models with the\\nhierarchical joint action space and with the factored action space. Environment statistics are fairly\\ncomparable; however, we see that the factored action space model samples far more null actions.\\nThis is an important example of the factored action space failing to correctly model the distribution in\\nthe dataset because, due to null action filtering, there are 0 null actions in the dataset these models\\ntrain on. Despite this, the factored model samples many null actions because the prediction for each\\nkey is not conditioned on other keypresses.\\nE.4\\nFoundation Model Training\\nThe foundation model training is similar to the IDM training, with the exception of labels being\\nIDM-generated pseudo labels. The hyperparameters used for foundation model training are listed in\\nTable 4.\\nHyperparameter\\nValue\\nLearning rate\\n0.002147\\nWeight decay\\n0.0625\\nEpochs\\n30\\nBatch size\\n880\\nTable 4: Hyperparameters for foundation model training\\nF\\nBehavioral Cloning Fine-Tuning\\nBehavior cloning fine-tuning is similar to the foundation model training, except we either use a focused\\nsubset of all the videos (early_game dataset, described in A.3) with pseudo labels, or contractor data\\n(contractor_house dataset, described in B.4) with ground-truth labels. The hyperparameters used\\nfor behavior cloning fine-tuning are listed in Table 5. We used 16 A100 GPUs for about 6 hours when\\nfine-tuning on contractor_house dataset, and 16 A100 GPUs for about 2 days when fine-tuning\\non early_game dataset.\\nHyperparameter\\nValue\\nLearning rate\\n0.000181\\nWeight decay\\n0.039428\\nEpochs\\n2\\nBatch size\\n16\\nTable 5: Hyperparameters for behavior cloning fine-tuning\\nG\\nReinforcement Learning Fine-Tuning\\nG.1\\nReinforcement Learning Fine-Tuning Training Details\\nRL experiments were performed with the phasic policy gradient (PPG) algorithm,64 an RL algorithm\\nbased on the proximal policy optimization (PPO) algorithm77 that increases sample efficiency by\\nperforming additional passes over the collected data to optimize the value function as well as an\\n26\\n\\n\\nauxiliary value function. These algorithms have been described extensively in previous work,64,77\\nso here we describe them only briefly. A major inefficiency when training on-policy algorithms is\\nthat, to remain on-policy, one can only take a single gradient step before new rollout data needs\\nto be gathered to continue optimization. To alleviate the potentially destructive effects of taking\\nmultiple optimization steps in a single iteration, PPO prevents the policy from changing too much\\nin a single step by clipping the loss when the difference between the current policy and the policy\\nbefore the update becomes too large.77 We also use generalized advantage estimation (GAE), which\\ncan speed-up credit assignment by looking more than 1 step into the future when determining the\\nadvantage of an action, with the look-ahead being determined by hyperparameter λ.78\\nPPG improves the sample efficiency of PPO when the policy and value function share the same\\nnetwork by following different optimization processes for the policy, the value function, and their\\nshared representation. PPG splits optimization in two phases: a wake phase and a sleep phase. In the\\nwake phase, the policy and value function are optimized as in normal PPO training, with the only\\nexception being that every sample is used at most once, which prevents the policy from overfitting on\\nthese samples. In the sleep phase PPG optimizes the value function and an auxiliary value function\\n(which is optimized with the exact same loss as the regular value function, but its output is never used\\nduring training), while keeping a Kullback-Leibler (KL) divergence loss to the policy before the start\\nof the sleep phase to ensure that the policy does not change. Because the policy is not optimized\\nin this step, PPG does allow samples to be reused multiple times in this phase. The assumption\\nbehind optimizing the value function during the sleep phase is that value function optimization is\\nless sensitive to being trained multiple times on the same sample. Optimizing the auxiliary value\\nfunction does not directly affect either the value function or the policy, but it can improve the shared\\nrepresentation of both functions (the assumption being that predicting the value-function requires\\nencoding all features that are important for distinguishing states). The coefficients for the three\\nlosses (value function loss, auxiliary value function loss, and KL loss) are listed in Table 6. In our\\nexperiments a single iteration consists of two sleep cycles and one wake cycle.\\nBecause the value and auxiliary value functions are not optimized during BC pre-training, they\\nare initialized at the start of RL fine-tuning. Each value function is implemented as a single, fully\\nconnected layer on top of the last residual transformer block of the pretrained model (Appendix D.1).\\nThe weights of the auxiliary value function are randomly initialized while the weights of the regular\\nvalue function are initialized with zero weights, which appeared to prevent destructive updates early\\nin training that could happen with a randomly initialized value function. To prevent the value-function\\nloss from having gradients that depend greatly on the magnitude of the reward, we normalize the\\nvalue-function target by subtracting the mean and dividing by the standard deviation, which are\\nestimated through an exponentially weighted moving average.\\nTo prevent catastrophically forgetting the skills of the pretrained network when RL fine-tuning, we\\napply an auxiliary KL divergence loss between the RL model and the frozen pretrained policy.10 This\\nloss is defined as:\\nLklpt = ρKL(πpt, πθ)\\n(2)\\nWhere πθ is the the policy being trained, πpt is the frozen pretrained policy, KL(πpt, πθ) is the\\nKullback-Leibler divergence between the policy being trained and the pretrained policy, and ρ is a\\ncoefficient to weight this loss relative to other losses.\\nIn the fine-tuning experiments, this KL divergence loss replaces the common entropy maximization\\nloss, which is often added to RL experiments to encourage exploration.79,80 The idea behind entropy\\nmaximization is that, when all actions appear to have equal value, such as when the agent has not\\nlearned about the next reward, it should maximize its entropy to increase the chance that it discovers\\nthe next reward. Blindly exploring by maximizing entropy is effective when the state and action\\nspaces are sufficiently small or the reward is sufficiently dense, but becomes infeasible when the\\nstate and action spaces are large and rewards are sparse, which is the case in the diamond-pickaxe\\ntask. Instead of blindly exploring through uniform-random actions, we assume that the pretrained\\npolicy has an action distribution that is much more likely to take sequences of actions that lead\\nto interestingly new states, and thus, in states where the agent assigns equal value to each of its\\nactions, it should mimic the action-distribution of the pretrained policy instead of a uniform-random\\naction distribution. In experiments with a randomly initialized policy we do include the entropy\\nmaximization loss with a coefficient of 0.01, which has been an effective setting in other Minecraft\\nwork.30 Empirically, we found that a high coefficient ρ for this KL divergence loss would prevent\\nthe agent from properly optimizing the reward function while a low coefficient ρ was ineffective at\\n27\\n\\n\\nHyperparameter\\nValue\\nLearning rate:\\n2 × 10−5\\nWeight decay:\\n0.04\\nBatch size:\\n40\\nBatches per iteration:\\n48\\nContext length:\\n128\\nDiscount factor (γ):\\n0.999\\nGAE λ:\\n0.95\\nPPO clip:\\n0.2\\nMax Grad norm:\\n5\\nMax Staleness:\\n2\\nPPG sleep cycles:\\n2\\nPPG sleep value-function coefficient:\\n0.5\\nPPG sleep auxiliary value-function coefficient:\\n0.5\\nPPG sleep KL coefficient:\\n1.0\\nPPG sleep max Sample Reuse:\\n6\\nKL divergence coefficient ρ:\\n0.2\\nCoefficient ρ decay:\\n0.9995\\nTable 6: Hyperparameters for RL experiments. These are the hyperparameters for all treatments with\\ntwo exceptions. First, when fine-tuning from the early-game model without a KL divergence loss,\\nin addition to the KL divergence loss being set to 0, the learning rate was set to 3 × 10−6 (the best\\nsetting out of a sweep over 5 different learning rates), as we found that performance was substantially\\nlower with the standard learning rate of 2 × 10−5 and the agent did not even learn to collect logs.\\nWe suspect that the reason that the learning rate needed to be lowered when fine-tuning without a\\nKL loss is that the KL loss prevents making optimization steps that change the policy too much in a\\nsingle step, especially in early iterations when the value function has not been optimized yet, and\\nthe KL loss thus makes it possible to optimize with a higher learning rate. Second, when running\\nRL from a randomly initialized policy there is no KL divergence loss or KL divergence decay, but\\ninstead we use an entropy bonus of 0.01, which reportedly worked well in previous work.30\\nprotecting the learned skills of the pretrained policy and preventing catastrophic forgetting. As such,\\nwe start with a relatively high coefficient ρ and decay it by a fixed factor after each iteration (Table 6).\\nThis method protects policy skills in early iterations while guaranteeing that the policy can eventually\\nmaximize the reward function, regardless of how different its behavior has to be to do so relative to\\nthe pretrained policy.\\nFor the reward function we estimated the rough quantities of each item that a human player might\\ngather when trying to craft a diamond pickaxe, and we reward the model for gathering up to that\\nquantity for each item. We started these estimates by iterating over the technology tree backward from\\na diamond pickaxe and adding the requirements for each item to the reward function (e.g. first we\\nadded a diamond pickaxe to the reward function, then we added the 3 diamonds and 2 sticks required\\nfor crafting a diamond pickaxe, then we added the 1 iron pickaxe required for mining diamonds,\\nand so on). Then we added coal and torches to the reward function, with coal being useful as fuel\\nwhen smelting iron and for crafting torches while the torches themselves improve visibility and\\nprevent enemies from spawning. Finally, we reward the model for bringing additional logs (5 logs are\\nrequired to craft all items in the reward function, but we reward up to 8 logs), which can be used as\\nfuel or crafted into a crafting table or sticks if the agent runs out. In practice the agent rarely collects\\nthe additional logs, places the torches, or uses coal as fuel when smelting, but the reward function\\nwas based on human expectations on what would be useful to execute this task, rather than designed\\naround how an RL model behaves after training. Finally, to encourage the agent to keep mining\\ndiamonds and crafting diamond pickaxes after it has crafted its first diamond pickaxe, we did not put\\na limit on the number of diamonds or diamond pickaxes that would be rewarded.\\nThe rewards for the different items are separated into 4 tiers, roughly depending on how late a player\\nwould usually get the relevant item. The first tier consists of all wooden and stone items and has a\\nbase reward of 1, the second tier consists of all items requiring coal with a base reward of 2, the third\\ntier consists of all items requiring iron with a base reward of 4, and the final tier is diamond with a\\nbase reward of 8. Thus items later in the sequence of items towards a diamond pickaxe generally\\n28\\n\\n\\nItem\\nQuantity rewarded\\nReward per item\\nLog\\n8\\n1/8\\nPlanks\\n20\\n1/20\\nStick\\n16\\n1/16\\nCrafting table\\n1\\n1\\nWooden pickaxe\\n1\\n1\\nCobblestone\\n11\\n1/11\\nStone pickaxe\\n1\\n1\\nFurnace\\n1\\n1\\nCoal\\n5\\n2/5\\nTorch\\n16\\n1/8\\nIron ore\\n3\\n4/3\\nIron ingot\\n3\\n4/3\\nIron pickaxe\\n1\\n4\\nDiamond\\ninf\\n8/3\\nDiamond pickaxe\\ninf\\n8\\nTable 7: Reward per item and total quantity rewarded.\\ngive a higher reward. To make sure that the agent does not over-value items that are supposed to\\nbe gathered in bulk (e.g. the agent is rewarded for up to 20 planks but only up to 1 crafting table,\\nwhich can cause the agent to focus on planks at the expense of creating a crafting table), we divide\\nthe base reward of each item by the total quantity that the agent gets rewarded for (for the purpose\\nof determining the reward, the total quantity for diamonds is 3 and the total quantity for diamond\\npickaxes is 1, even though we did not put a limit on the number of these items being rewarded). For\\nexample, the agent is rewarded for 3 iron ore, which has a base reward of 4 for being in the iron tier\\nand up to 3 blocks of iron ore are rewarded, thus the reward per block of iron ore is 4/3. The quantity\\nand reward for each item are listed in Table 7.\\nWhile every item in the sequence towards a diamond pickaxe is rewarded, the reward function is still\\nsparse and, in some cases, even deceptive. The sparsity comes from the fact that it can take thousands\\nof actions to find the next reward, even after the agent has acquired all the necessary prerequisites\\n(e.g. human players often take more than 10,000 actions to find a diamond after crafting an iron\\npickaxe). The reward function can be deceptive when the most efficient method for getting one item\\ncan make it far more difficult to get the next item. For example, a good strategy for the agent to\\ncraft a stone pickaxe quickly is to mine (i.e. spend a few seconds to pick up) its crafting table after\\ncrafting a wooden pickaxe, such that the agent has immediate access to a crafting table as soon as it\\nhas collected enough cobblestone. However, the fastest way to get a reward for gathering cobblestone\\nis to mine down immediately after crafting a wooden pickaxe, while leaving the crafting table behind.\\nThus following the optimal strategy for gathering cobblestone makes it more difficult to learn to craft\\na stone pickaxe.\\nExperiments ran for approximately 6 days (144 hours) on 80 GPUs (for policy optimization) and\\n56,719 CPUs (mostly for collecting rollouts from Minecraft). In this time the algorithm performed\\nroughly 4,000 optimization iterations and collected roughly 1.4 million Minecraft episodes consisting\\nof 12,000 frames each, for a total of 16.8 billion frames.\\nG.2\\nReinforcement Learning Fine-Tuning Additional Data\\nAdditional figures that are helpful for understanding the main results of the RL fine-tuning experiments\\nare presented in this section. First, we show the items-over-training figure when RL fine-tuning from\\nthe early-game model without a KL loss (Fig. 16). When training without a KL loss, the model\\nonly learns to obtain the four items that the early-game model is capable of getting zero-shot, which\\nare logs, planks, sticks, and crafting tables. Second, we present preliminary experiments in which\\nwe directly compare RL fine-tuning from the house-building model and RL fine-tuning from the\\nearly-game model (Fig. 17). These experiments differ from the main experiments in that, for both\\ntreatments shown here, the KL loss coefficient was set to 0.4, the learning rate was set to 6 × 10−5,\\nand the reward for each item was 1/quantity for all items (i.e. items closer to the diamond pickaxe\\ndid not have an increased reward). While RL fine-tuning from the house-building model initially\\n29\\n\\n\\nworked better than RL fine-tuning from the early-game model, fine-tuning from the early-game model\\nworked better after 800,000 episodes and showed signs of smelting iron ingots, which is why the\\nearly-game model was chosen for the main experiments.\\n0.0\\n0.2\\n0.4\\n0.6\\n0.8\\n1.0\\n1.2\\n1.4\\nEpisodes\\n1e6\\n0\\n20\\n40\\n60\\n80\\n100\\n% episodes\\nNo KL-loss\\nFigure 16: Items obtained when RL fine-tuning from the early-game model without a KL loss. The\\nmodel learns to obtain all items that the early-game model can craft zero-shot, which are logs, planks,\\nsticks, and a crafting table. In contrast to the treatment with a KL-penalty, it does not learn any items\\nbeyond these initial four, likely because skills that are not performed zero-shot, and for which the\\nmodel thus does not initially see any reward, are catastrophically forgotten while the first four items\\nare learned.\\n0\\n100000\\n200000\\n300000\\n400000\\n500000\\n600000\\n700000\\n800000\\nEpisodes\\n0\\n2\\n4\\n6\\n8\\nReward\\nReward over episodes\\nRL from Early-Game model\\nRL from House-Builing model\\n0\\n100000\\n200000\\n300000\\n400000\\n500000\\n600000\\n700000\\n800000\\nEpisodes\\n0.000\\n0.005\\n0.010\\n0.015\\n0.020\\n0.025\\n0.030\\n0.035\\nIron Ingots Obtained\\nIron Ingots Obtained Per Episodes\\nRL from Early-Game model\\nRL from House-Builing model\\nFigure 17: Preliminary experiments when RL fine-tuning from the early-game model compared to\\nRL fine-tuning from the house-building model. (Left) While reward initially increases faster when\\nfine-tuning from the house-building model, fine-tuning form the early-game model eventually obtains\\na slightly higher reward. (Right) RL fine-tuning from the early-game model has a higher likelihood\\nof smelting an iron-ingot, which is why the early-game model was chosen for future RL fine-tuning\\nexperiments.\\nH\\nFoundation Model Scaling\\nIn early experiments we found that increasing model size led to models staying in the efficient\\nlearning regime longer into training.63 Here we compare the 0.5B model described in Section 4.2 to\\nboth a 248M and 71M parameter model. Both of these models are trained for 15 epochs as compared\\nto the 30 epochs the 0.5B model trained for. These models have the same architecture as the 0.5B\\nmodel but each layer in the 248M parameter model has 1/2 the width and each layer in the 71M\\nparameter model 1/3 the width. The 71M model was trained with an initial learning rate of 0.001586,\\nbatch size of 480, and weight decay of 0.044506. The 248M model had an initial learning rate of\\n0.001831, batch size of 640, and weight decay of 0.051376.\\nIn Figure 18 we show validation loss on web_clean with IDM pseudo-labels, loss on the contractor\\ndataset used to train the IDM with ground truth labels collected during contractor play, and zero-shot\\nenvironment performance for the 71M, 248M, and 0.5B models. While larger models have better\\nvalidation loss on web_clean, these results do not tell the clear story that the 0.5B model is better\\nthan its smaller counterparts. The 71M model has the lowest contractor dataset loss while having the\\nhighest web_clean loss, and it also has the best zero-shot environment performance. In fact, we see\\nthat the 71M model even had non-zero wooden tool crafting (Fig. 18 bottom left). The 248M model\\nalso appears to be better at crafting than the 0.5B, and also has lower contractor dataset loss.\\nWhile the zero-shot results suggest smaller models are better, fine-tuning tells another story. When\\nfine-tuning to contractor_house, model size rank ordering reverses and now the 0.5B model\\nperforms best both in validation loss (Fig. 19 left) and in environment performance (Fig. 19 right)\\n30\\n\\n\\n10 3\\n10 2\\n10 1\\n100\\n101\\n~Compute\\n2.5\\n4\\n6\\n10\\nWeb Clean Loss\\nLoss Web Clean Validation Dataset\\n71M\\n248M\\n0.5B\\n10 3\\n10 2\\n10 1\\n100\\n101\\n~Compute\\n2.5\\n4\\n6\\n10\\nIDM Contractor Dataset Loss\\nLoss on IDM Contractor Dataset\\n71M\\n248M\\n0.5B\\nbasic\\nmining\\nlogs\\nplanks\\ncrafting\\ntables\\ntotal\\ncrafting\\n10 2\\n10 1\\n100\\n101\\nCollected or Crafted\\nZero-Shot Performance vs Model Size\\n71M\\n248M\\n0.5B\\n0\\n2\\n4\\n6\\n8\\n10\\n12\\n14\\n16\\nTraining Progress (Epoch)\\n0\\n10 3\\n10 2\\n10 1\\n100\\n101\\nCrafting or Collection\\nZero-Shot Performance over Training\\n(71M Parameter Model)\\nbasic mining\\nlogs\\nplanks\\ncrafting tables\\ntotal crafting\\nwooden tools\\n0\\n2\\n4\\n6\\n8\\n10\\n12\\n14\\n16\\nTraining Progress (Epoch)\\n0\\n10 3\\n10 2\\n10 1\\n100\\n101\\nZero-Shot Performance over Training\\n(248M Parameter Model)\\n0\\n5\\n10\\n15\\n20\\n25\\nTraining Progress (Epoch)\\n0\\n10 3\\n10 2\\n10 1\\n100\\n101\\nZero-Shot Performance over Training\\n(0.5B Parameter Model)\\nFigure 18: Training and Zero-Shot Performance versus Model Scale. In the first two plots the x-axis\\nis compute normalized to that used by the 71M parameter model, such that after 15 epochs of training\\nthe 71M model has used 1 \\\"compute\\\". The 248M parameter model and the 71M model are trained on\\nthe same amount of data (15 epochs), and the 0.5B parameter model is trained on 30 epochs of data.\\n(Top Left) Loss on the web_clean validation dataset. (Top Middle) Loss on the IDM contractor\\ndataset; note that these models were trained only on web_clean and not on any contractor data.\\n(Top Right) Zero-shot environment rollout performance at the end of training. (Bottom) Zero-shot\\nenvironment rollout performance over training for the 71M model (bottom left), 248M model (bottom\\nmiddle), and 0.5B model (bottom right).\\nbasic\\nmining\\nlogs\\nplanks crafting\\ntables\\ntotal\\ncrafting\\nwooden\\ntools\\nstone\\ntools\\n10 3\\n10 2\\n10 1\\n100\\n101\\n102\\n103\\n104\\nCollected or Crafted\\nFine-Tuning to Contractor House Dataset\\n71M\\n248M\\n0.5B\\n0.0\\n0.5\\n1.0\\n1.5\\n2.0\\n2.5\\n3.0\\nFine-Tuning Epoch\\n2.2\\n2.4\\n2.6\\n2.8\\n3.0\\ncontractor_house Validation Loss\\nLoss on contractor_house\\n71M\\n248M\\n0.5B\\n0.0\\n0.5\\n1.0\\n1.5\\n2.0\\n2.5\\n3.0\\nFine-Tuning Epoch\\n2.4\\n2.5\\n2.6\\n2.7\\n2.8\\n2.9\\n3.0\\nLoss on IDM Contractor Dataset\\nLoss on full IDM Contractor Dataset\\n71M\\n248M\\n0.5B\\nFigure 19: contractor_house fine-tuning performance versus model size. (Left) Loss on the\\ncontractor_house holdout validation set. (Middle) Loss on the full contractor dataset collected\\nto train the IDM; this dataset is disjoint from contractor_house. (Right) Environment rollout\\nperformance at the end of fine-tuning.\\nfollowed by the 248M model and then the 71M model. Environment model rollouts are performed\\nusing the same game engine that we use to collect contractor data, which could be visually distinct\\nfrom videos taken from the web. It is plausible that the larger models overfocus on the visual\\npeculiarities in web data during pretraining since they have worse contractor data loss (Fig.18 top\\nmiddle), and this causes them to perform more poorly in the environment zero-shot. However, we\\nhypothesize that because the contractor_house dataset we fine-tune to is collected from our game\\nengine, the larger models that are a better overall Minecraft prior (as indicated by lower web_clean\\nvalidation loss in Fig.18 top left) can quickly shift their low level features to perform better on data\\ncoming from our game engine, resulting in better environment rollout performance. This hypothesis\\nis further supported by Fig. 19 (middle) showing loss on the contractor dataset collected for IDM\\ntraining, which has no overlap with contractor_house. After just a few steps of fine-tuning to\\ncontractor_house, all models quickly improve in loss on the full IDM contractor dataset, with\\nlarger models now performing best. While not conclusive, we believe this investigation provides\\nsome intuition for future studies of model scaling for sequential decision making problems.\\n31\\n\\n\\nI\\nText Conditioning\\nGoal-conditioned policies81,82 make it possible for a single agent to perform a wide variety of\\ngoals in a single environment, which is particularly relevant in open-ended environments such as\\nMinecraft. In recent work, goal specification has increasingly taken the form of domain specific\\nlanguages83, or even natural language84,85. The benefits of language-conditioned agents can be\\ntremendous, especially natural-language-conditioned agents, as their goal space contains a wide\\nvariety of potentially very complex tasks. Text conditional models have shown an amazing ability to\\nperform tasks zero-shot (or learn them few-shot) including generalizing in impressive ways via the\\ncompositional and combinatorial possibilities allowed by natural language (e.g. GPT1 and DALL·E\\n286). We hypothesize that we should expect similar capabilities to emerge with natural-language-\\nconditioned virtual agents, if they are similarly trained on enormous amounts of data (that goes from\\na natural language description to a sequence of actions that completes the specified goal). In this\\nsection we take preliminary steps toward that future. Our preliminary experiments provide evidence\\nthat it is possible to pretrain a natural-language-conditioned model for Minecraft using the general\\napproach presented in this paper (VPT) plus conditioning on the speech that often accompanies\\nvideos.\\nIn online videos, the human actor sometimes indicates their intent in their verbal commentary\\n(e.g. “Let’s go chop some trees to make a wooden axe” or “now let’s learn how to crop photos in\\nPhotoshop”). Conditioning on this closed caption data could produce a steerable pre-trained model:\\ni.e., it may later be possible to condition the model with text such as “I am going to craft a wooden\\npickaxe” or “I am going to build a house,” and have the agent perform those tasks specifically rather\\nthan simply follow typical human behavior (as was investigated in the rest of this paper). An alternate\\nway to produce a steerable agent is via RL fine-tuning, which we could have done in Section 4.4\\nby adding a bit indicating the task to be completed, as has been done in prior work30. However,\\nconditioning on natural language offers many benefits over that approach. First, it is flexible and\\npowerful, being able to express any task. Second, one does not need to preconceive of the task to\\nbe completed ahead of time. This would allow for general, capable, zero-shot agents like GPT, but\\nextending those capabilities to embodied tasks such as completing tasks on computers or in simulated\\n3D worlds. Third, text conditioning can be used even when tasks are difficult to specify via reward\\nfunctions (e.g. “Let’s build a house” or–if the agent is capable of doing it–more complex things like\\n“I will now build a castle surrounded by a moat”). In the limit, VPT+text could conceivably produce\\npowerful, capable, natural-language-conditional agents with the powers of GPT to meta-learn, follow\\ninstructions, and complete tasks zero or few shot, but in the form of agents that can act in virtual\\nworlds, complete tasks on computers, and in other similar embodied sequential decision domains.\\nWe do not reach those lofty goals in this work, but we began a first step towards exploring in that\\ndirection.\\nMany Minecraft videos feature audio commentary from the player. This commentary is sometimes\\npresent in the form of closed captions for the videos, or could be extracted post-hoc using automated\\nspeech recognition (ASR).87 Our dataset features about 17k hours of content with associated closed\\ncaptions.\\nWe fine-tuned the 220 million parameter VPT foundation model used in the RL-fine-tuning ex-\\nperiments (chosen vs. 0.5B for the same reason: to reduce compute costs) with an additional\\ntext-conditioning input on the subset of our data for which closed captions are available. To obtain the\\nconditioning input, we first split videos into 30 second chunks. The same text is associated with every\\nframe in a given chunk, and is made up of all the closed captions occurring within that chunk, as well\\nas the line of text preceding and following the chunk (if any). Because the vast majority (around\\n95%) of our closed caption data lacks capitalization and punctuation, it is punctuated using the rpunct\\nlibrary88. We then obtain a text embedding vector of length 4,096 from the OpenAI embedding API89,\\nwhich is processed by a randomly initialized multi-layer perceptron (MLP) with two hidden layers of\\nsize 2,048. The resulting activations are added for each frame to the pretrained model activations\\nbefore the transformer layers (pretransformerActivations += mlp(textEmbedding)). The\\nmodel is fine-tuned for four epochs.\\nOur model shows evidence of steerability. When conditioned on sentences that incite the agent to\\nexplore (such as “I’m going to explore” and “I’m going to find water”) the agent travels significantly\\nfarther from its spawn point (Figure 20a). Additionally, we can steer the agent to preferentially collect\\n32\\n\\n\\ndig\\ndirt\\nexplore\\nhouse\\nseed\\nwater\\nwood\\nConditioning\\n120\\n130\\n140\\n150\\nTravel distance (blocks)\\nTravel Distance with Conditioning\\n(a)\\ndig\\ndirt\\nexplore\\nhouse\\nseed\\nwater\\nwood\\nConditioning\\n0.4\\n0.5\\n0.6\\n0.7\\n0.8\\n0.9\\nWheat seeds collected\\nSeed Collection with Conditioning\\n(b)\\ndig\\ndirt\\nexplore\\nhouse\\nseed\\nwater\\nwood\\nConditioning\\n1.00\\n1.25\\n1.50\\n1.75\\n2.00\\n2.25\\n2.50\\n2.75\\nOak logs collected\\nLog Collection with Conditioning\\n(c)\\ndig\\ndirt\\nexplore\\nhouse\\nseed\\nwater\\nwood\\nConditioning\\n5\\n6\\n7\\n8\\n9\\n10\\nDirt collected\\nDirt Collection with Conditioning\\n(d)\\nFigure 20: Evidence for conditioning. In each plot, the variants expected to stand out are shown in\\nbold. The strings corresponding to each variant are shown in Table 8. Statistics are measured over 5\\nminute episodes. (a) Distance traveled by the agent . Both “explore” and “water” text strings should\\nencourage a steerable agent to move more than when doing other tasks, which is what occurs. Grass\\n(which is needed to get seeds) is not present in all biomes, which is likely why the “seed” condition\\nproduces more travel (as the agent sometimes needs to move to a biome with grass). The travel\\ndistance is the Euclidean distance from the spawn point to the farthest point the agent reached during\\nthe episode on the horizontal (x-z) plane. (b) Collection of wheat seeds. The “seed” variant collects\\nsubstantially more than other variants, as expected of a steerable agent. (c) Collection of oak (the\\nmost common type of wood) logs. The “wood” variant collects significantly more oak logs, as is to\\nbe expected of a steerable agent (we speculate that the “water” variant collects less because there are\\nno trees in water). (d) Collection of dirt. The “dirt” and “dig” variants collect a large amount, and are\\nthe variants that are (indirectly in the case of “dig”) conditioned to collect dirt. It is easy to mistakenly\\naim at the ground rather than at grass or trees when collecting seeds or wood, which likely explains\\nthe slightly higher amount of dirt collected by these variants. In all cases, the error bars are 95%\\nconfidence intervals of the mean, over 1,000 episodes per conditioning variant. Treatments for which\\nthe bars in each bar plot do not overlap are statistically significantly different at a p < 0.05 level.\\nVariant name\\nString\\ndig\\nI’m going to dig as far as possible\\ndirt\\nI’m going to collect dirt\\nexplore\\nI’m going to explore\\nhouse\\nI’m going to make a house\\nseed\\nI’m going to collect seeds\\nwater\\nI’m going to find water\\nwood\\nI’m going to chop wood\\nTable 8: Strings corresponding to each conditioning variant.\\n33\\n\\n\\nearly game items such as seeds, wood, and dirt by conditioning with text such as “I’m going to collect\\nseeds/chop wood/collect dirt” (Figure 20b,c,d).\\nWhile our results show some level of steerability, more work is required to increase it. For example,\\nwe were not able to successfully steer agents to gather flowers or to hunt, both of which are possible\\nin the early game, but less common (and, in the case of hunting animals, much more difficult) than\\ngathering dirt, wood, or seeds. Likewise, an experiment in which the agent is presented with a\\ncrafting window and various resources, and conditioned to craft a given item (e.g. “I’m going to\\ncraft a wooden axe”) failed to show that the conditioning had a significant effect on which items got\\ncrafted. Instead, it seemed the agent was more influenced by the prior, unconditional probability\\nof what human players would craft next given the resources available, which is not too surprising\\nsince in Minecraft, especially in the early game, there is a relatively consistent path to gathering\\nresources in a specific order go produce more powerful tools (Fig. 6). For example, if the agent had\\nthe resources to make a stone pickaxe and we asked it instead to make a (weaker) wooden pickaxe, it\\noften would make the stone pickaxe anyway. Finally, looking at videos of agent behaviors failed to\\nconvince us that the “house” conditioning causes the agents to take more steps towards building a\\nhouse than other variants.\\nThus, our results show that it is possible to train a somewhat steerable natural-language-conditioned\\nagent. However, its steerability is still too weak to be practically useful, and it is far from what we\\nbelieve could be accomplished with more research, data, and training compute. Another exciting\\nresearch direction is to have the model predict future text as well as just the next action.\\n34\\n\\n\\nVOYAGER: An Open-Ended Embodied Agent\\nwith Large Language Models\\nGuanzhi Wang1 2 #, Yuqi Xie3, Yunfan Jiang4∗, Ajay Mandlekar1∗,\\nChaowei Xiao1 5, Yuke Zhu1 3, Linxi “Jim” Fan1† #, Anima Anandkumar1 2†\\n1NVIDIA, 2Caltech, 3UT Austin, 4Stanford, 5UW Madison\\n∗Equal contribution\\n†Equal advising\\n# Corresponding authors\\nhttps://voyager.minedojo.org\\nAbstract\\nWe introduce VOYAGER, the first LLM-powered embodied lifelong learning agent\\nin Minecraft that continuously explores the world, acquires diverse skills, and\\nmakes novel discoveries without human intervention. VOYAGER consists of three\\nkey components: 1) an automatic curriculum that maximizes exploration, 2) an\\never-growing skill library of executable code for storing and retrieving complex\\nbehaviors, and 3) a new iterative prompting mechanism that incorporates environ-\\nment feedback, execution errors, and self-verification for program improvement.\\nVOYAGER interacts with GPT-4 via blackbox queries, which bypasses the need for\\nmodel parameter fine-tuning. The skills developed by VOYAGER are temporally\\nextended, interpretable, and compositional, which compounds the agent’s abilities\\nrapidly and alleviates catastrophic forgetting.\\nEmpirically, VOYAGER shows\\nstrong in-context lifelong learning capability and exhibits exceptional proficiency\\nin playing Minecraft. It obtains 3.3× more unique items, travels 2.3× longer\\ndistances, and unlocks key tech tree milestones up to 15.3× faster than prior SOTA.\\nVOYAGER is able to utilize the learned skill library in a new Minecraft world to\\nsolve novel tasks from scratch, while other techniques struggle to generalize.\\nFigure 1: VOYAGER discovers new Minecraft items and skills continually by self-driven exploration,\\nsignificantly outperforming the baselines. X-axis denotes the number of prompting iterations.\\n1\\narXiv:2305.16291v2  [cs.AI]  19 Oct 2023\\n\\n\\nMine Wood  Log\\nMake Crafting Table\\nCraft Stone Sword\\nCraft Shield\\nMake Furnace\\nCook Steak\\nCombat Zombie\\n     Mine Wood Log\\nMake Crafting Table\\nCombat \\nZombie\\nMine Diamond\\nNew \\nTask\\nCode as \\nActions\\nRefine Program\\nEnv Feedback\\nExecution Errors\\nUpdate \\nExploration \\nProgress\\nSkill \\nRetrieval\\nAdd New Skill\\nAutomatic Curriculum\\nIterative Prompting Mechanism\\nSkill Library\\nEnvironment\\nSelf-Verification\\nFigure 2: VOYAGER consists of three key components: an automatic curriculum for open-ended\\nexploration, a skill library for increasingly complex behaviors, and an iterative prompting mechanism\\nthat uses code as action space.\\n1\\nIntroduction\\nBuilding generally capable embodied agents that continuously explore, plan, and develop new skills\\nin open-ended worlds is a grand challenge for the AI community [1–5]. Classical approaches\\nemploy reinforcement learning (RL) [6, 7] and imitation learning [8–10] that operate on primitive\\nactions, which could be challenging for systematic exploration [11–15], interpretability [16–18], and\\ngeneralization [19–21]. Recent advances in large language model (LLM) based agents harness the\\nworld knowledge encapsulated in pre-trained LLMs to generate consistent action plans or executable\\npolicies [16, 22, 19]. They are applied to embodied tasks like games and robotics [23–27], as well as\\nNLP tasks without embodiment [28–30]. However, these agents are not lifelong learners that can\\nprogressively acquire, update, accumulate, and transfer knowledge over extended time spans [31, 32].\\nLet us consider Minecraft as an example. Unlike most other games studied in AI [33, 34, 10],\\nMinecraft does not impose a predefined end goal or a fixed storyline but rather provides a unique\\nplayground with endless possibilities [23]. Minecraft requires players to explore vast, procedurally\\ngenerated 3D terrains and unlock a tech tree using gathered resources. Human players typically start\\nby learning the basics, such as mining wood and cooking food, before advancing to more complex\\ntasks like combating monsters and crafting diamond tools. We argue that an effective lifelong learning\\nagent should have similar capabilities as human players: (1) propose suitable tasks based on its\\ncurrent skill level and world state, e.g., learn to harvest sand and cactus before iron if it finds itself in\\na desert rather than a forest; (2) refine skills based on environmental feedback and commit mastered\\nskills to memory for future reuse in similar situations (e.g. fighting zombies is similar to fighting\\nspiders); (3) continually explore the world and seek out new tasks in a self-driven manner.\\nTowards these goals, we introduce VOYAGER, the first LLM-powered embodied lifelong learning\\nagent to drive exploration, master a wide range of skills, and make new discoveries continually\\nwithout human intervention in Minecraft. VOYAGER is made possible through three key modules\\n(Fig. 2): 1) an automatic curriculum that maximizes exploration; 2) a skill library for storing\\nand retrieving complex behaviors; and 3) a new iterative prompting mechanism that generates\\nexecutable code for embodied control. We opt to use code as the action space instead of low-level\\nmotor commands because programs can naturally represent temporally extended and compositional\\nactions [16, 22], which are essential for many long-horizon tasks in Minecraft. VOYAGER interacts\\nwith a blackbox LLM (GPT-4 [35]) through prompting and in-context learning [36–38]. Our approach\\nbypasses the need for model parameter access and explicit gradient-based training or finetuning.\\nMore specifically, VOYAGER attempts to solve progressively harder tasks proposed by the automatic\\ncurriculum, which takes into account the exploration progress and the agent’s state. The curriculum\\nis generated by GPT-4 based on the overarching goal of “discovering as many diverse things as\\npossible”. This approach can be perceived as an in-context form of novelty search [39, 40]. VOYAGER\\nincrementally builds a skill library by storing the action programs that help solve a task successfully.\\n2\\n\\n\\nInventory (5/36): {'oak_planks': 3, 'stick': \\n4, 'crafting_table': 1, 'stone': 3, \\n'wooden_pickaxe': 1}\\nBiome: river\\nInventory (4/36): {'sugar_cane': 3, 'paper': \\n3, 'fishing_rod': 1, 'wooden_pickaxe': 1}\\nNearby entities: pig, cat, villager\\nHealth: 12/20\\nHunger: 0/20\\nInventory (6/36): {'furnace': 1, \\n'stone_pickaxe': 1, 'oak_planks': 7, \\n'crafting_table': 1, 'raw_iron': 4, 'coal': 1} \\nTime: night\\nNearby entities: zombie\\nEquipment: 'stone_sword', 'shield'\\nReasoning: Since you have a wooden pickaxe and some stones, it would be \\nbeneficial to upgrade your pickaxe to a stone pickaxe for better efficiency.\\nTask: Craft 1 stone pickaxe.\\nReasoning: Since you have a fishing rod in your inventory and you are near a river \\nbiome, it would be a good opportunity to catch some fish for food and experience. \\nTask: Catch 1 fish.\\nReasoning: Your hunger is at 0, which means you need to find food to replenish \\nyour hunger. Since there are pigs nearby, you can kill one to obtain raw porkchops.\\nTask: Kill 1 pig.\\nReasoning: You have raw iron and coal, and you have a furnace. It's time to smelt \\nthe iron to make iron ingots, which can be used to craft better tools and armor.\\nTask: Smelt 4 raw iron.\\nReasoning: Since it's night and there's a zombie nearby, it's a good opportunity to \\ntry killing the zombie now that you have a stone sword and a shield equipped.\\nTask: Kill 1 zombie.\\nGPT-4\\nGPT-4\\nGPT-4\\nGPT-4\\nGPT-4\\nFigure 3: Tasks proposed by the automatic curriculum. We only display the partial prompt for brevity.\\nSee Appendix, Sec. A.3 for the full prompt structure.\\nEach program is indexed by the embedding of its description, which can be retrieved in similar\\nsituations in the future. Complex skills can be synthesized by composing simpler programs, which\\ncompounds VOYAGER’s capabilities rapidly over time and alleviates catastrophic forgetting in other\\ncontinual learning methods [31, 32].\\nHowever, LLMs struggle to produce the correct action code consistently in one shot [41]. To address\\nthis challenge, we propose an iterative prompting mechanism that: (1) executes the generated\\nprogram to obtain observations from the Minecraft simulation (such as inventory listing and nearby\\ncreatures) and error trace from the code interpreter (if any); (2) incorporates the feedback into GPT-4’s\\nprompt for another round of code refinement; and (3) repeats the process until a self-verification\\nmodule confirms the task completion, at which point we commit the program to the skill library (e.g.,\\ncraftStoneShovel() and combatZombieWithSword()) and query the automatic curriculum for\\nthe next milestone (Fig. 2).\\nEmpirically, VOYAGER demonstrates strong in-context lifelong learning capabilities. It can construct\\nan ever-growing skill library of action programs that are reusable, interpretable, and generalizable\\nto novel tasks. We evaluate VOYAGER systematically against other LLM-based agent techniques\\n(e.g., ReAct [29], Reflexion [30], AutoGPT [28]) in MineDojo [23], an open-source Minecraft AI\\nframework. VOYAGER outperforms prior SOTA by obtaining 3.3× more unique items, unlocking key\\ntech tree milestones up to 15.3× faster, and traversing 2.3× longer distances. We further demonstrate\\nthat VOYAGER is able to utilize the learned skill library in a new Minecraft world to solve novel tasks\\nfrom scratch, while other methods struggle to generalize.\\n2\\nMethod\\nVOYAGER consists of three novel components: (1) an automatic curriculum (Sec. 2.1) that suggests\\nobjectives for open-ended exploration, (2) a skill library (Sec. 2.2) for developing increasingly\\ncomplex behaviors, and (3) an iterative prompting mechanism (Sec. 2.3) that generates executable\\ncode for embodied control. Full prompts are presented in Appendix, Sec. A.\\n2.1\\nAutomatic Curriculum\\nEmbodied agents encounter a variety of objectives with different complexity levels in open-ended\\nenvironments. An automatic curriculum offers numerous benefits for open-ended exploration, ensur-\\ning a challenging but manageable learning process, fostering curiosity-driven intrinsic motivation\\nfor agents to learn and explore, and encouraging the development of general and flexible problem-\\nsolving strategies [42–44]. Our automatic curriculum capitalizes on the internet-scale knowledge\\ncontained within GPT-4 by prompting it to provide a steady stream of new tasks or challenges. The\\ncurriculum unfolds in a bottom-up fashion, allowing for considerable adaptability and responsiveness\\nto the exploration progress and the agent’s current state (Fig. 3). As VOYAGER progresses to harder\\nself-driven goals, it naturally learns a variety of skills, such as “mining a diamond”.\\n3\\n\\n\\nProgram Description\\nSkill Library\\nTop-5 Relevant Skills\\nProgram Generated by GPT-4\\nTask: Craft Iron Pickaxe\\nKey\\nAdd\\nRetrieve\\nValue\\nSkill Library\\nQuery\\nHow to craft an iron pickaxe in \\nMinecraft?\\nTo craft an iron pickaxe, you \\nneed to 3 iron ingots and 2 \\nsticks. Once you have gathered \\nthe materials, ....\\n----------------------------------\\n         Environment Feedback\\nMine Wood  Log\\nMake Crafting Table\\nCraft Wooden Pickaxe\\nCraft Stone Sword\\nMake Furnace\\n...\\nCombat Cow\\nCook Steak\\nCraft Iron Axe\\nCombat Zombie\\nSmelt Iron Ingot\\nCraft Stick\\nMake Crafting Table\\nMake Furnace\\nCraft Wooden Pickaxe\\nGPT-3.5\\nEmbedding\\nEmbedding\\nGPT-3.5\\nFigure 4: Skill library. Top: Adding a new skill. Each time GPT-4 generates and verifies a new\\nskill, we add it to the skill library, represented by a vector database. The key is the embedding vector\\nof the program description (generated by GPT-3.5), while the value is the program itself. Bottom:\\nSkill retrieval. When faced with a new task proposed by the automatic curriculum, we first leverage\\nGPT-3.5 to generate a general suggestion for solving the task, which is combined with environment\\nfeedback as the query context. Subsequently, we perform querying to identify the top-5 relevant skills.\\nThe input prompt to GPT-4 consists of several components:\\n(1) Directives encouraging diverse behaviors and imposing constraints,\\nsuch as\\n“My ultimate goal is to discover as many diverse things as possible\\n...\\nThe next task should not be too hard since I may not have the\\nnecessary resources or have learned enough skills to complete it\\nyet.”;\\n(2) The agent’s current state, including inventory, equipment, nearby blocks and entities,\\nbiome, time, health and hunger bars, and position;\\n(3) Previously completed and failed tasks, reflecting the agent’s current exploration progress\\nand capabilities frontier;\\n(4) Additional context: We also leverage GPT-3.5 to self-ask questions based on the agent’s\\ncurrent state and exploration progress and self-answer questions. We opt to use GPT-3.5\\ninstead of GPT-4 for standard NLP tasks due to budgetary considerations.\\n2.2\\nSkill Library\\nWith the automatic curriculum consistently proposing increasingly complex tasks, it is essential to\\nhave a skill library that serves as a basis for learning and evolution. Inspired by the generality, inter-\\npretability, and universality of programs [45], we represent each skill with executable code that scaf-\\nfolds temporally extended actions for completing a specific task proposed by the automatic curriculum.\\nThe input prompt to GPT-4 consists of the following components:\\n(1) Guidelines\\nfor\\ncode\\ngeneration,\\nsuch\\nas\\n“Your function will be reused\\nfor building more complex functions.\\nTherefore, you should make\\nit generic and reusable.”;\\n(2) Control primitive APIs, and relevant skills retrieved from the skill library, which are\\ncrucial for in-context learning [36–38] to work well;\\n(3) The generated code from the last round, environment feedback, execution errors, and\\ncritique, based on which GPT-4 can self-improve (Sec. 2.3);\\n(4) The agent’s current state, including inventory, equipment, nearby blocks and entities,\\nbiome, time, health and hunger bars, and position;\\n4\\n\\n\\nI cannot make stick because I need:  2 more planks\\nI cannot make stone_shovel because I need:  2 more stick\\nthrow new Error(`No item named ${name}`);\\nNo item named acacia_axe\\nat line 18:await craftItem(bot, \\\"acacia_axe\\\", 1);\\nEnvironment Feedback\\nExecution Error\\nGPT-4\\nGPT-4\\nFigure 5: Left: Environment feedback. GPT-4 realizes it needs 2 more planks before crafting sticks.\\nRight: Execution error. GPT-4 realizes it should craft a wooden axe instead of an acacia axe since\\nthere is no acacia axe in Minecraft. We only display the partial prompt for brevity. The full prompt\\nstructure for code generation is in Appendix, Sec. A.4.\\n(5) Chain-of-thought prompting [46] to do reasoning before code generation.\\nWe iteratively refine the program through a novel iterative prompting mechanism (Sec. 2.3), in-\\ncorporate it into the skill library as a new skill, and index it by the embedding of its description\\n(Fig. 4, top). For skill retrieval, we query the skill library with the embedding of self-generated task\\nplans and environment feedback (Fig. 4, bottom). By continuously expanding and refining the skill\\nlibrary, VOYAGER can learn, adapt, and excel in a wide spectrum of tasks, consistently pushing the\\nboundaries of its capabilities in the open world.\\n2.3\\nIterative Prompting Mechanism\\nWe introduce an iterative prompting mechanism for self-improvement through three types of feedback:\\n(1) Environment feedback, which illustrates the intermediate progress of program execution\\n(Fig. 5, left). For example, “I cannot make an iron chestplate because I need:\\n7 more iron ingots” highlights the cause of failure in crafting an iron chestplate. We use\\nbot.chat() inside control primitive APIs to generate environment feedback and prompt\\nGPT-4 to use this function as well during code generation;\\n(2) Execution errors from the program interpreter that reveal any invalid operations or syntax\\nerrors in programs, which are valuable for bug fixing (Fig. 5, right);\\n(3) Self-verification for checking task success. Instead of manually coding success checkers\\nfor each new task proposed by the automatic curriculum, we instantiate another GPT-4\\nagent for self-verification. By providing VOYAGER’s current state and the task to GPT-4,\\nwe ask it to act as a critic [47–49] and inform us whether the program achieves the task.\\nIn addition, if the task fails, it provides a critique by suggesting how to complete the task\\n(Fig. 6). Hence, our self-verification is more comprehensive than self-reflection [30] by both\\nchecking success and reflecting on mistakes.\\nDuring each round of code generation, we execute the generated program to obtain environment\\nfeedback and execution errors from the code interpreter, which are incorporated into GPT-4’s prompt\\nfor the next round of code refinement. This iterative process repeats until self-verification validates\\n5\\n\\n\\nInventory (8/36): {'oak_planks': 5, 'cobblestone': 2, \\n'porkchop': 2, 'wooden_sword': 1, 'coal': 5, 'wooden_pickaxe': \\n1, 'oak_log': 3, 'dirt': 9}\\nTask: Mine 5 coal ores\\nInventory (10/36): {'raw_copper': 9, 'copper_ingot': 3, \\n'acacia_planks': 1, 'raw_iron': 1, 'stick': 1, 'iron_sword': 1, \\n'iron_pickaxe': 1, 'iron_ingot': 3, 'crafting_table': 1, 'furnace': 1}\\nTask: Craft a spyglass\\nInventory (7/36): {'oak_log': 5, 'oak_planks': 1, \\n'wooden_pickaxe': 1, 'wooden_sword': 1, 'porkchop': 2, \\n'white_wool': 2, 'mutton': 6}\\nTask: Kill 3 sheep\\nInventory (9/36): {'string': 4, 'coal': 1, 'rotten_flesh': 1, \\n'iron_sword': 1, 'furnace': 1, 'dirt': 6, 'stone_shovel': 1, \\n'wooden_pickaxe': 1, 'granite': 5}\\nTask: Kill 1 zombie\\nReasoning: Mining coal_ore in Minecraft will get coal. You have 5 coal in your \\ninventory.\\nSuccess: True\\nReasoning: To craft a spyglass, you need 2 copper ingots and 1 amethyst shard. \\nYou have 3 copper ingots, but you don't have any amethyst shards.\\nSuccess: False\\nCritique: Find and mine an amethyst shard underground.\\nReasoning: You have 2 white_wool and 6 mutton in your inventory, which indicates \\nthat you killed 2 sheep. You needed to kill 3 sheep.\\nSuccess: False\\nCritique: Find and kill one more sheep to complete the task.\\nReasoning: You have 1 rotten_flesh in your inventory, which means you have killed \\nat least 1 zombie.\\nSuccess: True\\nGPT-4\\nGPT-4\\nGPT-4\\nGPT-4\\nFigure 6: Self-verification examples. We only display the partial prompt for brevity. See Appendix,\\nSec. A.5 for the full prompt structure.\\nthe task’s completion, at which point we add this new skill to the skill library and ask the automatic\\ncurriculum for a new objective (Fig. 2). If the agent gets stuck after 4 rounds of code generation, then\\nwe query the curriculum for another task. This iterative prompting approach significantly improves\\nprogram synthesis for embodied control, enabling VOYAGER to continuously acquire diverse skills\\nwithout human intervention.\\n3\\nExperiments\\n3.1\\nExperimental Setup\\nWe leverage OpenAI’s gpt-4-0314 [35] and gpt-3.5-turbo-0301 [50] APIs for text completion,\\nalong with text-embedding-ada-002 [51] API for text embedding. We set all temperatures to\\n0 except for the automatic curriculum, which uses temperature = 0.1 to encourage task diversity. Our\\nsimulation environment is built on top of MineDojo [23] and leverages Mineflayer [52] JavaScript\\nAPIs for motor controls. See Appendix, Sec. B.1 for more details.\\n3.2\\nBaselines\\nBecause there is no LLM-based agents that work out of the box for Minecraft, we make our best\\neffort to select a number of representative algorithms as baselines. These methods are originally\\ndesigned only for NLP tasks without embodiment, therefore we have to re-interpret them to be\\nexecutable in MineDojo and compatible with our experimental setting:\\nReAct [29] uses chain-of-thought prompting [46] by generating both reasoning traces and action\\nplans with LLMs. We provide it with our environment feedback and the agent states as observations.\\nReflexion [30] is built on top of ReAct [29] with self-reflection to infer more intuitive future actions.\\nWe provide it with execution errors and our self-verification module.\\nAutoGPT [28] is a popular software tool that automates NLP tasks by decomposing a high-level\\ngoal into multiple subgoals and executing them in a ReAct-style loop. We re-implement AutoGPT\\nby using GPT-4 to do task decomposition and provide it with the agent states, environment feedback,\\nand execution errors as observations for subgoal execution. Compared with VOYAGER, AutoGPT\\nlacks the skill library for accumulating knowledge, self-verification for assessing task success, and\\nautomatic curriculum for open-ended exploration.\\nNote that we do not directly compare with prior methods that take Minecraft screen pixels as input\\nand output low-level controls [53–55]. It would not be an apple-to-apple comparison, because we rely\\non the high-level Mineflayer [52] API to control the agent. Our work’s focus is on pushing the limits\\nof GPT-4 for lifelong embodied agent learning, rather than solving the 3D perception or sensorimotor\\ncontrol problems. VOYAGER is orthogonal and can be combined with gradient-based approaches like\\n6\\n\\n\\nTable 1: Tech tree mastery. Fractions indicate the number of successful trials out of three total runs.\\n0/3 means the method fails to unlock a level of the tech tree within the maximal prompting iterations\\n(160). Numbers are prompting iterations averaged over three trials. The fewer the iterations, the\\nmore efficient the method.\\nMethod\\nWooden Tool\\nStone Tool\\nIron Tool\\nDiamond Tool\\nReAct [29]\\nN/A (0/\\n3)\\nN/A (0/\\n3)\\nN/A (0/\\n3)\\nN/A (0/\\n3)\\nReflexion [30]\\nN/A (0/\\n3)\\nN/A (0/\\n3)\\nN/A (0/\\n3)\\nN/A (0/\\n3)\\nAutoGPT [28]\\n92 ± 72 (3/\\n3)\\n94 ± 72 (3/\\n3)\\n135 ± 103 (3/\\n3)\\nN/A (0/\\n3)\\nVOYAGER w/o Skill Library\\n7 ± 2 (3/\\n3)\\n9 ± 4 (3/\\n3)\\n29 ± 11 (3/\\n3)\\nN/A (0/\\n3)\\nVOYAGER (Ours)\\n6 ± 2 (3/\\n3)\\n11 ± 2 (3/\\n3)\\n21 ± 7 (3/\\n3)\\n102 (1/\\n3)\\nFigure 7: Map coverage: bird’s eye views of Minecraft maps. VOYAGER is able to traverse 2.3×\\nlonger distances compared to baselines while crossing diverse terrains.\\nVPT [8] as long as the controller provides a code API. We make a system-level comparison between\\nVOYAGER and prior Minecraft agents in Table. A.2.\\n3.3\\nEvaluation Results\\nWe systematically evaluate VOYAGER and baselines on their exploration performance, tech tree\\nmastery, map coverage, and zero-shot generalization capability to novel tasks in a new world.\\nSignificantly better exploration.\\nResults of exploration performance are shown in Fig. 1.\\nVOYAGER’s superiority is evident in its ability to consistently make new strides, discovering 63\\nunique items within 160 prompting iterations, 3.3× many novel items compared to its counterparts.\\nOn the other hand, AutoGPT lags considerably in discovering new items, while ReAct and Reflexion\\nstruggle to make significant progress, given the abstract nature of the open-ended exploration goal\\nthat is challenging to execute without an appropriate curriculum.\\nConsistent tech tree mastery. The Minecraft tech tree tests the agent’s ability to craft and use a\\nhierarchy of tools. Progressing through this tree (wooden tool →stone tool →iron tool →diamond\\ntool) requires the agent to master systematic and compositional skills. Compared with baselines,\\nVOYAGER unlocks the wooden level 15.3× faster (in terms of the prompting iterations), the stone\\nlevel 8.5× faster, the iron level 6.4× faster, and VOYAGER is the only one to unlock the diamond level\\nof the tech tree (Fig. 2 and Table. 1). This underscores the effectiveness of the automatic curriculum,\\nwhich consistently presents challenges of suitable complexity to facilitate the agent’s progress.\\nExtensive map traversal. VOYAGER is able to navigate distances 2.3× longer compared to baselines\\nby traversing a variety of terrains, while the baseline agents often find themselves confined to local\\nareas, which significantly hampers their capacity to discover new knowledge (Fig. 7).\\n7\\n\\n\\nTable 2: Zero-shot generalization to unseen tasks. Fractions indicate the number of successful\\ntrials out of three total attempts. 0/3 means the method fails to solve the task within the maximal\\nprompting iterations (50). Numbers are prompting iterations averaged over three trials. The fewer\\nthe iterations, the more efficient the method.\\nMethod\\nDiamond Pickaxe\\nGolden Sword\\nLava Bucket\\nCompass\\nReAct [29]\\nN/A (0/\\n3)\\nN/A (0/\\n3)\\nN/A (0/\\n3)\\nN/A (0/\\n3)\\nReflexion [30]\\nN/A (0/\\n3)\\nN/A (0/\\n3)\\nN/A (0/\\n3)\\nN/A (0/\\n3)\\nAutoGPT [28]\\nN/A (0/\\n3)\\nN/A (0/\\n3)\\nN/A (0/\\n3)\\nN/A (0/\\n3)\\nAutoGPT [28] w/ Our Skill Library\\n39 (1/\\n3)\\n30 (1/\\n3)\\nN/A (0/\\n3)\\n30 (2/\\n3)\\nVOYAGER w/o Skill Library\\n36 (2/\\n3)\\n30 ± 9 (3/\\n3)\\n27 ± 9 (3/\\n3)\\n26 ± 3 (3/\\n3)\\nVOYAGER (Ours)\\n19 ± 3 (3/\\n3)\\n18 ± 7 (3/\\n3)\\n21 ± 5 (3/\\n3)\\n18 ± 2 (3/\\n3)\\nFigure 8: Zero-shot generalization to unseen tasks. We visualize the intermediate progress of each\\nmethod on two tasks. See Appendix, Sec. B.4.3 for the other two tasks. We do not plot ReAct and\\nReflexion since they do not make any meaningful progress.\\nEfficient zero-shot generalization to unseen tasks. To evaluate zero-shot generalization, we clear\\nthe agent’s inventory, reset it to a newly instantiated world, and test it with unseen tasks. For both\\nVOYAGER and AutoGPT, we utilize GPT-4 to break down the task into a series of subgoals. Table. 2\\nand Fig. 8 show VOYAGER can consistently solve all the tasks, while baselines cannot solve any task\\nwithin 50 prompting iterations. What’s interesting to note is that our skill library constructed from\\nlifelong learning not only enhances VOYAGER’s performance but also gives a boost to AutoGPT.\\nThis demonstrates that the skill library serves as a versatile tool that can be readily employed by other\\nmethods, effectively acting as a plug-and-play asset to enhance performance.\\n3.4\\nAblation Studies\\nWe ablate 6 design choices (automatic curriculum, skill library, environment feedback, execution\\nerrors, self-verification, and GPT-4 for code generation) in VOYAGER and study their impact on\\nexploration performance (see Appendix, Sec. B.3 for details of each ablated variant). Results are\\nshown in Fig. 9. We highlight the key findings below:\\n• Automatic curriculum is crucial for the agent’s consistent progress. The discovered item\\ncount drops by 93% if the curriculum is replaced with a random one, because certain tasks\\nmay be too challenging if attempted out of order. On the other hand, a manually designed\\ncurriculum requires significant Minecraft-specific expertise, and does not take into account\\nthe agent’s live situation. It falls short in the experimental results compared to our automatic\\ncurriculum.\\n• VOYAGER w/o skill library exhibits a tendency to plateau in the later stages. This\\nunderscores the pivotal role that the skill library plays in VOYAGER. It helps create more\\ncomplex actions and steadily pushes the agent’s boundaries by encouraging new skills to be\\nbuilt upon older ones.\\n8\\n\\n\\nFigure 9: Left: Ablation studies for the automatic curriculum, skill library, and GPT-4. GPT-3.5\\nmeans replacing GPT-4 with GPT-3.5 for code generation. VOYAGER outperforms all the alternatives,\\ndemonstrating the critical role of each component. Right: Ablation studies for the iterative\\nprompting mechanism. VOYAGER surpasses all the other options, thereby highlighting the essential\\nsignificance of each type of feedback in the iterative prompting mechanism.\\nFigure 10: VOYAGER builds 3D structures with human feedback. The progress of building designs\\nthat integrate human input is demonstrated from left to right.\\n• Self-verification is the most important among all the feedback types. Removing the\\nmodule leads to a significant drop (−73%) in the discovered item count. Self-verification\\nserves as a critical mechanism to decide when to move on to a new task or reattempt a\\npreviously unsuccessful task.\\n• GPT-4 significantly outperforms GPT-3.5 in code generation and obtains 5.7× more\\nunique items, as GPT-4 exhibits a quantum leap in coding abilities. This finding corroborates\\nrecent studies in the literature [56, 57].\\n3.5\\nMultimodal Feedback from Humans\\nVOYAGER does not currently support visual perception, because the available version of GPT-4 API\\nis text-only at the time of this writing. However, VOYAGER has the potential to be augmented by\\nmultimodal perception models [58, 59] to achieve more impressive tasks. We demonstrate that given\\nhuman feedback, VOYAGER is able to construct complex 3D structures in Minecraft, such as a Nether\\nPortal and a house (Fig. 10). There are two ways to integrate human feedback:\\n(1) Human as a critic (equivalent to VOYAGER’s self-verification module): humans provide\\nvisual critique to VOYAGER, allowing it to modify the code from the previous round. This\\nfeedback is essential for correcting certain errors in the spatial details of a 3D structure that\\nVOYAGER cannot perceive directly.\\n(2) Human as a curriculum (equivalent to VOYAGER’s automatic curriculum module): humans\\nbreak down a complex building task into smaller steps, guiding VOYAGER to complete them\\nincrementally. This approach improves VOYAGER’s ability to handle more sophisticated 3D\\nconstruction tasks.\\n9\\n\\n\\n4\\nLimitations and Future Work\\nCost. The GPT-4 API incurs significant costs. It is 15× more expensive than GPT-3.5. Nevertheless,\\nVOYAGER requires the quantum leap in code generation quality from GPT-4 (Fig. 9), which GPT-3.5\\nand open-source LLMs cannot provide [60].\\nInaccuracies. Despite the iterative prompting mechanism, there are still cases where the agent gets\\nstuck and fails to generate the correct skill. The automatic curriculum has the flexibility to reattempt\\nthis task at a later time. Occasionally, self-verification module may also fail, such as not recognizing\\nspider string as a success signal of beating a spider.\\nHallucinations. The automatic curriculum occasionally proposes unachievable tasks. For example, it\\nmay ask the agent to craft a “copper sword\\\" or “copper chestplate\\\", which are items that do not exist\\nwithin the game. Hallucinations also occur during the code generation process. For instance, GPT-4\\ntends to use cobblestone as a fuel input, despite being an invalid fuel source in the game. Additionally,\\nit may call functions absent in the provided control primitive APIs, leading to code execution errors.\\nWe are confident that improvements in the GPT API models as well as novel techniques for finetuning\\nopen-source LLMs will overcome these limitations in the future.\\n5\\nRelated work\\nDecision-making Agents in Minecraft.\\nMinecraft is an open-ended 3D world with incredibly\\nflexible game mechanics supporting a broad spectrum of activities. Built upon notable Minecraft\\nbenchmarks [23, 61–65], Minecraft learning algorithms can be divided into two categories: 1)\\nLow-level controller: Many prior efforts leverage hierarchical reinforcement learning to learn from\\nhuman demonstrations [66–68]. Kanitscheider et al. [14] design a curriculum based on success rates,\\nbut its objectives are limited to curated items. MineDojo [23] and VPT [8] utilize YouTube videos\\nfor large-scale pre-training. DreamerV3 [69], on the other hand, learns a world model to explore\\nthe environment and collect diamonds. 2) High-level planner: Volum et al. [70] leverage few-shot\\nprompting with Codex [41] to generate executable policies, but they require additional human\\ninteraction. Recent works leverage LLMs as a high-level planner in Minecraft by decomposing\\na high-level task into several subgoals following Minecraft recipes [55, 53, 71], thus lacking full\\nexploration flexibility. Like these latter works, VOYAGER also uses LLMs as a high-level planner by\\nprompting GPT-4 and utilizes Mineflayer [52] as a low-level controller following Volum et al. [70].\\nUnlike prior works, VOYAGER employs an automatic curriculum that unfolds in a bottom-up manner,\\ndriven by curiosity, and therefore enables open-ended exploration.\\nLarge Language Models for Agent Planning.\\nInspired by the strong emergent capabilities of\\nLLMs, such as zero-shot prompting and complex reasoning [72, 37, 38, 36, 73, 74], embodied agent\\nresearch [75–78] has witnessed a significant increase in the utilization of LLMs for planning purposes.\\nRecent efforts can be roughly classified into two groups. 1) Large language models for robot\\nlearning: Many prior works apply LLMs to generate subgoals for robot planning [27, 27, 25, 79, 80].\\nInner Monologue [26] incorporates environment feedback for robot planning with LLMs. Code as\\nPolicies [16] and ProgPrompt [22] directly leverage LLMs to generate executable robot policies.\\nVIMA [19] and PaLM-E [59] fine-tune pre-trained LLMs to support multimodal prompts. 2)\\nLarge language models for text agents: ReAct [29] leverages chain-of-thought prompting [46] and\\ngenerates both reasoning traces and task-specific actions with LLMs. Reflexion [30] is built upon\\nReAct [29] with self-reflection to enhance reasoning. AutoGPT [28] is a popular tool that automates\\nNLP tasks by crafting a curriculum of multiple subgoals for completing a high-level goal while\\nincorporating ReAct [29]’s reasoning and acting loops. DERA [81] frames a task as a dialogue\\nbetween two GPT-4 [35] agents. Generative Agents [82] leverages ChatGPT [50] to simulate human\\nbehaviors by storing agents’ experiences as memories and retrieving those for planning, but its agent\\nactions are not executable. SPRING [83] is a concurrent work that uses GPT-4 to extract game\\nmechanics from game manuals, based on which it answers questions arranged in a directed acyclic\\ngraph and predicts the next action. All these works lack a skill library for developing more complex\\nbehaviors, which are crucial components for the success of VOYAGER in lifelong learning.\\nCode Generation with Execution.\\nCode generation has been a longstanding challenge in\\nNLP [41, 84, 85, 73, 37], with various works leveraging execution results to improve program\\n10\\n\\n\\nsynthesis. Execution-guided approaches leverage intermediate execution outcomes to guide program\\nsearch [86–88]. Another line of research utilizes majority voting to choose candidates based on their\\nexecution performance [89, 90]. Additionally, LEVER [91] trains a verifier to distinguish and reject\\nincorrect programs based on execution results. CLAIRIFY [92], on the other hand, generates code\\nfor planning chemistry experiments and makes use of a rule-based verifier to iteratively provide\\nerror feedback to LLMs. VOYAGER distinguishes itself from these works by integrating environment\\nfeedback, execution errors, and self-verification (to assess task success) into an iterative prompting\\nmechanism for embodied control.\\n6\\nConclusion\\nIn this work, we introduce VOYAGER, the first LLM-powered embodied lifelong learning agent,\\nwhich leverages GPT-4 to explore the world continuously, develop increasingly sophisticated skills,\\nand make new discoveries consistently without human intervention. VOYAGER exhibits superior\\nperformance in discovering novel items, unlocking the Minecraft tech tree, traversing diverse terrains,\\nand applying its learned skill library to unseen tasks in a newly instantiated world. VOYAGER serves\\nas a starting point to develop powerful generalist agents without tuning the model parameters.\\n7\\nBroader Impacts\\nOur research is conducted within Minecraft, a safe and harmless 3D video game environment. While\\nVOYAGER is designed to be generally applicable to other domains, such as robotics, its application to\\nphysical robots would require additional attention and the implementation of safety constraints by\\nhumans to ensure responsible and secure deployment.\\n8\\nAcknowledgements\\nWe are extremely grateful to Ziming Zhu, Kaiyu Yang, Rafał Kocielnik, Colin White, Or Sharir, Sahin\\nLale, De-An Huang, Jean Kossaifi, Yuncong Yang, Charles Zhang, Minchao Huang, and many other\\ncolleagues and friends for their helpful feedback and insightful discussions. This work is done during\\nGuanzhi Wang’s internship at NVIDIA. Guanzhi Wang is supported by the Kortschak fellowship in\\nComputing and Mathematical Sciences at Caltech.\\nReferences\\n[1] Eric Kolve, Roozbeh Mottaghi, Winson Han, Eli VanderBilt, Luca Weihs, Alvaro Herrasti,\\nDaniel Gordon, Yuke Zhu, Abhinav Gupta, and Ali Farhadi. Ai2-thor: An interactive 3d\\nenvironment for visual ai. arXiv preprint arXiv: Arxiv-1712.05474, 2017.\\n[2] Manolis Savva, Jitendra Malik, Devi Parikh, Dhruv Batra, Abhishek Kadian, Oleksandr\\nMaksymets, Yili Zhao, Erik Wijmans, Bhavana Jain, Julian Straub, Jia Liu, and Vladlen\\nKoltun. Habitat: A platform for embodied AI research. In 2019 IEEE/CVF International\\nConference on Computer Vision, ICCV 2019, Seoul, Korea (South), October 27 - November 2,\\n2019, pages 9338–9346. IEEE, 2019.\\n[3] Yuke Zhu, Josiah Wong, Ajay Mandlekar, and Roberto Martín-Martín. robosuite: A mod-\\nular simulation framework and benchmark for robot learning. arXiv preprint arXiv: Arxiv-\\n2009.12293, 2020.\\n[4] Fei Xia, William B. Shen, Chengshu Li, Priya Kasimbeg, Micael Tchapmi, Alexander Toshev,\\nLi Fei-Fei, Roberto Martín-Martín, and Silvio Savarese. Interactive gibson benchmark (igibson\\n0.5): A benchmark for interactive navigation in cluttered environments. arXiv preprint arXiv:\\nArxiv-1910.14442, 2019.\\n[5] Bokui Shen, Fei Xia, Chengshu Li, Roberto Martín-Martín, Linxi Fan, Guanzhi Wang, Claudia\\nPérez-D’Arpino, Shyamal Buch, Sanjana Srivastava, Lyne P. Tchapmi, Micael E. Tchapmi, Kent\\nVainio, Josiah Wong, Li Fei-Fei, and Silvio Savarese. igibson 1.0: a simulation environment for\\ninteractive tasks in large realistic scenes. arXiv preprint arXiv: Arxiv-2012.02924, 2020.\\n11\\n\\n\\n[6] Jens Kober, J Andrew Bagnell, and Jan Peters. Reinforcement learning in robotics: A survey.\\nThe International Journal of Robotics Research, 32(11):1238–1274, 2013.\\n[7] Kai Arulkumaran, Marc Peter Deisenroth, Miles Brundage, and Anil Anthony Bharath. Deep\\nreinforcement learning: A brief survey. IEEE Signal Processing Magazine, 34(6):26–38, 2017.\\n[8] Bowen Baker, Ilge Akkaya, Peter Zhokhov, Joost Huizinga, Jie Tang, Adrien Ecoffet, Brandon\\nHoughton, Raul Sampedro, and Jeff Clune. Video pretraining (vpt): Learning to act by watching\\nunlabeled online videos. arXiv preprint arXiv: Arxiv-2206.11795, 2022.\\n[9] DeepMind Interactive Agents Team, Josh Abramson, Arun Ahuja, Arthur Brussee, Federico\\nCarnevale, Mary Cassin, Felix Fischer, Petko Georgiev, Alex Goldin, Mansi Gupta, Tim\\nHarley, Felix Hill, Peter C Humphreys, Alden Hung, Jessica Landon, Timothy Lillicrap, Hamza\\nMerzic, Alistair Muldal, Adam Santoro, Guy Scully, Tamara von Glehn, Greg Wayne, Nathaniel\\nWong, Chen Yan, and Rui Zhu. Creating multimodal interactive agents with imitation and\\nself-supervised learning. arXiv preprint arXiv: Arxiv-2112.03763, 2021.\\n[10] Oriol Vinyals, Igor Babuschkin, Junyoung Chung, Michael Mathieu, Max Jaderberg, Wo-\\njciech M Czarnecki, Andrew Dudzik, Aja Huang, Petko Georgiev, Richard Powell, et al.\\nAlphastar: Mastering the real-time strategy game starcraft ii. DeepMind blog, 2, 2019.\\n[11] Adrien Ecoffet, Joost Huizinga, Joel Lehman, Kenneth O. Stanley, and Jeff Clune. Go-explore:\\na new approach for hard-exploration problems. arXiv preprint arXiv: Arxiv-1901.10995, 2019.\\n[12] Joost Huizinga and Jeff Clune. Evolving multimodal robot behavior via many stepping stones\\nwith the combinatorial multiobjective evolutionary algorithm.\\nEvolutionary computation,\\n30(2):131–164, 2022.\\n[13] Rui Wang, Joel Lehman, Aditya Rawal, Jiale Zhi, Yulun Li, Jeffrey Clune, and Kenneth O.\\nStanley. Enhanced POET: open-ended reinforcement learning through unbounded invention of\\nlearning challenges and their solutions. In Proceedings of the 37th International Conference on\\nMachine Learning, ICML 2020, 13-18 July 2020, Virtual Event, volume 119 of Proceedings of\\nMachine Learning Research, pages 9940–9951. PMLR, 2020.\\n[14] Ingmar Kanitscheider, Joost Huizinga, David Farhi, William Hebgen Guss, Brandon Houghton,\\nRaul Sampedro, Peter Zhokhov, Bowen Baker, Adrien Ecoffet, Jie Tang, Oleg Klimov, and Jeff\\nClune. Multi-task curriculum learning in a complex, visual, hard-exploration domain: Minecraft.\\narXiv preprint arXiv: Arxiv-2106.14876, 2021.\\n[15] Michael Dennis, Natasha Jaques, Eugene Vinitsky, Alexandre M. Bayen, Stuart Russell, Andrew\\nCritch, and Sergey Levine. Emergent complexity and zero-shot transfer via unsupervised\\nenvironment design. In Hugo Larochelle, Marc’Aurelio Ranzato, Raia Hadsell, Maria-Florina\\nBalcan, and Hsuan-Tien Lin, editors, Advances in Neural Information Processing Systems 33:\\nAnnual Conference on Neural Information Processing Systems 2020, NeurIPS 2020, December\\n6-12, 2020, virtual, 2020.\\n[16] Jacky Liang, Wenlong Huang, Fei Xia, Peng Xu, Karol Hausman, Brian Ichter, Pete Florence,\\nand Andy Zeng. Code as policies: Language model programs for embodied control. arXiv\\npreprint arXiv: Arxiv-2209.07753, 2022.\\n[17] Shao-Hua Sun, Te-Lin Wu, and Joseph J. Lim. Program guided agent. In 8th International\\nConference on Learning Representations, ICLR 2020, Addis Ababa, Ethiopia, April 26-30, 2020.\\nOpenReview.net, 2020.\\n[18] Zelin Zhao, Karan Samel, Binghong Chen, and Le Song. Proto: Program-guided transformer for\\nprogram-guided tasks. In Marc’Aurelio Ranzato, Alina Beygelzimer, Yann N. Dauphin, Percy\\nLiang, and Jennifer Wortman Vaughan, editors, Advances in Neural Information Processing\\nSystems 34: Annual Conference on Neural Information Processing Systems 2021, NeurIPS\\n2021, December 6-14, 2021, virtual, pages 17021–17036, 2021.\\n[19] Yunfan Jiang, Agrim Gupta, Zichen Zhang, Guanzhi Wang, Yongqiang Dou, Yanjun Chen,\\nLi Fei-Fei, Anima Anandkumar, Yuke Zhu, and Linxi (Jim) Fan. Vima: General robot manipu-\\nlation with multimodal prompts. ARXIV.ORG, 2022.\\n12\\n\\n\\n[20] Mohit Shridhar, Lucas Manuelli, and Dieter Fox. Cliport: What and where pathways for robotic\\nmanipulation. arXiv preprint arXiv: Arxiv-2109.12098, 2021.\\n[21] Linxi Fan, Guanzhi Wang, De-An Huang, Zhiding Yu, Li Fei-Fei, Yuke Zhu, and Animashree\\nAnandkumar. SECANT: self-expert cloning for zero-shot generalization of visual policies. In\\nMarina Meila and Tong Zhang, editors, Proceedings of the 38th International Conference on\\nMachine Learning, ICML 2021, 18-24 July 2021, Virtual Event, volume 139 of Proceedings of\\nMachine Learning Research, pages 3088–3099. PMLR, 2021.\\n[22] Ishika Singh, Valts Blukis, Arsalan Mousavian, Ankit Goyal, Danfei Xu, Jonathan Tremblay,\\nDieter Fox, Jesse Thomason, and Animesh Garg. Progprompt: Generating situated robot task\\nplans using large language models. arXiv preprint arXiv: Arxiv-2209.11302, 2022.\\n[23] Linxi Fan, Guanzhi Wang, Yunfan Jiang, Ajay Mandlekar, Yuncong Yang, Haoyi Zhu, Andrew\\nTang, De-An Huang, Yuke Zhu, and Anima Anandkumar. Minedojo: Building open-ended\\nembodied agents with internet-scale knowledge. arXiv preprint arXiv: Arxiv-2206.08853, 2022.\\n[24] Andy Zeng, Adrian Wong, Stefan Welker, Krzysztof Choromanski, Federico Tombari, Aveek\\nPurohit, Michael Ryoo, Vikas Sindhwani, Johnny Lee, Vincent Vanhoucke, and Pete Florence.\\nSocratic models: Composing zero-shot multimodal reasoning with language. arXiv preprint\\narXiv: Arxiv-2204.00598, 2022.\\n[25] Michael Ahn, Anthony Brohan, Noah Brown, Yevgen Chebotar, Omar Cortes, Byron David,\\nChelsea Finn, Keerthana Gopalakrishnan, Karol Hausman, Alex Herzog, Daniel Ho, Jasmine\\nHsu, Julian Ibarz, Brian Ichter, Alex Irpan, Eric Jang, Rosario Jauregui Ruano, Kyle Jeffrey,\\nSally Jesmonth, Nikhil J Joshi, Ryan Julian, Dmitry Kalashnikov, Yuheng Kuang, Kuang-Huei\\nLee, Sergey Levine, Yao Lu, Linda Luu, Carolina Parada, Peter Pastor, Jornell Quiambao,\\nKanishka Rao, Jarek Rettinghouse, Diego Reyes, Pierre Sermanet, Nicolas Sievers, Clayton Tan,\\nAlexander Toshev, Vincent Vanhoucke, Fei Xia, Ted Xiao, Peng Xu, Sichun Xu, and Mengyuan\\nYan. Do as i can, not as i say: Grounding language in robotic affordances. arXiv preprint arXiv:\\nArxiv-2204.01691, 2022.\\n[26] Wenlong Huang, Fei Xia, Ted Xiao, Harris Chan, Jacky Liang, Pete Florence, Andy Zeng,\\nJonathan Tompson, Igor Mordatch, Yevgen Chebotar, Pierre Sermanet, Noah Brown, Tomas\\nJackson, Linda Luu, Sergey Levine, Karol Hausman, and Brian Ichter. Inner monologue:\\nEmbodied reasoning through planning with language models. arXiv preprint arXiv: Arxiv-\\n2207.05608, 2022.\\n[27] Wenlong Huang, Pieter Abbeel, Deepak Pathak, and Igor Mordatch. Language models as zero-\\nshot planners: Extracting actionable knowledge for embodied agents. In Kamalika Chaudhuri,\\nStefanie Jegelka, Le Song, Csaba Szepesvári, Gang Niu, and Sivan Sabato, editors, International\\nConference on Machine Learning, ICML 2022, 17-23 July 2022, Baltimore, Maryland, USA,\\nvolume 162 of Proceedings of Machine Learning Research, pages 9118–9147. PMLR, 2022.\\n[28] Significant-gravitas/auto-gpt: An experimental open-source attempt to make gpt-4 fully au-\\ntonomous., 2023.\\n[29] Shunyu Yao, Jeffrey Zhao, Dian Yu, Nan Du, Izhak Shafran, Karthik Narasimhan, and Yuan\\nCao. React: Synergizing reasoning and acting in language models. arXiv preprint arXiv:\\nArxiv-2210.03629, 2022.\\n[30] Noah Shinn, Beck Labash, and Ashwin Gopinath. Reflexion: an autonomous agent with\\ndynamic memory and self-reflection. arXiv preprint arXiv: Arxiv-2303.11366, 2023.\\n[31] German Ignacio Parisi, Ronald Kemker, Jose L. Part, Christopher Kanan, and Stefan Wermter.\\nContinual lifelong learning with neural networks: A review. Neural Networks, 113:54–71, 2019.\\n[32] Liyuan Wang, Xingxing Zhang, Hang Su, and Jun Zhu. A comprehensive survey of continual\\nlearning: Theory, method and application. arXiv preprint arXiv: Arxiv-2302.00487, 2023.\\n[33] Volodymyr Mnih, Koray Kavukcuoglu, David Silver, Alex Graves, Ioannis Antonoglou, Daan\\nWierstra, and Martin Riedmiller. Playing atari with deep reinforcement learning. arXiv preprint\\narXiv: Arxiv-1312.5602, 2013.\\n13\\n\\n\\n[34] OpenAI, :, Christopher Berner, Greg Brockman, Brooke Chan, Vicki Cheung, Przemysław\\nD˛\\nebiak, Christy Dennison, David Farhi, Quirin Fischer, Shariq Hashme, Chris Hesse, Rafal Józe-\\nfowicz, Scott Gray, Catherine Olsson, Jakub Pachocki, Michael Petrov, Henrique P. d. O. Pinto,\\nJonathan Raiman, Tim Salimans, Jeremy Schlatter, Jonas Schneider, Szymon Sidor, Ilya\\nSutskever, Jie Tang, Filip Wolski, and Susan Zhang. Dota 2 with large scale deep reinforcement\\nlearning. arXiv preprint arXiv: Arxiv-1912.06680, 2019.\\n[35] OpenAI. Gpt-4 technical report. arXiv preprint arXiv: Arxiv-2303.08774, 2023.\\n[36] Jason Wei, Yi Tay, Rishi Bommasani, Colin Raffel, Barret Zoph, Sebastian Borgeaud, Dani\\nYogatama, Maarten Bosma, Denny Zhou, Donald Metzler, Ed H. Chi, Tatsunori Hashimoto,\\nOriol Vinyals, Percy Liang, Jeff Dean, and William Fedus. Emergent abilities of large language\\nmodels. arXiv preprint arXiv: Arxiv-2206.07682, 2022.\\n[37] Tom B. Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal,\\nArvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel\\nHerbert-Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel M.\\nZiegler, Jeffrey Wu, Clemens Winter, Christopher Hesse, Mark Chen, Eric Sigler, Mateusz\\nLitwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam McCandlish, Alec\\nRadford, Ilya Sutskever, and Dario Amodei. Language models are few-shot learners. In Hugo\\nLarochelle, Marc’Aurelio Ranzato, Raia Hadsell, Maria-Florina Balcan, and Hsuan-Tien Lin,\\neditors, Advances in Neural Information Processing Systems 33: Annual Conference on Neural\\nInformation Processing Systems 2020, NeurIPS 2020, December 6-12, 2020, virtual, 2020.\\n[38] Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena,\\nYanqi Zhou, Wei Li, and Peter J. Liu. Exploring the limits of transfer learning with a unified\\ntext-to-text transformer. J. Mach. Learn. Res., 21:140:1–140:67, 2020.\\n[39] Benjamin Eysenbach, Abhishek Gupta, Julian Ibarz, and Sergey Levine. Diversity is all you\\nneed: Learning skills without a reward function. In 7th International Conference on Learning\\nRepresentations, ICLR 2019, New Orleans, LA, USA, May 6-9, 2019. OpenReview.net, 2019.\\n[40] Edoardo Conti, Vashisht Madhavan, Felipe Petroski Such, Joel Lehman, Kenneth O. Stanley,\\nand Jeff Clune. Improving exploration in evolution strategies for deep reinforcement learning via\\na population of novelty-seeking agents. In Samy Bengio, Hanna M. Wallach, Hugo Larochelle,\\nKristen Grauman, Nicolò Cesa-Bianchi, and Roman Garnett, editors, Advances in Neural\\nInformation Processing Systems 31: Annual Conference on Neural Information Processing\\nSystems 2018, NeurIPS 2018, December 3-8, 2018, Montréal, Canada, pages 5032–5043, 2018.\\n[41] Mark Chen, Jerry Tworek, Heewoo Jun, Qiming Yuan, Henrique Ponde de Oliveira Pinto,\\nJared Kaplan, Harri Edwards, Yuri Burda, Nicholas Joseph, Greg Brockman, Alex Ray, Raul\\nPuri, Gretchen Krueger, Michael Petrov, Heidy Khlaaf, Girish Sastry, Pamela Mishkin, Brooke\\nChan, Scott Gray, Nick Ryder, Mikhail Pavlov, Alethea Power, Lukasz Kaiser, Mohammad\\nBavarian, Clemens Winter, Philippe Tillet, Felipe Petroski Such, Dave Cummings, Matthias\\nPlappert, Fotios Chantzis, Elizabeth Barnes, Ariel Herbert-Voss, William Hebgen Guss, Alex\\nNichol, Alex Paino, Nikolas Tezak, Jie Tang, Igor Babuschkin, Suchir Balaji, Shantanu Jain,\\nWilliam Saunders, Christopher Hesse, Andrew N. Carr, Jan Leike, Josh Achiam, Vedant Misra,\\nEvan Morikawa, Alec Radford, Matthew Knight, Miles Brundage, Mira Murati, Katie Mayer,\\nPeter Welinder, Bob McGrew, Dario Amodei, Sam McCandlish, Ilya Sutskever, and Wojciech\\nZaremba. Evaluating large language models trained on code. arXiv preprint arXiv: Arxiv-\\n2107.03374, 2021.\\n[42] Rui Wang, Joel Lehman, Jeff Clune, and Kenneth O. Stanley. Paired open-ended trailblazer\\n(poet): Endlessly generating increasingly complex and diverse learning environments and their\\nsolutions. arXiv preprint arXiv: Arxiv-1901.01753, 2019.\\n[43] Rémy Portelas, Cédric Colas, Lilian Weng, Katja Hofmann, and Pierre-Yves Oudeyer. Auto-\\nmatic curriculum learning for deep RL: A short survey. In Christian Bessiere, editor, Proceedings\\nof the Twenty-Ninth International Joint Conference on Artificial Intelligence, IJCAI 2020, pages\\n4819–4825. ijcai.org, 2020.\\n14\\n\\n\\n[44] Sébastien Forestier, Rémy Portelas, Yoan Mollard, and Pierre-Yves Oudeyer. Intrinsically\\nmotivated goal exploration processes with automatic curriculum learning. The Journal of\\nMachine Learning Research, 23(1):6818–6858, 2022.\\n[45] Kevin Ellis, Catherine Wong, Maxwell Nye, Mathias Sable-Meyer, Luc Cary, Lucas Morales,\\nLuke Hewitt, Armando Solar-Lezama, and Joshua B. Tenenbaum. Dreamcoder: Growing\\ngeneralizable, interpretable knowledge with wake-sleep bayesian program learning. arXiv\\npreprint arXiv: Arxiv-2006.08381, 2020.\\n[46] Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Ed Chi, Quoc Le, and Denny\\nZhou. Chain of thought prompting elicits reasoning in large language models. arXiv preprint\\narXiv: Arxiv-2201.11903, 2022.\\n[47] Volodymyr Mnih, Adrià Puigdomènech Badia, Mehdi Mirza, Alex Graves, Timothy P. Lillicrap,\\nTim Harley, David Silver, and Koray Kavukcuoglu. Asynchronous methods for deep rein-\\nforcement learning. In Maria-Florina Balcan and Kilian Q. Weinberger, editors, Proceedings\\nof the 33nd International Conference on Machine Learning, ICML 2016, New York City, NY,\\nUSA, June 19-24, 2016, volume 48 of JMLR Workshop and Conference Proceedings, pages\\n1928–1937. JMLR.org, 2016.\\n[48] John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal\\npolicy optimization algorithms. arXiv preprint arXiv: Arxiv-1707.06347, 2017.\\n[49] Timothy P. Lillicrap, Jonathan J. Hunt, Alexander Pritzel, Nicolas Heess, Tom Erez, Yuval\\nTassa, David Silver, and Daan Wierstra. Continuous control with deep reinforcement learning.\\nIn Yoshua Bengio and Yann LeCun, editors, 4th International Conference on Learning Repre-\\nsentations, ICLR 2016, San Juan, Puerto Rico, May 2-4, 2016, Conference Track Proceedings,\\n2016.\\n[50] Introducing chatgpt, 2022.\\n[51] New and improved embedding model, 2022.\\n[52] PrismarineJS. Prismarinejs/mineflayer: Create minecraft bots with a powerful, stable, and high\\nlevel javascript api., 2013.\\n[53] Kolby Nottingham, Prithviraj Ammanabrolu, Alane Suhr, Yejin Choi, Hanna Hajishirzi, Sameer\\nSingh, and Roy Fox. Do embodied agents dream of pixelated sheep?: Embodied decision\\nmaking using language guided world modelling. ARXIV.ORG, 2023.\\n[54] Shaofei Cai, Zihao Wang, Xiaojian Ma, Anji Liu, and Yitao Liang. Open-world multi-task\\ncontrol through goal-aware representation learning and adaptive horizon prediction. arXiv\\npreprint arXiv: Arxiv-2301.10034, 2023.\\n[55] Zihao Wang, Shaofei Cai, Anji Liu, Xiaojian Ma, and Yitao Liang. Describe, explain, plan and\\nselect: Interactive planning with large language models enables open-world multi-task agents.\\narXiv preprint arXiv: Arxiv-2302.01560, 2023.\\n[56] Sébastien Bubeck, Varun Chandrasekaran, Ronen Eldan, Johannes Gehrke, Eric Horvitz, Ece\\nKamar, Peter Lee, Yin Tat Lee, Yuanzhi Li, Scott Lundberg, Harsha Nori, Hamid Palangi,\\nMarco Tulio Ribeiro, and Yi Zhang. Sparks of artificial general intelligence: Early experiments\\nwith gpt-4. arXiv preprint arXiv: Arxiv-2303.12712, 2023.\\n[57] Yiheng Liu, Tianle Han, Siyuan Ma, Jiayue Zhang, Yuanyuan Yang, Jiaming Tian, Hao He,\\nAntong Li, Mengshen He, Zhengliang Liu, Zihao Wu, Dajiang Zhu, Xiang Li, Ning Qiang,\\nDingang Shen, Tianming Liu, and Bao Ge. Summary of chatgpt/gpt-4 research and perspective\\ntowards the future of large language models. arXiv preprint arXiv: Arxiv-2304.01852, 2023.\\n[58] Shikun Liu, Linxi Fan, Edward Johns, Zhiding Yu, Chaowei Xiao, and Anima Anandkumar.\\nPrismer: A vision-language model with an ensemble of experts. arXiv preprint arXiv: Arxiv-\\n2303.02506, 2023.\\n15\\n\\n\\n[59] Danny Driess, Fei Xia, Mehdi S. M. Sajjadi, Corey Lynch, Aakanksha Chowdhery, Brian Ichter,\\nAyzaan Wahid, Jonathan Tompson, Quan Vuong, Tianhe Yu, Wenlong Huang, Yevgen Chebotar,\\nPierre Sermanet, Daniel Duckworth, Sergey Levine, Vincent Vanhoucke, Karol Hausman, Marc\\nToussaint, Klaus Greff, Andy Zeng, Igor Mordatch, and Pete Florence. Palm-e: An embodied\\nmultimodal language model. arXiv preprint arXiv: Arxiv-2303.03378, 2023.\\n[60] Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timo-\\nthée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, Aurelien Rodriguez,\\nArmand Joulin, Edouard Grave, and Guillaume Lample. Llama: Open and efficient foundation\\nlanguage models. arXiv preprint arXiv: Arxiv-2302.13971, 2023.\\n[61] William H. Guss, Brandon Houghton, Nicholay Topin, Phillip Wang, Cayden Codel, Manuela\\nVeloso, and Ruslan Salakhutdinov. Minerl: A large-scale dataset of minecraft demonstrations.\\nIn Sarit Kraus, editor, Proceedings of the Twenty-Eighth International Joint Conference on\\nArtificial Intelligence, IJCAI 2019, Macao, China, August 10-16, 2019, pages 2442–2448.\\nijcai.org, 2019.\\n[62] William H. Guss, Cayden Codel, Katja Hofmann, Brandon Houghton, Noboru Kuno, Stephanie\\nMilani, Sharada Mohanty, Diego Perez Liebana, Ruslan Salakhutdinov, Nicholay Topin,\\nManuela Veloso, and Phillip Wang. The minerl 2019 competition on sample efficient re-\\ninforcement learning using human priors. arXiv preprint arXiv: Arxiv-1904.10079, 2019.\\n[63] William H. Guss, Mario Ynocente Castro, Sam Devlin, Brandon Houghton, Noboru Sean Kuno,\\nCrissman Loomis, Stephanie Milani, Sharada Mohanty, Keisuke Nakata, Ruslan Salakhutdinov,\\nJohn Schulman, Shinya Shiroshita, Nicholay Topin, Avinash Ummadisingu, and Oriol Vinyals.\\nThe minerl 2020 competition on sample efficient reinforcement learning using human priors.\\narXiv preprint arXiv: Arxiv-2101.11071, 2021.\\n[64] Anssi Kanervisto, Stephanie Milani, Karolis Ramanauskas, Nicholay Topin, Zichuan Lin, Jun-\\nyou Li, Jianing Shi, Deheng Ye, Qiang Fu, Wei Yang, Weijun Hong, Zhongyue Huang, Haicheng\\nChen, Guangjun Zeng, Yue Lin, Vincent Micheli, Eloi Alonso, François Fleuret, Alexander\\nNikulin, Yury Belousov, Oleg Svidchenko, and Aleksei Shpilman. Minerl diamond 2021\\ncompetition: Overview, results, and lessons learned. arXiv preprint arXiv: Arxiv-2202.10583,\\n2022.\\n[65] Matthew Johnson, Katja Hofmann, Tim Hutton, and David Bignell. The malmo platform for\\nartificial intelligence experimentation. In Subbarao Kambhampati, editor, Proceedings of the\\nTwenty-Fifth International Joint Conference on Artificial Intelligence, IJCAI 2016, New York,\\nNY, USA, 9-15 July 2016, pages 4246–4247. IJCAI/AAAI Press, 2016.\\n[66] Zichuan Lin, Junyou Li, Jianing Shi, Deheng Ye, Qiang Fu, and Wei Yang. Juewu-mc: Playing\\nminecraft with sample-efficient hierarchical reinforcement learning. arXiv preprint arXiv:\\nArxiv-2112.04907, 2021.\\n[67] Hangyu Mao, Chao Wang, Xiaotian Hao, Yihuan Mao, Yiming Lu, Chengjie Wu, Jianye\\nHao, Dong Li, and Pingzhong Tang. Seihai: A sample-efficient hierarchical ai for the minerl\\ncompetition. arXiv preprint arXiv: Arxiv-2111.08857, 2021.\\n[68] Alexey Skrynnik, Aleksey Staroverov, Ermek Aitygulov, Kirill Aksenov, Vasilii Davydov, and\\nAleksandr I. Panov. Hierarchical deep q-network from imperfect demonstrations in minecraft.\\nCogn. Syst. Res., 65:74–78, 2021.\\n[69] Danijar Hafner, Jurgis Pasukonis, Jimmy Ba, and Timothy Lillicrap. Mastering diverse domains\\nthrough world models. arXiv preprint arXiv: Arxiv-2301.04104, 2023.\\n[70] Ryan Volum, Sudha Rao, Michael Xu, Gabriel DesGarennes, Chris Brockett, Benjamin\\nVan Durme, Olivia Deng, Akanksha Malhotra, and Bill Dolan. Craft an iron sword: Dy-\\nnamically generating interactive game characters by prompting large language models tuned on\\ncode. In Proceedings of the 3rd Wordplay: When Language Meets Games Workshop (Wordplay\\n2022), pages 25–43, Seattle, United States, 2022. Association for Computational Linguistics.\\n[71] Haoqi Yuan, Chi Zhang, Hongcheng Wang, Feiyang Xie, Penglin Cai, Hao Dong, and Zongqing\\nLu. Plan4mc: Skill reinforcement learning and planning for open-world minecraft tasks. arXiv\\npreprint arXiv: 2303.16563, 2023.\\n16\\n\\n\\n[72] Rishi Bommasani, Drew A. Hudson, Ehsan Adeli, Russ Altman, Simran Arora, Sydney von Arx,\\nMichael S. Bernstein, Jeannette Bohg, Antoine Bosselut, Emma Brunskill, Erik Brynjolfsson,\\nShyamal Buch, Dallas Card, Rodrigo Castellon, Niladri Chatterji, Annie Chen, Kathleen Creel,\\nJared Quincy Davis, Dora Demszky, Chris Donahue, Moussa Doumbouya, Esin Durmus, Stefano\\nErmon, John Etchemendy, Kawin Ethayarajh, Li Fei-Fei, Chelsea Finn, Trevor Gale, Lauren\\nGillespie, Karan Goel, Noah Goodman, Shelby Grossman, Neel Guha, Tatsunori Hashimoto,\\nPeter Henderson, John Hewitt, Daniel E. Ho, Jenny Hong, Kyle Hsu, Jing Huang, Thomas\\nIcard, Saahil Jain, Dan Jurafsky, Pratyusha Kalluri, Siddharth Karamcheti, Geoff Keeling,\\nFereshte Khani, Omar Khattab, Pang Wei Koh, Mark Krass, Ranjay Krishna, Rohith Kuditipudi,\\nAnanya Kumar, Faisal Ladhak, Mina Lee, Tony Lee, Jure Leskovec, Isabelle Levent, Xiang Lisa\\nLi, Xuechen Li, Tengyu Ma, Ali Malik, Christopher D. Manning, Suvir Mirchandani, Eric\\nMitchell, Zanele Munyikwa, Suraj Nair, Avanika Narayan, Deepak Narayanan, Ben Newman,\\nAllen Nie, Juan Carlos Niebles, Hamed Nilforoshan, Julian Nyarko, Giray Ogut, Laurel Orr,\\nIsabel Papadimitriou, Joon Sung Park, Chris Piech, Eva Portelance, Christopher Potts, Aditi\\nRaghunathan, Rob Reich, Hongyu Ren, Frieda Rong, Yusuf Roohani, Camilo Ruiz, Jack\\nRyan, Christopher Ré, Dorsa Sadigh, Shiori Sagawa, Keshav Santhanam, Andy Shih, Krishnan\\nSrinivasan, Alex Tamkin, Rohan Taori, Armin W. Thomas, Florian Tramèr, Rose E. Wang,\\nWilliam Wang, Bohan Wu, Jiajun Wu, Yuhuai Wu, Sang Michael Xie, Michihiro Yasunaga,\\nJiaxuan You, Matei Zaharia, Michael Zhang, Tianyi Zhang, Xikun Zhang, Yuhui Zhang, Lucia\\nZheng, Kaitlyn Zhou, and Percy Liang. On the opportunities and risks of foundation models.\\narXiv preprint arXiv: Arxiv-2108.07258, 2021.\\n[73] Aakanksha Chowdhery, Sharan Narang, Jacob Devlin, Maarten Bosma, Gaurav Mishra, Adam\\nRoberts, Paul Barham, Hyung Won Chung, Charles Sutton, Sebastian Gehrmann, Parker\\nSchuh, Kensen Shi, Sasha Tsvyashchenko, Joshua Maynez, Abhishek Rao, Parker Barnes,\\nYi Tay, Noam Shazeer, Vinodkumar Prabhakaran, Emily Reif, Nan Du, Ben Hutchinson,\\nReiner Pope, James Bradbury, Jacob Austin, Michael Isard, Guy Gur-Ari, Pengcheng Yin,\\nToju Duke, Anselm Levskaya, Sanjay Ghemawat, Sunipa Dev, Henryk Michalewski, Xavier\\nGarcia, Vedant Misra, Kevin Robinson, Liam Fedus, Denny Zhou, Daphne Ippolito, David\\nLuan, Hyeontaek Lim, Barret Zoph, Alexander Spiridonov, Ryan Sepassi, David Dohan, Shivani\\nAgrawal, Mark Omernick, Andrew M. Dai, Thanumalayan Sankaranarayana Pillai, Marie Pellat,\\nAitor Lewkowycz, Erica Moreira, Rewon Child, Oleksandr Polozov, Katherine Lee, Zongwei\\nZhou, Xuezhi Wang, Brennan Saeta, Mark Diaz, Orhan Firat, Michele Catasta, Jason Wei,\\nKathy Meier-Hellstern, Douglas Eck, Jeff Dean, Slav Petrov, and Noah Fiedel. Palm: Scaling\\nlanguage modeling with pathways. arXiv preprint arXiv: Arxiv-2204.02311, 2022.\\n[74] Hyung Won Chung, Le Hou, Shayne Longpre, Barret Zoph, Yi Tay, William Fedus, Eric Li,\\nXuezhi Wang, Mostafa Dehghani, Siddhartha Brahma, Albert Webson, Shixiang Shane Gu,\\nZhuyun Dai, Mirac Suzgun, Xinyun Chen, Aakanksha Chowdhery, Sharan Narang, Gaurav\\nMishra, Adams Yu, Vincent Zhao, Yanping Huang, Andrew Dai, Hongkun Yu, Slav Petrov,\\nEd H. Chi, Jeff Dean, Jacob Devlin, Adam Roberts, Denny Zhou, Quoc V. Le, and Jason Wei.\\nScaling instruction-finetuned language models. arXiv preprint arXiv: Arxiv-2210.11416, 2022.\\n[75] Jiafei Duan, Samson Yu, Hui Li Tan, Hongyuan Zhu, and Cheston Tan. A survey of embodied\\nAI: from simulators to research tasks. IEEE Trans. Emerg. Top. Comput. Intell., 6(2):230–244,\\n2022.\\n[76] Dhruv Batra, Angel X. Chang, Sonia Chernova, Andrew J. Davison, Jia Deng, Vladlen Koltun,\\nSergey Levine, Jitendra Malik, Igor Mordatch, Roozbeh Mottaghi, Manolis Savva, and Hao Su.\\nRearrangement: A challenge for embodied ai. arXiv preprint arXiv: Arxiv-2011.01975, 2020.\\n[77] Harish Ravichandar, Athanasios S Polydoros, Sonia Chernova, and Aude Billard. Recent\\nadvances in robot learning from demonstration.\\nAnnual review of control, robotics, and\\nautonomous systems, 3:297–330, 2020.\\n[78] Jack Collins, Shelvin Chand, Anthony Vanderkop, and David Howard. A review of physics\\nsimulators for robotic applications. IEEE Access, 9:51416–51431, 2021.\\n[79] So Yeon Min, Devendra Singh Chaplot, Pradeep Ravikumar, Yonatan Bisk, and R. Salakhutdi-\\nnov. Film: Following instructions in language with modular methods. International Conference\\non Learning Representations, 2021.\\n17\\n\\n\\n[80] Valts Blukis, Chris Paxton, Dieter Fox, Animesh Garg, and Yoav Artzi. A persistent spatial\\nsemantic representation for high-level natural language instruction execution. In 5th Annual\\nConference on Robot Learning, 2021.\\n[81] Varun Nair, Elliot Schumacher, Geoffrey Tso, and Anitha Kannan. Dera: Enhancing large\\nlanguage model completions with dialog-enabled resolving agents. arXiv preprint arXiv:\\nArxiv-2303.17071, 2023.\\n[82] Joon Sung Park, Joseph C. O’Brien, Carrie J. Cai, Meredith Ringel Morris, Percy Liang, and\\nMichael S. Bernstein. Generative agents: Interactive simulacra of human behavior. arXiv\\npreprint arXiv: Arxiv-2304.03442, 2023.\\n[83] Yue Wu, Shrimai Prabhumoye, So Yeon Min, Yonatan Bisk, Ruslan Salakhutdinov, Amos\\nAzaria, Tom Mitchell, and Yuanzhi Li. Spring: Gpt-4 out-performs rl algorithms by studying\\npapers and reasoning. arXiv preprint arXiv: 2305.15486, 2023.\\n[84] Erik Nijkamp, Bo Pang, Hiroaki Hayashi, Lifu Tu, Huan Wang, Yingbo Zhou, Silvio Savarese,\\nand Caiming Xiong. A conversational paradigm for program synthesis. arXiv preprint arXiv:\\nArxiv-2203.13474, 2022.\\n[85] Hung Le, Yue Wang, Akhilesh Deepak Gotmare, Silvio Savarese, and Steven C. H. Hoi. Coderl:\\nMastering code generation through pretrained models and deep reinforcement learning. arXiv\\npreprint arXiv: Arxiv-2207.01780, 2022.\\n[86] Xinyun Chen, Chang Liu, and Dawn Song. Execution-guided neural program synthesis. In 7th\\nInternational Conference on Learning Representations, ICLR 2019, New Orleans, LA, USA,\\nMay 6-9, 2019. OpenReview.net, 2019.\\n[87] Xinyun Chen, Dawn Song, and Yuandong Tian. Latent execution for neural program synthesis.\\narXiv preprint arXiv: Arxiv-2107.00101, 2021.\\n[88] Kevin Ellis, Maxwell I. Nye, Yewen Pu, Felix Sosa, Josh Tenenbaum, and Armando Solar-\\nLezama. Write, execute, assess: Program synthesis with a REPL. In Hanna M. Wallach, Hugo\\nLarochelle, Alina Beygelzimer, Florence d’Alché-Buc, Emily B. Fox, and Roman Garnett,\\neditors, Advances in Neural Information Processing Systems 32: Annual Conference on Neural\\nInformation Processing Systems 2019, NeurIPS 2019, December 8-14, 2019, Vancouver, BC,\\nCanada, pages 9165–9174, 2019.\\n[89] Yujia Li, David Choi, Junyoung Chung, Nate Kushman, Julian Schrittwieser, Rémi Leblond,\\nTom Eccles, James Keeling, Felix Gimeno, Agustin Dal Lago, Thomas Hubert, Peter Choy,\\nCyprien de Masson d’Autume, Igor Babuschkin, Xinyun Chen, Po-Sen Huang, Johannes Welbl,\\nSven Gowal, Alexey Cherepanov, James Molloy, Daniel J. Mankowitz, Esme Sutherland Robson,\\nPushmeet Kohli, Nando de Freitas, Koray Kavukcuoglu, and Oriol Vinyals. Competition-level\\ncode generation with alphacode. arXiv preprint arXiv: Arxiv-2203.07814, 2022.\\n[90] Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser,\\nMatthias Plappert, Jerry Tworek, Jacob Hilton, Reiichiro Nakano, Christopher Hesse, and\\nJohn Schulman. Training verifiers to solve math word problems. arXiv preprint arXiv: Arxiv-\\n2110.14168, 2021.\\n[91] Ansong Ni, Srini Iyer, Dragomir Radev, Ves Stoyanov, Wen tau Yih, Sida I. Wang, and\\nXi Victoria Lin. Lever: Learning to verify language-to-code generation with execution. arXiv\\npreprint arXiv: Arxiv-2302.08468, 2023.\\n[92] Marta Skreta, Naruki Yoshikawa, Sebastian Arellano-Rubach, Zhi Ji, Lasse Bjørn Kristensen,\\nKourosh Darvish, Alán Aspuru-Guzik, Florian Shkurti, and Animesh Garg. Errors are useful\\nprompts: Instruction guided task programming with verifier-assisted iterative prompting. arXiv\\npreprint arXiv: Arxiv-2303.14100, 2023.\\n18\\n\\n\\nA\\nMethod\\nA.1\\nVOYAGER Algorithm\\nPseudocode 1: VOYAGER algorithm.\\ndef\\nvoyager(\\nenvironment ,\\n# environment\\nthat uses code as action\\nspace\\ncurriculum_agent ,\\n# curriculum\\nagent for\\nproposing\\nthe next task\\naction_agent ,\\n# action\\nagent for code\\ngeneration\\ncritic_agent ,\\n# critic\\nagent for self -verification\\nskill_manager ,\\n# skill\\nmanager\\nfor adding new skills and skill\\nretrieval\\n):\\nagent_state = environment.reset ()\\nwhile\\nTrue:\\nexploration_progress = (\\ncurriculum_agent . get_exploration_progress (\\ncurriculum_agent . get_completed_tasks (),\\ncurriculum_agent . get_failed_tasks (),\\n)\\n)\\ntask = curriculum_agent . propose_next_task (\\nagent_state , exploration_progress\\n)\\ncode = None\\nenvironment_feedback = None\\nexecution_errors = None\\ncritique = None\\nsuccess = False\\n# try at most 4 rounds\\nbefore\\nmoving on to the next task\\nfor i in range (4):\\nskills = skill_manager . retrieve_skills (\\ntask , environment_feedback\\n)\\ncode = action_agent. generate_code (\\ntask ,\\ncode ,\\nenvironment_feedback ,\\nexecution_errors ,\\ncritique ,\\nskills ,\\n)\\n(\\nagent_state ,\\nenvironment_feedback ,\\nexecution_errors ,\\n) = environment.step(code)\\nsuccess , critique = critic_agent. check_task_success (\\ntask , agent_state\\n)\\nif success:\\nbreak\\nif success:\\nskill_manager.add_skill(code)\\ncurriculum_agent . add_completed_task (task)\\nelse:\\ncurriculum_agent . add_failed_task (task)\\nA.2\\nPrompting\\nGPT-4 and GPT-3.5 offer users the ability to designate the role of each prompt message among three\\noptions:\\n19\\n\\n\\n• System: A high-level instruction that guides the model behavior throughout the conversation.\\nIt sets the overall tone and objective for the interaction.\\n• User: A detailed instruction that guides the assistant for the next immediate response.\\n• Assistant: A response message generated the model.\\nSee https://platform.openai.com/docs/guides/chat/introduction for more details.\\nTo save token usage, instead of engaging in multi-round conversations, we concatenate a system\\nprompt and a user prompt to obtain each assistant’s response.\\nA.3\\nAutomatic Curriculum\\nA.3.1\\nComponents in the Prompt\\nThe input prompt to GPT-4 consists of several components:\\n(1) Directives encouraging diverse behaviors and imposing constraints (so that the proposed\\ntask is achievable and verifiable): See Sec. A.3.4 for the full prompt;\\n(2) The agent’s current state:\\n• Inventory: A dictionary of items with counts, for example, {‘cobblestone’: 4, ‘furnace’:\\n1, ‘stone_pickaxe’: 1, ‘oak_planks’: 7, ‘dirt’: 6, ‘wooden_pickaxe’: 1, ‘crafting_table’:\\n1, ‘raw_iron’: 4, ‘coal’: 1};\\n• Equipment: Armors or weapons equipped by the agents;\\n• Nearby blocks: A set of block names within a 32-block distance to the agent, for\\nexample, ‘dirt’, ‘water’, ‘spruce_planks’, ‘grass_block’, ‘dirt_path’, ‘sugar_cane’,\\n‘fern’;\\n• Other blocks that are recently seen: Blocks that are not nearby or in the inventory;\\n• Nearby entities: A set of entity names within a 32-block distance to the agent, for\\nexample, ‘pig’, ‘cat’, ‘villager’, ‘zombie’;\\n• A list of chests that are seen by the agent: Chests are external containers where the\\nagent can deposit items. If a chest is not opened before, its content is “Unknown”.\\nOtherwise, the items inside each chest are shown to the agent.\\n• Biome: For example, ‘plains’, ‘flower_forest’, ‘meadow’, ‘river’, ‘beach’, ‘for-\\nest’, ‘snowy_slopes’, ‘frozen_peaks’, ‘old_growth_birch_forest’, ‘ocean’, ‘sun-\\nflower_plains’, ‘stony_shore’;\\n• Time: One of ‘sunrise’, ‘day’, ‘noon’, ‘sunset’, ‘night’, ‘midnight’;\\n• Health and hunger bars: The max value is 20;\\n• Position: 3D coordinate (x, y, z) of the agent’s position in the Minecraft world;\\n(3) Previously completed and failed tasks;\\n(4) Additional context: See Sec. A.3.2;\\n(5) Chain-of-thought prompting [46] in response: We request GPT-4 to first reason about the\\ncurrent progress and then suggest the next task.\\nA.3.2\\nAdditional Context\\nWe leverage GPT-3.5 to self-ask questions to provide additional context. Each question is paired with\\na concept that is used for retrieving the most relevant document from the wiki knowledge base [23].\\nWe feed the document content to GPT-3.5 for self-answering questions. In practice, using a wiki\\nknowledge base is optional since GPT-3.5 already possesses a good understanding of Minecraft\\ngame mechanics. However, the external knowledge base becomes advantageous if GPT-3.5 is not\\npre-trained in that specific domain. See Sec. A.3.4 for the full prompt.\\nA.3.3\\nWarm-up Schedule\\nIn practice, we adopt a warm-up schedule to gradually incorporate the agent’s state and the additional\\ncontext into the prompt based on how many tasks the agent has completed. This ensures that the\\nprompt is exposed to increasing amounts of information over the exploration progress and therefore\\n20\\n\\n\\nbegins with basic skills and progressively advances towards more intricate and diverse ones. The\\nwarm-up setting that we use across all the experiments is shown in Table. A.1.\\nTable A.1: Warm-up schedule for automatic curriculum.\\nInformation in the prompt\\nAfter how many tasks are completed\\ncore inventory (only including log, planks, stick,\\ncrafting table, furnace, dirt, coal, pickaxe, sword,\\nand axe)\\n0\\nequipment\\n0\\nnearby blocks\\n0\\nposition\\n0\\nnearby entities\\n5\\nfull inventory\\n7\\nother blocks that are recently seen\\n10\\nbiome\\n10\\nhealth bar\\n15\\nhunger bar\\n15\\ntime\\n15\\nadditional context\\n15\\nA.3.4\\nFull Prompt\\nPrompt 1: Full system prompt for automatic curriculum. The list of question-answer pairs represents\\nthe additional context.\\nYou are a helpful\\nassistant\\nthat\\ntells me the next\\nimmediate\\ntask to\\ndo in Minecraft. My ultimate\\ngoal is to discover as many\\ndiverse\\nthings as possible , accomplish as many\\ndiverse\\ntasks as possible\\nand become the best\\nMinecraft\\nplayer in the world.\\nI will give you the\\nfollowing\\ninformation:\\nQuestion 1: ...\\nAnswer: ...\\nQuestion 2: ...\\nAnswer: ...\\nQuestion 3: ...\\nAnswer: ...\\n...\\nBiome: ...\\nTime: ...\\nNearby\\nblocks: ...\\nOther\\nblocks\\nthat are\\nrecently\\nseen: ...\\nNearby\\nentities (nearest to farthest): ...\\nHealth: Higher\\nthan 15 means I’m healthy.\\nHunger: Higher\\nthan 15 means I’m not hungry.\\nPosition: ...\\nEquipment: If I have\\nbetter\\narmor in my inventory , you should ask me\\nto equip it.\\nInventory (xx /36): ...\\nChests: You can ask me to deposit or take\\nitems\\nfrom\\nthese\\nchests.\\nThere\\nalso\\nmight be some\\nunknown chest , you should ask me to open\\nand check\\nitems\\ninside the\\nunknown\\nchest.\\nCompleted\\ntasks so far: ...\\nFailed\\ntasks\\nthat are too hard: ...\\nYou must\\nfollow the\\nfollowing\\ncriteria:\\n1) You should act as a mentor and guide me to the next task\\nbased on\\nmy current\\nlearning\\nprogress.\\n2) Please be very\\nspecific\\nabout\\nwhat\\nresources I need to collect ,\\nwhat I need to craft , or what mobs I need to kill.\\n21\\n\\n\\n3) The next task\\nshould\\nfollow a concise\\nformat , such as \\\"Mine [\\nquantity] [block ]\\\", \\\"Craft [quantity] [item]\\\", \\\"Smelt [quantity] [\\nitem]\\\", \\\"Kill [quantity] [mob]\\\", \\\"Cook [quantity] [food]\\\", \\\"Equip\\n[item ]\\\" etc. It should be a single\\nphrase. Do not\\npropose\\nmultiple\\ntasks at the same time. Do not\\nmention\\nanything\\nelse.\\n4) The next task\\nshould not be too hard\\nsince I may not have the\\nnecessary\\nresources or have\\nlearned\\nenough\\nskills to complete it\\nyet.\\n5) The next task\\nshould be novel and\\ninteresting. I should\\nlook for\\nrare\\nresources , upgrade my equipment\\nand tools\\nusing\\nbetter\\nmaterials , and\\ndiscover\\nnew things. I should not be doing the same\\nthing\\nover and over\\nagain.\\n6) I may\\nsometimes\\nneed to repeat\\nsome\\ntasks if I need to collect\\nmore\\nresources to complete\\nmore\\ndifficult\\ntasks. Only\\nrepeat\\ntasks if\\nnecessary.\\n7) Do not ask me to build or dig\\nshelter\\neven if it ’s at night. I want\\nto explore\\nthe world and\\ndiscover\\nnew things. I don ’t want to\\nstay in one place.\\n8) Tasks\\nthat\\nrequire\\ninformation\\nbeyond the player ’s status to verify\\nshould be avoided. For instance , \\\"Placing 4 torches\\\" and \\\"Dig a 2\\nx1x2 hole\\\" are not ideal\\nsince\\nthey\\nrequire\\nvisual\\nconfirmation\\nfrom the screen. All the placing , building , planting , and\\ntrading\\ntasks\\nshould be avoided. Do not\\npropose\\ntask\\nstarting\\nwith\\nthese\\nkeywords.\\nYou should\\nonly\\nrespond in the format as described\\nbelow:\\nRESPONSE\\nFORMAT:\\nReasoning: Based on the\\ninformation I listed above , do reasoning\\nabout\\nwhat the next task\\nshould be.\\nTask: The next task.\\nHere ’s an example\\nresponse:\\nReasoning: The\\ninventory is empty now , chop down a tree to get some\\nwood.\\nTask: Obtain a wood log.\\nPrompt 2: Full system prompt for asking questions. We provide both good and bad examples as\\nfew-shot exemplars.\\nYou are a helpful\\nassistant\\nthat asks\\nquestions to help me decide the\\nnext\\nimmediate\\ntask to do in Minecraft. My ultimate\\ngoal is to\\ndiscover as many\\nthings as possible , accomplish as many\\ntasks as\\npossible\\nand become the best\\nMinecraft\\nplayer in the world.\\nI will give you the\\nfollowing\\ninformation:\\nBiome: ...\\nTime: ...\\nNearby\\nblocks: ...\\nOther\\nblocks\\nthat are\\nrecently\\nseen: ...\\nNearby\\nentities (nearest to farthest): ...\\nHealth: ...\\nHunger: ...\\nPosition: ...\\nEquipment: ...\\nInventory (xx /36): ...\\nChests: ...\\nCompleted\\ntasks so far: ...\\nFailed\\ntasks\\nthat are too hard: ...\\nYou must\\nfollow the\\nfollowing\\ncriteria:\\n1) You should ask at least 5 questions (but no more than 10 questions)\\nto help me decide the next\\nimmediate\\ntask to do. Each\\nquestion\\nshould be followed by the\\nconcept\\nthat the\\nquestion is about.\\n2) Your\\nquestion\\nshould be specific to a concept in Minecraft.\\nBad\\nexample (the\\nquestion is too\\ngeneral):\\n22\\n\\n\\nQuestion: What is the best way to play\\nMinecraft?\\nConcept: unknown\\nBad\\nexample (axe is still general , you should\\nspecify\\nthe type of\\naxe such as wooden axe):\\nWhat are the\\nbenefits of using an axe to gather\\nresources?\\nConcept: axe\\nGood\\nexample:\\nQuestion: How to make a wooden\\npickaxe?\\nConcept: wooden\\npickaxe\\n3) Your\\nquestions\\nshould be self -contained\\nand not\\nrequire\\nany\\ncontext\\n.\\nBad\\nexample (the\\nquestion\\nrequires\\nthe\\ncontext of my current\\nbiome):\\nQuestion: What are the blocks\\nthat I can find in my current\\nbiome?\\nConcept: unknown\\nBad\\nexample (the\\nquestion\\nrequires\\nthe\\ncontext of my current\\ninventory):\\nQuestion: What are the\\nresources\\nyou need the most\\ncurrently?\\nConcept: unknown\\nBad\\nexample (the\\nquestion\\nrequires\\nthe\\ncontext of my current\\ninventory):\\nQuestion: Do you have any gold or emerald\\nresources?\\nConcept: gold\\nBad\\nexample (the\\nquestion\\nrequires\\nthe\\ncontext of my nearby\\nentities\\n):\\nQuestion: Can you see any\\nanimals\\nnearby\\nthat you can kill for\\nfood?\\nConcept: food\\nBad\\nexample (the\\nquestion\\nrequires\\nthe\\ncontext of my nearby\\nblocks):\\nQuestion: Is there any water\\nsource\\nnearby?\\nConcept: water\\nGood\\nexample:\\nQuestion: What are the blocks\\nthat I can find in the sparse\\njungle\\n?\\nConcept: sparse\\njungle\\n4) Do not ask\\nquestions\\nabout\\nbuilding\\ntasks (such as building a\\nshelter) since\\nthey are too hard for me to do.\\nLet ’s say your\\ncurrent\\nbiome is sparse\\njungle. You can ask\\nquestions\\nlike:\\nQuestion: What are the items\\nthat I can find in the sparse\\njungle?\\nConcept: sparse\\njungle\\nQuestion: What are the mobs that I can find in the sparse\\njungle?\\nConcept: sparse\\njungle\\nLet ’s say you see a creeper\\nnearby , and you have not\\ndefeated a\\ncreeper\\nbefore. You can ask a question\\nlike:\\nQuestion: How to defeat the\\ncreeper?\\nConcept: creeper\\nLet ’s say you last\\ncompleted\\ntask is \\\"Craft a wooden\\npickaxe \\\". You can\\nask a question\\nlike:\\nQuestion: What are the\\nsuggested\\ntasks\\nthat I can do after\\ncrafting a\\nwooden\\npickaxe?\\nConcept: wooden\\npickaxe\\nHere are some more\\nquestion\\nand\\nconcept\\nexamples:\\nQuestion: What are the ores that I can find in the sparse\\njungle?\\nConcept: sparse\\njungle\\n(the above\\nconcept\\nshould not be \\\"ore\\\" because I need to look up the\\npage of \\\"sparse\\njungle\\\" to find out what ores I can find in the\\nsparse\\njungle)\\nQuestion: How can you obtain\\nfood in the sparse\\njungle?\\nConcept: sparse\\njungle\\n23\\n\\n\\n(the above\\nconcept\\nshould not be \\\"food\\\" because I need to look up the\\npage of \\\"sparse\\njungle\\\" to find out what food I can obtain in the\\nsparse\\njungle)\\nQuestion: How can you use the\\nfurnace to upgrade\\nyour\\nequipment\\nand\\nmake\\nuseful\\nitems?\\nConcept: furnace\\nQuestion: How to obtain a diamond\\nore?\\nConcept: diamond\\nore\\nQuestion: What are the\\nbenefits of using a stone\\npickaxe\\nover a wooden\\npickaxe?\\nConcept: stone\\npickaxe\\nQuestion: What are the tools\\nthat you can craft\\nusing\\nwood\\nplanks and\\nsticks?\\nConcept: wood\\nplanks\\nYou should\\nonly\\nrespond in the format as described\\nbelow:\\nRESPONSE\\nFORMAT:\\nReasoning: ...\\nQuestion 1: ...\\nConcept 1: ...\\nQuestion 2: ...\\nConcept 2: ...\\nQuestion 3: ...\\nConcept 3: ...\\nQuestion 4: ...\\nConcept 4: ...\\nQuestion 5: ...\\nConcept 5: ...\\n...\\nPrompt 3: Full system prompt for answering questions. Context represents the optional content from\\na wiki knowledge base.\\nYou are a helpful\\nassistant\\nthat\\nanswer my question\\nabout\\nMinecraft.\\nI will give you the\\nfollowing\\ninformation:\\nQuestion: ...\\nYou will\\nanswer the\\nquestion\\nbased on the\\ncontext (only if available\\nand\\nhelpful) and your own\\nknowledge of Minecraft.\\n1) Start\\nyour\\nanswer\\nwith \\\"Answer: \\\".\\n2) Answer \\\"Answer: Unknown\\\" if you don ’t know the answer.\\nA.4\\nSkill Library\\nA.4.1\\nComponents in the Prompt\\nThe input prompt to GPT-4 consists of the following components:\\n(1) Guidelines for code generation: See Sec A.4.2 for the full prompt;\\n(2) Control primitive APIs implemented by us: These APIs serve a dual purpose: they demon-\\nstrate the usage of Mineflayer APIs, and they can be directly called by GPT-4.\\n• exploreUntil(bot, direction, maxTime = 60, callback): Allow the agent\\nto explore in a fixed direction for maxTime. The callback is the stopping condition\\nimplemented by the agent to determine when to stop exploring;\\n• mineBlock(bot, name, count = 1): Mine and collect the specified number of\\nblocks within a 32-block distance;\\n• craftItem(bot, name, count = 1): Craft the item with a crafting table nearby;\\n• placeItem(bot, name, position): Place the block at the specified position;\\n• smeltItem(bot, itemName, fuelName, count = 1): Smelt the item with the\\nspecified fuel. There must be a furnace nearby;\\n24\\n\\n\\n• killMob(bot, mobName, timeout = 300):\\nAttack the mob and collect its\\ndropped item;\\n• getItemFromChest(bot, chestPosition, itemsToGet): Move to the chest at\\nthe specified position and get items from the chest;\\n• depositItemIntoChest(bot, chestPosition, itemsToDeposit):\\nMove to\\nthe chest at the specified position and deposit items into the chest;\\n(3) Control primitive APIs provided by Mineflayer:\\n• await bot.pathfinder.goto(goal): Go to a specific position. See below for how\\nto set the goal;\\n• new GoalNear(x, y, z, range): Move the bot to a block within the specified\\nrange of the specified block;\\n• new GoalXZ(x, z): For long-range goals that don’t have a specific Y level;\\n• new GoalGetToBlock(x, y, z): Not get into the block, but get directly adjacent\\nto it. Useful for fishing, farming, filling a bucket, and using a bed.;\\n• new GoalFollow(entity, range): Follow the specified entity within the specified\\nrange;\\n• new GoalPlaceBlock(position, bot.world, {}): Position the bot in order to\\nplace a block;\\n• new GoalLookAtBlock(position, bot.world, {}): Path towards a position\\nwhere a face of the block at position is visible;\\n• bot.isABed(bedBlock): Return true if bedBlock is a bed;\\n• bot.blockAt(position): Return the block at position;\\n• await bot.equip(item, destination): Equip the item in the specified destina-\\ntion. destination must be one of “hand”, “head”, “torso”, “legs”, “feet”, “off-hand”;\\n• await bot.consume(): Consume the item in the bot’s hand. You must equip the\\nitem to consume first. Useful for eating food, drinking potions, etc.;\\n• await bot.fish(): Let bot fish. Before calling this function, you must first get to a\\nwater block and then equip a fishing rod. The bot will automatically stop fishing when\\nit catches a fish;\\n• await bot.sleep(bedBlock): Sleep until sunrise. You must get to a bed block\\nfirst;\\n• await bot.activateBlock(block): This is the same as right-clicking a block in\\nthe game. Useful for buttons, doors, etc. You must get to the block first;\\n• await bot.lookAt(position): Look at the specified position. You must go near\\nthe position before you look at it. To fill a bucket with water, you must look at it first;\\n• await bot.activateItem(): This is the same as right-clicking to use the item in\\nthe bot’s hand. Useful for using a bucket, etc. You must equip the item to activate first;\\n• await bot.useOn(entity): This is the same as right-clicking an entity in the game.\\nUseful for shearing a sheep. You must get to the entity first;\\n(4) Retrieved skills from the skill library;\\n(5) Generated code from the last round;\\n(6) Environment feedback: The chat log in the prompt;\\n(7) Execution errors;\\n(8) Critique from the self-verification module;\\n(9) The agent’s current state: See Sec. A.3.1 for each element of the agent’s state;\\n(10) Task proposed by the automatic curriculum;\\n(11) Task context: We prompt GPT-3.5 to ask for general suggestions about how to solve the\\ntask. In practice, this part is handled by the automatic curriculum since it has a systematic\\nmechanism for question-answering (Sec. A.3.2);\\n(12) Chain-of-thought prompting [46] in response: We ask GPT-4 to first explain the reason why\\nthe code from the last round fails, then give step-by-step plans to finish the task, and finally\\ngenerate code. See Sec. A.4.2 for the full prompt.\\n25\\n\\n\\nA.4.2\\nFull Prompt\\nPrompt 4: Full system prompt for code generation.\\nYou are a helpful\\nassistant\\nthat\\nwrites\\nMineflayer\\njavascript\\ncode to\\ncomplete\\nany\\nMinecraft\\ntask\\nspecified by me.\\nHere are some\\nuseful\\nprograms\\nwritten\\nwith\\nMineflayer\\nAPIs.\\n/*\\nExplore\\nuntil\\nfind an iron_ore , use Vec3(0, -1, 0) because\\niron ores\\nare\\nusually\\nunderground\\nawait\\nexploreUntil(bot , new Vec3(0, -1, 0), 60, () => {\\nconst\\niron_ore = bot.findBlock ({\\nmatching: mcData.blocksByName [\\\" iron_ore \\\"].id ,\\nmaxDistance: 32,\\n});\\nreturn\\niron_ore;\\n});\\nExplore\\nuntil\\nfind a pig , use Vec3(1, 0, 1) because\\npigs are\\nusually\\non the\\nsurface\\nlet pig = await\\nexploreUntil(bot , new Vec3(1, 0, 1), 60, () => {\\nconst pig = bot.nearestEntity (( entity) => {\\nreturn (\\nentity.name === \\\"pig\\\" &&\\nentity.position.distanceTo(bot.entity.position) < 32\\n);\\n});\\nreturn pig;\\n});\\n*/\\nasync\\nfunction\\nexploreUntil (bot , direction , maxTime = 60, callback) {\\n/*\\nImplementation of this\\nfunction is omitted.\\ndirection: Vec3 , can only\\ncontain\\nvalue of\\n-1, 0 or 1\\nmaxTime: number , the max time for\\nexploration\\ncallback: function , early\\nstop\\ncondition , will be called\\neach\\nsecond , exploration\\nwill stop if return\\nvalue is not null\\nReturn: null if explore\\ntimeout , otherwise\\nreturn the return\\nvalue\\nof callback\\n*/\\n}\\n// Mine 3 cobblestone: mineBlock(bot , \\\"stone\\\", 3);\\nasync\\nfunction\\nmineBlock(bot , name , count = 1) {\\nconst\\nblocks = bot.findBlocks ({\\nmatching: (block) => {\\nreturn\\nblock.name === name;\\n},\\nmaxDistance: 32,\\ncount: count ,\\n});\\nconst\\ntargets = [];\\nfor (let i = 0; i < Math.min(blocks.length , count); i++) {\\ntargets.push(bot.blockAt(blocks[i]));\\n}\\nawait bot.collectBlock.collect(targets , { ignoreNoPath: true });\\n}\\n// Craft 8 oak_planks\\nfrom 2 oak_log (do the recipe 2 times):\\ncraftItem(bot , \\\"oak_planks\\\", 2);\\n26\\n\\n\\n// You must\\nplace a crafting\\ntable\\nbefore\\ncalling\\nthis\\nfunction\\nasync\\nfunction\\ncraftItem(bot , name , count = 1) {\\nconst\\nitem = mcData.itemsByName[name ];\\nconst\\ncraftingTable = bot.findBlock ({\\nmatching: mcData.blocksByName . crafting_table .id ,\\nmaxDistance: 32,\\n});\\nawait bot.pathfinder.goto(\\nnew\\nGoalLookAtBlock ( craftingTable .position , bot.world)\\n);\\nconst\\nrecipe = bot.recipesFor(item.id , null , 1, craftingTable)[0];\\nawait bot.craft(recipe , count , craftingTable );\\n}\\n// Place a crafting_table\\nnear the player , Vec3(1, 0, 0) is just an\\nexample , you shouldn ’t always use that: placeItem(bot , \\\"\\ncrafting_table \\\", bot.entity.position.offset (1, 0, 0));\\nasync\\nfunction\\nplaceItem(bot , name , position) {\\nconst\\nitem = bot.inventory. findInventoryItem (mcData.itemsByName[\\nname ].id);\\n// find a reference\\nblock\\nconst\\nfaceVectors = [\\nnew Vec3(0, 1, 0),\\nnew Vec3(0, -1, 0),\\nnew Vec3(1, 0, 0),\\nnew Vec3(-1, 0, 0),\\nnew Vec3(0, 0, 1),\\nnew Vec3(0, 0,\\n-1),\\n];\\nlet\\nreferenceBlock = null;\\nlet\\nfaceVector = null;\\nfor (const\\nvector of faceVectors) {\\nconst\\nblock = bot.blockAt(position.minus(vector));\\nif (block ?. name !== \\\"air\\\") {\\nreferenceBlock = block;\\nfaceVector = vector;\\nbreak;\\n}\\n}\\n// You must\\nfirst go to the block\\nposition\\nyou want to place\\nawait bot.pathfinder.goto(new\\nGoalPlaceBlock (position , bot.world ,\\n{}));\\n// You must\\nequip the item\\nright\\nbefore\\ncalling\\nplaceBlock\\nawait bot.equip(item , \\\"hand \\\");\\nawait bot.placeBlock(referenceBlock , faceVector);\\n}\\n// Smelt 1 raw_iron\\ninto 1 iron_ingot\\nusing 1 oak_planks as fuel:\\nsmeltItem(bot , \\\"raw_iron\\\", \\\"oak_planks \\\");\\n// You must\\nplace a furnace\\nbefore\\ncalling\\nthis\\nfunction\\nasync\\nfunction\\nsmeltItem(bot , itemName , fuelName , count = 1) {\\nconst\\nitem = mcData.itemsByName[itemName ];\\nconst\\nfuel = mcData.itemsByName[fuelName ];\\nconst\\nfurnaceBlock = bot.findBlock ({\\nmatching: mcData.blocksByName .furnace.id ,\\nmaxDistance: 32,\\n});\\nawait bot.pathfinder.goto(\\nnew\\nGoalLookAtBlock (furnaceBlock.position , bot.world)\\n);\\nconst\\nfurnace = await bot.openFurnace(furnaceBlock);\\nfor (let i = 0; i < count; i++) {\\nawait\\nfurnace.putFuel(fuel.id , null , 1);\\n27\\n\\n\\nawait\\nfurnace.putInput(item.id , null , 1);\\n// Wait 12 seconds\\nfor the\\nfurnace to smelt the item\\nawait bot.waitForTicks (12 * 20);\\nawait\\nfurnace.takeOutput ();\\n}\\nawait\\nfurnace.close ();\\n}\\n// Kill a pig and\\ncollect\\nthe\\ndropped\\nitem: killMob(bot , \\\"pig\\\", 300);\\nasync\\nfunction\\nkillMob(bot , mobName , timeout = 300) {\\nconst\\nentity = bot.nearestEntity (\\n(entity) =>\\nentity.name ===\\nmobName &&\\nentity.position.distanceTo(bot.entity.position) < 32\\n);\\nawait bot.pvp.attack(entity);\\nawait bot.pathfinder.goto(\\nnew\\nGoalBlock(entity.position.x, entity.position.y, entity.\\nposition.z)\\n);\\n}\\n// Get a torch\\nfrom\\nchest at (30, 65, 100): getItemFromChest (bot , new\\nVec3 (30, 65, 100) , {\\\" torch \\\": 1});\\n// This\\nfunction\\nwill work no matter how far the bot is from the chest\\n.\\nasync\\nfunction\\ngetItemFromChest (bot , chestPosition , itemsToGet) {\\nawait\\nmoveToChest(bot , chestPosition );\\nconst\\nchestBlock = bot.blockAt( chestPosition );\\nconst\\nchest = await bot.openContainer (chestBlock);\\nfor (const\\nname in itemsToGet) {\\nconst\\nitemByName = mcData.itemsByName[name ];\\nconst\\nitem = chest. findContainerItem (itemByName.id);\\nawait\\nchest.withdraw(item.type , null , itemsToGet[name ]);\\n}\\nawait\\ncloseChest(bot , chestBlock);\\n}\\n// Deposit a torch\\ninto\\nchest at (30, 65, 100): depositItemIntoChest (\\nbot , new Vec3 (30, 65, 100) , {\\\" torch \\\": 1});\\n// This\\nfunction\\nwill work no matter how far the bot is from the chest\\n.\\nasync\\nfunction\\ndepositItemIntoChest (bot , chestPosition , itemsToDeposit\\n) {\\nawait\\nmoveToChest(bot , chestPosition );\\nconst\\nchestBlock = bot.blockAt( chestPosition );\\nconst\\nchest = await bot.openContainer (chestBlock);\\nfor (const\\nname in itemsToDeposit ) {\\nconst\\nitemByName = mcData.itemsByName[name ];\\nconst\\nitem = bot.inventory. findInventoryItem (itemByName.id);\\nawait\\nchest.deposit(item.type , null , itemsToDeposit [name ]);\\n}\\nawait\\ncloseChest(bot , chestBlock);\\n}\\n// Check the items\\ninside the chest at (30, 65, 100):\\ncheckItemInsideChest (bot , new Vec3 (30, 65, 100));\\n// You only need to call this\\nfunction\\nonce\\nwithout\\nany action to\\nfinish\\ntask of checking\\nitems\\ninside the chest.\\nasync\\nfunction\\ncheckItemInsideChest (bot , chestPosition ) {\\nawait\\nmoveToChest(bot , chestPosition );\\nconst\\nchestBlock = bot.blockAt( chestPosition );\\nawait bot.openContainer (chestBlock);\\n// You must\\nclose the chest\\nafter\\nopening it if you are asked to\\nopen a chest\\n28\\n\\n\\nawait\\ncloseChest(bot , chestBlock);\\n}\\nawait bot.pathfinder.goto(goal); // A very\\nuseful\\nfunction. This\\nfunction\\nmay change\\nyour main -hand\\nequipment.\\n// Following\\nare some\\nGoals you can use:\\nnew\\nGoalNear(x, y, z, range); // Move the bot to a block\\nwithin the\\nspecified\\nrange of the\\nspecified\\nblock. ‘x‘, ‘y‘, ‘z‘, and ‘range ‘\\nare ‘number ‘\\nnew GoalXZ(x, z); // Useful for long -range\\ngoals\\nthat don ’t have a\\nspecific Y level. ‘x‘ and ‘z‘ are ‘number ‘\\nnew\\nGoalGetToBlock (x, y, z); // Not get into the block , but get\\ndirectly\\nadjacent to it. Useful for fishing , farming , filling\\nbucket , and beds. ‘x‘, ‘y‘, and ‘z‘ are ‘number ‘\\nnew\\nGoalFollow(entity , range); // Follow the\\nspecified\\nentity\\nwithin\\nthe\\nspecified\\nrange. ‘entity ‘ is ‘Entity ‘, ‘range ‘ is ‘number ‘\\nnew\\nGoalPlaceBlock (position , bot.world , {}); // Position\\nthe bot in\\norder to place a block. ‘position ‘ is ‘Vec3 ‘\\nnew\\nGoalLookAtBlock (position , bot.world , {}); // Path into a position\\nwhere a blockface of the block at position is visible. ‘position ‘\\nis ‘Vec3 ‘\\n// These are other\\nMineflayer\\nfunctions\\nyou can use:\\nbot.isABed(bedBlock); // Return\\ntrue if ‘bedBlock ‘ is a bed\\nbot.blockAt(position); // Return the block at ‘position ‘. ‘position ‘\\nis ‘Vec3 ‘\\n// These are other\\nMineflayer\\nasync\\nfunctions\\nyou can use:\\nawait bot.equip(item , destination); // Equip the item in the\\nspecified\\ndestination. ‘item ‘ is ‘Item ‘, ‘destination ‘ can only be \\\"hand\\\",\\n\\\"head\\\", \\\"torso\\\", \\\"legs\\\", \\\"feet\\\", \\\"off -hand\\\"\\nawait bot.consume (); // Consume\\nthe item in the bot ’s hand. You must\\nequip the item to consume\\nfirst. Useful for eating food , drinking\\npotions , etc.\\nawait bot.fish (); // Let bot fish. Before\\ncalling\\nthis function , you\\nmust\\nfirst get to a water\\nblock and then\\nequip a fishing\\nrod. The\\nbot will\\nautomatically\\nstop\\nfishing\\nwhen it catches a fish\\nawait bot.sleep(bedBlock); // Sleep\\nuntil\\nsunrise. You must get to a\\nbed block\\nfirst\\nawait bot.activateBlock(block); // This is the same as right -clicking\\na block in the game. Useful for buttons , doors , using hoes , etc.\\nYou must get to the block\\nfirst\\nawait bot.lookAt(position); // Look at the\\nspecified\\nposition. You\\nmust go near the\\nposition\\nbefore you look at it. To fill\\nbucket\\nwith water , you must\\nlookAt\\nfirst. ‘position ‘ is ‘Vec3 ‘\\nawait bot.activateItem (); // This is the same as right -clicking to use\\nthe item in the bot ’s hand. Useful for using buckets , etc. You\\nmust\\nequip the item to activate\\nfirst\\nawait bot.useOn(entity); // This is the same as right -clicking an\\nentity in the game. Useful for\\nshearing sheep , equipping\\nharnesses\\n, etc. You must get to the entity\\nfirst\\n{ retrieved_skills }\\nAt each\\nround of conversation , I will give you\\nCode from the last\\nround: ...\\nExecution\\nerror: ...\\nChat log: ...\\nBiome: ...\\nTime: ...\\nNearby\\nblocks: ...\\nNearby\\nentities (nearest to farthest):\\nHealth: ...\\n29\\n\\n\\nHunger: ...\\nPosition: ...\\nEquipment: ...\\nInventory (xx /36): ...\\nChests: ...\\nTask: ...\\nContext: ...\\nCritique: ...\\nYou should\\nthen\\nrespond to me with\\nExplain (if applicable): Are there any steps\\nmissing in your plan? Why\\ndoes the code not\\ncomplete\\nthe task? What does the chat log and\\nexecution\\nerror\\nimply?\\nPlan: How to complete\\nthe task step by step. You should pay\\nattention\\nto Inventory\\nsince it tells\\nwhat you have. The task\\ncompleteness\\ncheck is also\\nbased on your\\nfinal\\ninventory.\\nCode:\\n1) Write an async\\nfunction\\ntaking the bot as the only\\nargument.\\n2) Reuse the above\\nuseful\\nprograms as much as possible.\\n- Use ‘mineBlock(bot , name , count)‘ to collect\\nblocks. Do not\\nuse ‘bot.dig ‘ directly.\\n- Use ‘craftItem(bot , name , count)‘ to craft\\nitems. Do not use\\n‘bot.craft ‘ directly.\\n- Use ‘smeltItem(bot , name\\ncount)‘ to smelt\\nitems. Do not use\\n‘bot.openFurnace ‘ directly.\\n- Use ‘placeItem(bot , name , position)‘ to place\\nblocks. Do not\\nuse ‘bot.placeBlock ‘ directly.\\n- Use ‘killMob(bot , name , timeout)‘ to kill mobs. Do not use ‘\\nbot.attack ‘ directly.\\n3) Your\\nfunction\\nwill be reused for\\nbuilding\\nmore\\ncomplex\\nfunctions. Therefore , you should\\nmake it generic\\nand\\nreusable. You\\nshould not make\\nstrong\\nassumption\\nabout the\\ninventory (as it may\\nbe changed at a later\\ntime), and\\ntherefore\\nyou should\\nalways\\ncheck\\nwhether\\nyou have the\\nrequired\\nitems\\nbefore\\nusing\\nthem. If not ,\\nyou should\\nfirst\\ncollect\\nthe\\nrequired\\nitems and reuse the above\\nuseful\\nprograms.\\n4) Functions in the \\\"Code from the last\\nround\\\" section\\nwill not be\\nsaved or executed. Do not reuse\\nfunctions\\nlisted\\nthere.\\n5) Anything\\ndefined\\noutside a function\\nwill be ignored , define all\\nyour\\nvariables\\ninside\\nyour\\nfunctions.\\n6) Call ‘bot.chat ‘ to show the\\nintermediate\\nprogress.\\n7) Use ‘exploreUntil(bot , direction , maxDistance , callback)‘ when\\nyou cannot\\nfind\\nsomething. You should\\nfrequently\\ncall this\\nbefore\\nmining\\nblocks or killing\\nmobs. You should\\nselect a direction at\\nrandom\\nevery\\ntime\\ninstead of constantly\\nusing (1, 0, 1).\\n8) ‘maxDistance ‘ should\\nalways be 32 for ‘bot.findBlocks ‘ and ‘bot\\n.findBlock ‘. Do not cheat.\\n9) Do not write\\ninfinite\\nloops or recursive\\nfunctions.\\n10) Do not use ‘bot.on ‘ or ‘bot.once ‘ to register\\nevent\\nlisteners.\\nYou\\ndefinitely do not need them.\\n11) Name your\\nfunction in a meaningful\\nway (can infer the task\\nfrom the name).\\nYou should\\nonly\\nrespond in the format as described\\nbelow:\\nRESPONSE\\nFORMAT:\\nExplain: ...\\nPlan:\\n1) ...\\n2) ...\\n3) ...\\n...\\nCode:\\n‘‘‘javascript\\n// helper\\nfunctions (only if needed , try to avoid\\nthem)\\n...\\n30\\n\\n\\n// main\\nfunction\\nafter the helper\\nfunctions\\nasync\\nfunction\\nyourMainFunctionName (bot) {\\n// ...\\n}\\n‘‘‘\\nPrompt 5: Full system prompt for generating function descriptions. This is used when adding a new\\nskill to the skill library. We give a one-shot example in the prompt.\\nYou are a helpful\\nassistant\\nthat\\nwrites a description of the given\\nfunction\\nwritten in Mineflayer\\njavascript\\ncode.\\n1) Do not\\nmention\\nthe\\nfunction\\nname.\\n2) Do not\\nmention\\nanything\\nabout ‘bot.chat ‘ or helper\\nfunctions.\\n3) There\\nmight be some\\nhelper\\nfunctions\\nbefore the main function , but\\nyou only need to describe\\nthe main\\nfunction.\\n4) Try to summarize\\nthe\\nfunction in no more than 6 sentences.\\n5) Your\\nresponse\\nshould be a single\\nline of text.\\nFor example , if the\\nfunction is:\\nasync\\nfunction\\nmineCobblestone (bot) {\\n// Check if the wooden\\npickaxe is in the inventory , if not , craft\\none\\nlet\\nwoodenPickaxe = bot.inventory. findInventoryItem (mcData.\\nitemsByName [\\\" wooden_pickaxe \\\"].id);\\nif (! woodenPickaxe) {\\nbot.chat (\\\" Crafting a wooden\\npickaxe .\\\");\\nawait\\ncraftWoodenPickaxe (bot);\\nwoodenPickaxe = bot.inventory. findInventoryItem (mcData.itemsByName\\n[\\\" wooden_pickaxe \\\"].id);\\n}\\n// Equip the wooden\\npickaxe if it exists\\nif (woodenPickaxe) {\\nawait bot.equip(woodenPickaxe , \\\"hand \\\");\\n// Explore\\nuntil we find a stone\\nblock\\nawait\\nexploreUntil(bot , new Vec3(1, -1, 1), 60, () => {\\nconst\\nstone = bot.findBlock ({\\nmatching: mcData.blocksByName [\\\" stone \\\"].id ,\\nmaxDistance: 32\\n});\\nif (stone) {\\nreturn\\ntrue;\\n}\\n});\\n// Mine 8 cobblestone\\nblocks\\nusing the wooden\\npickaxe\\nbot.chat (\\\" Found a stone\\nblock. Mining 8 cobblestone\\nblocks .\\\");\\nawait\\nmineBlock(bot , \\\"stone\\\", 8);\\nbot.chat (\\\" Successfully\\nmined 8 cobblestone\\nblocks .\\\");\\n// Save the event of mining 8 cobblestone\\nbot.save (\\\" cobblestone_mined \\\");\\n} else {\\nbot.chat (\\\" Failed to craft a wooden\\npickaxe. Cannot\\nmine\\ncobblestone .\\\");\\n}\\n}\\nThe main\\nfunction is ‘mineCobblestone ‘.\\nThen you would\\nwrite:\\n31\\n\\n\\nThe\\nfunction is about\\nmining 8 cobblestones\\nusing a wooden\\npickaxe.\\nFirst\\ncheck if a wooden\\npickaxe is in the\\ninventory. If not , craft\\none. If the wooden\\npickaxe is available , equip the wooden\\npickaxe\\nin the hand. Next , explore\\nthe\\nenvironment\\nuntil\\nfinding a stone\\nblock. Once a stone\\nblock is found , mine a total of 8 cobblestone\\nblocks\\nusing the wooden\\npickaxe.\\nA.4.3\\nExamples\\nSkill library example 1: craftWoodenPlanks.\\nasync\\nfunction\\ncraftWoodenPlanks (bot) {\\nconst\\nlogNames = [\\\"oak_log\\\", \\\"birch_log\\\", \\\"spruce_log\\\", \\\"jungle_log\\\"\\n, \\\"acacia_log\\\", \\\"dark_oak_log \\\", \\\"mangrove_log\\\"];\\nconst\\nplankNames = [\\\"oak_planks\\\", \\\"birch_planks\\\", \\\" spruce_planks \\\", \\\"\\njungle_planks\\\", \\\"acacia_planks\\\", \\\" dark_oak_planks \\\", \\\"\\nmangrove_planks \\\"];\\nconst\\nlogInInventory = logNames.find(logName => bot.inventory.count(\\nmcData.itemsByName[logName ].id) > 0);\\nif (! logInInventory) {\\nbot.chat(\\\"No wooden log in inventory. Mining a wooden log ...\\\");\\nawait\\nmineWoodLog(bot);\\n}\\nconst\\nlogIndex = logNames.indexOf( logInInventory );\\nconst\\nplankName = plankNames[logIndex ];\\nbot.chat(‘Crafting 4 ${plankName }... ‘);\\nawait\\ncraftItem(bot , plankName , 1);\\nbot.chat(‘4 ${plankName} crafted .‘);\\n}\\nSkill library example 2: mineTenCobbledDeepslateBelowY0.\\nasync\\nfunction\\nmineTenCobbledDeepslateBelowY0 (bot) {\\n// Equip the iron\\npickaxe\\nconst\\nironPickaxe = bot.inventory. findInventoryItem (mcData.\\nitemsByName[\\\"iron_pickaxe\\\"].id);\\nawait bot.equip(ironPickaxe , \\\"hand\\\");\\n// Find\\ncobbled_deepslate\\nblocks\\nbelow Y=0\\nconst\\ncobbledDeepslateBlocks = await\\nexploreUntil(bot , new Vec3(1,\\n-1, 1), 60, () => {\\nconst\\ncobbledDeepslate = bot.findBlock ({\\nmatching: mcData.blocksByName [\\\" cobbled_deepslate \\\"].id ,\\nmaxDistance: 32,\\nposition: pos => pos.y < 0\\n});\\nreturn\\ncobbledDeepslate ;\\n});\\n// Mine 10 cobbled_deepslate\\nblocks\\nawait\\nmineBlock(bot , \\\" cobbled_deepslate \\\", 10);\\nbot.chat(\\\"10 cobbled_deepslate\\nmined\\nbelow Y=0.\\\");\\n}\\nSkill library example 3: smeltFiveRawIronV2.\\nasync\\nfunction\\nfindSuitablePosition (bot) {\\nconst\\noffsets = [new Vec3(1, 0, 0), new Vec3(-1, 0, 0), new Vec3(0,\\n0, 1), new Vec3(0, 0,\\n-1)];\\nfor (const\\noffset of offsets) {\\nconst\\nposition = bot.entity.position.offset(offset.x, offset.y,\\noffset.z);\\nconst\\nblock = bot.blockAt(position);\\n32\\n\\n\\nif (block.name === \\\"air\\\") {\\nreturn\\nposition;\\n}\\n}\\nreturn\\nnull;\\n}\\nasync\\nfunction\\nsmeltFiveRawIron (bot) {\\n// Check if there is a furnace in the\\ninventory\\nconst\\nfurnaceItem = bot.inventory. findInventoryItem (mcData.\\nitemsByName.furnace.id);\\n// If not , craft a furnace\\nusing the\\navailable\\ncobblestone\\nif (! furnaceItem) {\\nawait\\ncraftFurnace(bot);\\n}\\n// Find a suitable\\nposition to place the\\nfurnace\\nconst\\nfurnacePosition = await\\nfindSuitablePosition (bot);\\nif (! furnacePosition) {\\nbot.chat(\\\"Could not find a suitable\\nposition to place the\\nfurnace.\\n\\\");\\nreturn;\\n}\\n// Place the\\nfurnace at the\\nsuitable\\nposition\\nawait\\nplaceItem(bot , \\\"furnace\\\", furnacePosition );\\n// Smelt 5 raw iron\\nusing the\\navailable\\ncoal as fuel\\nawait\\nsmeltItem(bot , \\\"raw_iron\\\", \\\"coal\\\", 5);\\nbot.chat(\\\"5 raw iron\\nsmelted.\\\");\\n}\\nSkill library example 4: fillBucketWithWater.\\nasync\\nfunction\\nfillBucketWithWater (bot) {\\n// Find a water\\nblock\\nnearby\\nconst\\nwaterBlock = await\\nexploreUntil (bot , new Vec3(1, 0, 1), 60, ()\\n=> {\\nconst\\nwater = bot.findBlock ({\\nmatching: mcData.blocksByName .water.id ,\\nmaxDistance: 32\\n});\\nreturn\\nwater;\\n});\\nif (! waterBlock) {\\nbot.chat(\\\"Could not find\\nwater.\\\");\\nreturn;\\n}\\n// Go to the water\\nblock\\nconst\\nadjacentBlock= waterBlock.position.offset (0, 1, 0);\\nawait bot.pathfinder.goto(\\nnew\\nGoalGetToBlock (adjacentBlock .x, adjacentBlock .y,\\nadjacentBlock.z)\\n);\\n// Look at the water\\nblock\\nawait bot.lookAt(waterBlock.position , true);\\n// Equip the bucket\\nconst\\nbucket = bot.inventory. findInventoryItem (mcData.itemsByName.\\nbucket.id);\\nawait bot.equip(bucket , \\\"hand\\\");\\n33\\n\\n\\n// Activate\\nthe bucket to collect\\nwater\\nawait bot.activateItem ();\\nbot.chat(\\\"Filled the bucket\\nwith\\nwater.\\\");\\n}\\nSkill library example 5: catchFiveFishSafely.\\nasync\\nfunction\\ncatchFiveFishSafely (bot) {\\n// Check if the bot has a fishing\\nrod in its\\ninventory\\nlet\\nfishingRod = bot.inventory. findInventoryItem (mcData.itemsByName.\\nfishing_rod.id);\\nif (! fishingRod) {\\nawait\\ncraftFishingRod (bot);\\nfishingRod = bot.inventory. findInventoryItem (mcData.itemsByName.\\nfishing_rod.id);\\n}\\n// Find a nearby\\nwater\\nblock\\nlet\\nwaterBlock;\\nwhile (! waterBlock) {\\nwaterBlock = await\\nexploreUntil(bot , new Vec3(1, 0, 1), 60, () =>\\n{\\nconst\\nfoundWaterBlock = bot.findBlock ({\\nmatching: mcData.blocksByName .water.id ,\\nmaxDistance: 32\\n});\\nreturn\\nfoundWaterBlock ;\\n});\\nif (! waterBlock) {\\nbot.chat(\\\"No path to the water\\nblock. Trying to find\\nanother\\nwater\\nblock ...\\\");\\n}\\n}\\n// Move to a block\\nadjacent to the water\\nblock\\nconst\\nadjacentBlock = waterBlock.position.offset (0, 1, 0);\\nawait bot.pathfinder.goto(new\\nGoalBlock( adjacentBlock .x,\\nadjacentBlock.y, adjacentBlock.z));\\n// Look at the water\\nblock\\nawait bot.lookAt(waterBlock.position);\\n// Equip the\\nfishing\\nrod\\nawait bot.equip(fishingRod , \\\"hand\\\");\\n// Fish in the water 5 times\\nfor (let i = 0; i < 5; i++) {\\ntry {\\nawait bot.fish ();\\nbot.chat(‘Fish ${i + 1} caught .‘);\\n} catch (error) {\\nif (error.message\\n=== \\\"Fishing\\ncancelled\\\") {\\nbot.chat(\\\"Fishing\\nwas\\ncancelled. Trying\\nagain ...\\\");\\ni--; // Retry the same\\niteration\\n} else {\\nthrow\\nerror;\\n}\\n}\\n}\\n}\\n34\\n\\n\\nA.5\\nSelf-Verification\\nA.5.1\\nComponents in the Prompt\\nThe input prompt to GPT-4 consists of the following components:\\n(1) The agent’s state: We exclude other blocks that are recently seen and nearby entities from the\\nagent’s state since they are not useful for assessing the task’s completeness. See Sec. A.3.1\\nfor each element of the agent’s state;\\n(2) Task proposed by the automatic curriculum;\\n(3) Task context: We prompt GPT-3.5 to ask for general suggestions about how to solve the\\ntask. In practice, this part is handled by the automatic curriculum since it has a systematic\\nmechanism for question-answering (Sec. A.3.2);\\n(4) Chain-of-thought prompting [46] in response: We request GPT-4 to initially reason about\\nthe task’s success or failure, then output a boolean variable indicating the task’s outcome,\\nand finally provide a critique to the agent if the task fails.\\n(5) Few-shot examples for in-context learning [36–38].\\nA.5.2\\nFull Prompt\\nPrompt 6: Full system prompt for self-verification.\\nYou are an assistant\\nthat\\nassesses my progress of playing\\nMinecraft\\nand\\nprovides\\nuseful\\nguidance.\\nYou are\\nrequired to evaluate if I have met the task\\nrequirements .\\nExceeding\\nthe task\\nrequirements is also\\nconsidered a success\\nwhile\\nfailing to meet them\\nrequires\\nyou to provide\\ncritique to help me\\nimprove.\\nI will give you the\\nfollowing\\ninformation:\\nBiome: The biome\\nafter the task\\nexecution.\\nTime: The\\ncurrent\\ntime.\\nNearby\\nblocks: The\\nsurrounding\\nblocks. These\\nblocks are not\\ncollected\\nyet. However , this is useful for some\\nplacing or planting\\ntasks.\\nHealth: My current\\nhealth.\\nHunger: My current\\nhunger\\nlevel. For eating task , if my hunger\\nlevel\\nis 20.0, then I successfully\\nate the food.\\nPosition: My current\\nposition.\\nEquipment: My final\\nequipment. For\\ncrafting tasks , I sometimes\\nequip\\nthe\\ncrafted\\nitem.\\nInventory (xx /36): My final\\ninventory. For mining and\\nsmelting tasks ,\\nyou only need to check\\ninventory.\\nChests: If the task\\nrequires me to place\\nitems in a chest , you can\\nfind\\nchest\\ninformation\\nhere.\\nTask: The\\nobjective I need to accomplish.\\nContext: The\\ncontext of the task.\\nYou should\\nonly\\nrespond in JSON\\nformat as described\\nbelow:\\n{\\n\\\"reasoning \\\": \\\"reasoning\\\",\\n\\\"success \\\": boolean ,\\n\\\"critique \\\": \\\"critique\\\",\\n}\\nEnsure the\\nresponse\\ncan be parsed by Python ‘json.loads ‘, e.g.: no\\ntrailing\\ncommas , no single quotes , etc.\\nHere are some\\nexamples:\\nINPUT:\\nInventory\\n(2/36): {’oak_log ’:2, ’spruce_log ’:2}\\n35\\n\\n\\nTask: Mine 3 wood logs\\nRESPONSE:\\n{\\n\\\"reasoning \\\": \\\"You need to mine 3 wood logs. You have 2 oak logs\\nand 2 spruce logs , which add up to 4 wood logs.\\\",\\n\\\"success \\\": true ,\\n\\\"critique \\\": \\\"\\\"\\n}\\nINPUT:\\nInventory\\n(3/36): {’crafting_table ’: 1, ’spruce_planks ’: 6, ’stick ’:\\n4}\\nTask: Craft a wooden\\npickaxe\\nRESPONSE:\\n{\\n\\\"reasoning \\\": \\\"You have\\nenough\\nmaterials to craft a wooden\\npickaxe ,\\nbut you didn ’t craft it.\\\",\\n\\\"success \\\": false ,\\n\\\"critique \\\": \\\"Craft a wooden\\npickaxe\\nwith a crafting\\ntable\\nusing 3\\nspruce\\nplanks and 2 sticks .\\\"\\n}\\nINPUT:\\nInventory\\n(2/36): {’raw_iron ’: 5, ’stone_pickaxe ’: 1}\\nTask: Mine 5 iron_ore\\nRESPONSE:\\n{\\n\\\"reasoning \\\": \\\"Mining\\niron_ore in Minecraft\\nwill get\\nraw_iron. You\\nhave 5 raw_iron in your\\ninventory .\\\",\\n\\\"success \\\": true ,\\n\\\"critique \\\": \\\"\\\"\\n}\\nINPUT:\\nBiome: plains\\nNearby\\nblocks: stone , dirt , grass_block , grass , farmland , wheat\\nInventory\\n(26/36): ...\\nTask:\\nPlant 1 wheat\\nseed.\\nRESPONSE:\\n{\\n\\\"reasoning \\\": \\\"For\\nplanting tasks , inventory\\ninformation is useless\\n. In nearby blocks , there is farmland\\nand wheat , which\\nmeans you\\nsucceed to plant the wheat\\nseed.\\\",\\n\\\"success \\\": true ,\\n\\\"critique \\\": \\\"\\\"\\n}\\nINPUT:\\nInventory\\n(11/36): {... ,’rotten_flesh ’: 1}\\nTask: Kill 1 zombie\\nContext: ...\\nRESPONSE\\n{\\n36\\n\\n\\n\\\"reasoning \\\": \\\"You have\\nrotten\\nflesh in your\\ninventory , which\\nmeans\\nyou\\nsuccessfully\\nkilled one zombie .\\\",\\n\\\"success \\\": true ,\\n\\\"critique \\\": \\\"\\\"\\n}\\nINPUT:\\nHunger: 20.0/20.0\\nInventory\\n(11/36): ...\\nTask: Eat 1 ...\\nContext: ...\\nRESPONSE\\n{\\n\\\"reasoning \\\": \\\"For all eating task , if the player ’s hunger is 20.0 ,\\nthen the player\\nsuccessfully\\nate the food.\\\",\\n\\\"success \\\": true ,\\n\\\"critique \\\": \\\"\\\"\\n}\\nINPUT:\\nNearby\\nblocks: chest\\nInventory\\n(28/36): {’rail ’: 1, ’coal ’: 2, ’oak_planks ’: 13, ’\\ncopper_block ’: 1, ’diorite ’: 7, ’cooked_beef ’: 4, ’granite ’: 22, ’\\ncobbled_deepslate ’: 23, ’feather ’: 4, ’leather ’: 2, ’\\ncooked_chicken ’: 3, ’white_wool ’: 2, ’stick ’: 3, ’black_wool ’: 1,\\n’stone_sword ’: 2, ’stone_hoe ’: 1, ’stone_axe ’: 2, ’stone_shovel ’:\\n2, ’cooked_mutton ’: 4, ’cobblestone_wall ’: 18, ’crafting_table ’:\\n1, ’furnace ’: 1, ’iron_pickaxe ’: 1, ’stone_pickaxe ’: 1, ’\\nraw_copper ’: 12}\\nChests:\\n(81, 131, 16): {’andesite ’: 2, ’dirt ’: 2, ’cobblestone ’: 75, ’\\nwooden_pickaxe ’: 1, ’wooden_sword ’: 1}\\nTask: Deposit\\nuseless\\nitems\\ninto the chest at (81, 131, 16)\\nContext: ...\\nRESPONSE\\n{\\n\\\"reasoning \\\": \\\"You have 28 items in your\\ninventory\\nafter\\ndepositing\\n, which is more than 20. You need to deposit\\nmore\\nitems\\nfrom your\\ninventory to the chest.\\\",\\n\\\"success \\\": false ,\\n\\\"critique \\\": \\\"Deposit\\nmore\\nuseless\\nitems\\nsuch as copper_block ,\\ndiorite , granite , cobbled_deepslate , feather , and\\nleather to meet\\nthe\\nrequirement of having\\nonly 20 occupied\\nslots in your\\ninventory\\n.\\\"\\n}\\nA.6\\nSystem-level Comparison between VOYAGER and Prior Works\\nWe make a system-level comparison in Table. A.2. Voyager stands out as the only method featuring a\\ncombination of automatic curriculum, iterative planning, and a skill library. Moreover, it learns to\\nplay Minecraft without the need for any gradient update.\\n37\\n\\n\\nTable A.2: System-level comparison between VOYAGER and prior works.\\nVPT [8]\\nDreamerV3 [69] DECKARD [53] DEPS [55]\\nPlan4MC [71]\\nVOYAGER\\nDemos\\nVideos\\nNone\\nVideos\\nNone\\nNone\\nNone\\nRewards\\nSparse\\nDense\\nSparse\\nNone\\nDense\\nNone\\nObservations\\nPixels Only\\nPixels &\\nMeta\\nPixels &\\nInventory\\nFeedback &\\nInventory\\nPixels &\\nMeta\\nFeedback &\\nMeta &\\nInventory\\nActions\\nKeyboard\\n&\\nMouse\\nDiscrete\\nKeyboard\\n&\\nMouse\\nKeyboard\\n&\\nMouse\\nDiscrete\\nCode\\nAutomatic\\nCurriculum\\n✓\\n✓\\n(in-context\\nGPT-4\\npro-\\nposal)\\nIterative Plan-\\nning\\n✓\\n✓\\n(3\\ntypes\\nof\\nfeedback)\\nSkill Library\\n✓\\n(pre-defined)\\n✓\\n(self-\\ngenerated)\\nGradient-Free\\n✓\\nB\\nExperiments\\nB.1\\nExperimental Setup\\nOur simulation environment is built upon MineDojo [23] and utilizes Mineflayer [52] JavaScript APIs\\nfor motor controls (Sec. A.4.2). Additionally, we incorporate many bot.chat() into Mineflayer\\nfunctions to provide abundant environment feedback and implement various condition checks along\\nwith try-catch exceptions for continuous execution. If the bot dies, it is resurrected near the closest\\nground, and its inventory is preserved for uninterrupted exploration. The bot recycles its crafting table\\nand furnace after program execution. For detailed implementations, please refer to our codebase.\\nB.2\\nBaselines\\nReAct [29] uses chain-of-thought prompting [46] by generating both reasoning traces and action\\nplans with LLMs. We provide it with our environment feedback and the agent states as observations.\\nReAct undergoes one round of code generation from scratch, followed by three rounds of code\\nrefinement. This process is then repeated until the maximum prompting iteration is reached.\\nReflexion [30] is built on top of ReAct [29] with self-reflection to infer more intuitive future actions.\\nWe provide it with environment feedback, the agent states, execution errors, and our self-verification\\nmodule. Similar to ReAct, Reflexion undergoes one round of code generation from scratch, followed\\nby three rounds of code refinement. This process is then repeated until the maximum prompting\\niteration is reached.\\nAutoGPT [28] is a popular software tool that automates NLP tasks by decomposing a high-level goal\\ninto multiple subgoals and executing them in a ReAct-style loop. We re-implement AutoGPT by\\nusing GPT-4 to do task decomposition and provide it with the agent states, environment feedback,\\nand execution errors as observations for subgoal execution. Compared with VOYAGER, AutoGPT\\nlacks the skill library for accumulating knowledge, self-verification for assessing task success, and\\nautomatic curriculum for open-ended exploration. During each subgoal execution, if no execution\\nerror occurs, we consider the subgoal completed and proceed to the next one. Otherwise, we refine\\nthe program until three rounds of code refinement (equivalent to four rounds of code generation)\\nare completed, and then move on to the next subgoal. If three consecutive subgoals do not result in\\nacquiring a new item, we replan by rerunning the task decomposition.\\nThe task is “explore the world and get as many items as possible” for all baselines.\\n38\\n\\n\\nTable A.3: Comparison between VOYAGER and baselines.\\nReAct [29]\\nReflexion [30]\\nAutoGPT [28]\\nVOYAGER\\nChain-of-Thought [46]\\n✓\\n✓\\n✓\\n✓\\nSelf Verification\\n✓\\n✓\\nEnvironment Feedback\\n✓\\n✓\\n✓\\n✓\\nExecution Errors\\n✓\\n✓\\n✓\\nAgent State\\n✓\\n✓\\n✓\\n✓\\nSkill Library\\n✓\\nAutomatic Curriculum\\n✓\\nFigure A.1: Minecraft item icons with corresponding names.\\nB.3\\nAblations\\nWe ablate 6 design choices (automatic curriculum, skill library, environment feedback, execution\\nerrors, self-verification, and GPT-4 for code generation) in VOYAGER and study their impact on\\nexploration performance.\\n• Manual Curriculum: We substitute the automatic curriculum with a manually designed\\ncurriculum for mining a diamond: “Mine 3 wood log”, “Craft 1 crafting table”, “Craft\\n1 wooden pickaxe”, “Mine 11 cobblestone”, “Craft 1 stone pickaxe”, “Craft 1 furnace”,\\n“Mine 3 iron ore”, “Smelt 3 iron ore”, “Craft 1 iron pickaxe”, “Mine 1 diamond”. A manual\\ncurriculum requires human effort to design and is not scalable for open-ended exploration.\\n• Random Curriculum: We curate 101 items obtained by VOYAGER and create a random\\ncurriculum by randomly selecting one item as the next task.\\n• w/o Skill Library: We remove the skill library, eliminating skill retrieval for code generation.\\n• w/o Environment Feedback: We exclude environment feedback (chat log) from the prompt\\nfor code generation.\\n• w/o Execution Errors: We exclude execution errors from the prompt for code generation.\\n• w/o Self-Verification: For each task, we generate code without self-verification and it-\\neratively refine the program for 3 rounds (equivalent to 4 rounds of code generation in\\ntotal).\\n• GPT-3.5: We replace GPT-4 with GPT-3.5 for code generation. We retain GPT-4 for the\\nautomatic curriculum and the self-verification module.\\nB.4\\nEvaluation Results\\nB.4.1\\nSignificantly Better Exploration\\nThe meaning of each icon in Fig. 1 is shown in Fig. A.1.\\nWe run three trials for each method. The items collected by VOYAGER in each trial is\\n39\\n\\n\\n• Trial 1:\\n‘iron_ingot’, ‘stone_shovel’, ‘iron_leggings’, ‘fishing_rod’, ‘pufferfish’,\\n‘oak_log’, ‘cooked_mutton’, ‘green_dye’, ‘flint’, ‘chest’, ‘iron_sword’, ‘string’, ‘en-\\nder_pearl’, ‘raw_copper’, ‘crafting_table’, ‘cactus’, ‘lapis_lazuli’, ‘iron_pickaxe’, ‘cop-\\nper_ingot’, ‘stone_pickaxe’, ‘wooden_hoe’, ‘scaffolding’, ‘stick’, ‘porkchop’, ‘cop-\\nper_block’, ‘gravel’, ‘grass_block’, ‘white_bed’, ‘bone’, ‘dirt’, ‘mutton’, ‘white_wool’,\\n‘oak_sapling’, ‘coal’, ‘bamboo’, ‘wooden_pickaxe’, ‘rotten_flesh’, ‘cooked_porkchop’,\\n‘cod’, ‘iron_boots’, ‘lightning_rod’, ‘diorite’, ‘water_bucket’, ‘shears’, ‘furnace’, ‘andesite’,\\n‘granite’, ‘bucket’, ‘wooden_sword’, ‘sandstone’, ‘iron_helmet’, ‘raw_iron’, ‘sand’, ‘aca-\\ncia_log’, ‘cooked_cod’, ‘oak_planks’, ‘azure_bluet’, ‘iron_shovel’, ‘acacia_planks’, ‘shield’,\\n‘iron_axe’, ‘iron_chestplate’, ‘cobblestone’;\\n• Trial 2: ‘iron_ingot’, ‘tuff’, ‘stone_shovel’, ‘iron_leggings’, ‘fishing_rod’, ‘cooked_mutton’,\\n‘spruce_planks’, ‘gunpowder’, ‘amethyst_shard’, ‘chest’, ‘string’, ‘cooked_salmon’,\\n‘iron_sword’, ‘raw_copper’, ‘crafting_table’, ‘torch’, ‘lapis_lazuli’, ‘iron_pickaxe’, ‘cop-\\nper_ingot’, ‘stone_pickaxe’, ‘wooden_hoe’, ‘stick’, ‘amethyst_block’, ‘salmon’, ‘cal-\\ncite’, ‘gravel’, ‘white_bed’, ‘bone’, ‘dirt’, ‘mutton’, ‘white_wool’, ‘spyglass’, ‘coal’,\\n‘wooden_pickaxe’, ‘cod’, ‘iron_boots’, ‘lily_pad’, ‘cobbled_deepslate’, ‘lightning_rod’,\\n‘snowball’, ‘stone_axe’, ‘smooth_basalt’, ‘diorite’, ‘water_bucket’, ‘furnace’, ‘andesite’,\\n‘bucket’, ‘granite’, ‘shield’, ‘iron_helmet’, ‘raw_iron’, ‘cobblestone’, ‘spruce_log’,\\n‘cooked_cod’, ‘tripwire_hook’, ‘stone_hoe’, ‘iron_chestplate’, ‘stone_sword’;\\n• Trial 3:\\n‘spruce_planks’, ‘dirt’, ‘shield’, ‘redstone’, ‘clock’, ‘diamond_sword’,\\n‘iron_chestplate’, ‘stone_pickaxe’, ‘leather’, ‘string’, ‘chicken’, ‘chest’, ‘diorite’,\\n‘iron_leggings’, ‘black_wool’, ‘cobblestone_wall’, ‘cobblestone’, ‘cooked_chicken’,\\n‘feather’, ‘stone_sword’, ‘raw_gold’, ‘gravel’, ‘birch_planks’, ‘coal’, ‘cobbled_deepslate’,\\n‘oak_planks’, ‘iron_pickaxe’, ‘granite’, ‘tuff’, ‘crafting_table’, ‘iron_helmet’, ‘stone_hoe’,\\n‘iron_ingot’, ‘stone_axe’, ‘birch_boat’, ‘stick’, ‘sand’, ‘bone’, ‘raw_iron’, ‘beef’, ‘rail’,\\n‘oak_sapling’, ‘kelp’, ‘gold_ingot’, ‘birch_log’, ‘wheat_seeds’, ‘cooked_mutton’, ‘furnace’,\\n‘arrow’, ‘stone_shovel’, ‘white_wool’, ‘andesite’, ‘jungle_slab’, ‘mutton’, ‘iron_sword’,\\n‘copper_ingot’, ‘diamond’, ‘torch’, ‘oak_log’, ‘cooked_beef’, ‘copper_block’, ‘flint’,\\n‘bone_meal’, ‘raw_copper’, ‘wooden_pickaxe’, ‘iron_boots’, ‘wooden_sword’.\\nThe items collected by ReAct [29] in each trial is\\n• Trial 1: ‘bamboo’, ‘dirt’, ‘sand’, ‘wheat_seeds’;\\n• Trial 2: ‘dirt’, ‘rabbit’, ‘spruce_log’, ‘spruce_sapling’;\\n• Trial 3: ‘dirt’, ‘pointed_dripstone’;\\nThe items collected by Reflexion [30] in each trial is\\n• Trial 1: ‘crafting_table’, ‘orange_tulip’, ‘oak_planks’, ‘oak_log’, ‘dirt’;\\n• Trial 2: ‘spruce_log’, ‘dirt’, ‘clay_ball’, ‘sand’, ‘gravel’;\\n• Trial 3: ‘wheat_seeds’, ‘oak_log’, ‘dirt’, ‘birch_log’, ‘sand’.\\nThe items collected by AutoGPT [28] in each trial is\\n• Trial 1: ‘feather’, ‘oak_log’, ‘leather’, ‘stick’, ‘porkchop’, ‘chicken’, ‘crafting_table’,\\n‘wheat_seeds’, ‘oak_planks’, ‘dirt’, ‘mutton’;\\n• Trial 2:\\n‘wooden_pickaxe’, ‘iron_ingot’, ‘stone’, ‘coal’, ‘spruce_planks’, ‘string’,\\n‘raw_copper’, ‘crafting_table’, ‘diorite’, ‘andesite’, ‘furnace’, ‘torch’, ‘spruce_sapling’,\\n‘granite’, ‘iron_pickaxe’, ‘stone_pickaxe’, ‘wooden_axe’, ‘raw_iron’, ‘stick’, ‘spruce_log’,\\n‘dirt’, ‘cobblestone’;\\n• Trial 3: ‘wooden_shovel’, ‘wooden_pickaxe’, ‘iron_ingot’, ‘stone’, ‘cod’, ‘coal’, ‘oak_log’,\\n‘flint’, ‘raw_copper’, ‘crafting_table’, ‘diorite’, ‘furnace’, ‘andesite’, ‘torch’, ‘granite’,\\n‘lapis_lazuli’, ‘iron_pickaxe’, ‘stone_pickaxe’, ‘raw_iron’, ‘stick’, ‘gravel’, ‘oak_planks’,\\n‘dirt’, ‘iron_axe’, ‘cobblestone’.\\n40\\n\\n\\nFigure A.2: Map coverage: Two bird’s eye views of Minecraft maps. VOYAGER is able to traverse\\n2.3× longer distances compared to baselines while crossing diverse terrains. Trajectories are plotted\\nbased on the positions where each agent interacts with GPT-4.\\nB.4.2\\nExtensive Map Traversal\\nAgent trajectories for map coverage are displayed in Fig. A.2. Fig. 7 is plotted based on Fig. A.2 by\\ndrawing the smallest circle enclosing each trajectory. The terrains traversed by VOYAGER in each\\ntrial is\\n• Trial 1: ‘meadow’, ‘desert’, ‘river’, ‘savanna’, ‘forest’, ‘plains’, ‘bamboo_jungle’, ‘drip-\\nstone_caves’;\\n• Trial 2: ‘snowy_plains’, ‘frozen_river’, ‘dripstone_caves’, ‘snowy_taiga’, ‘beach’;\\n• Trial 3:\\n‘flower_forest’,\\n‘meadow’,\\n‘old_growth_birch_forest’,\\n‘snowy_slopes’,\\n‘frozen_peaks’, ‘forest’, ‘river’, ‘beach’, ‘ocean’, ‘sunflower_plains’, ‘plains’, ‘stony_shore’.\\nThe terrains traversed by ReAct [29] in each trial is\\n• Trial 1: ‘plains’, ‘desert’, ‘jungle’;\\n• Trial 2: ‘snowy_plains’, ‘snowy_taiga’, ‘snowy_slopes’;\\n• Trial 3: ‘dark_forest’, ‘dripstone_caves’, ‘grove’, ‘jagged_peaks’.\\nThe terrains traversed by Reflexion [30] in each trial is\\n• Trial 1: ‘plains’, ‘flower_forest’;\\n• Trial 2: ‘snowy_taiga’;\\n• Trial 3: ‘old_growth_birch_forest’, ‘river’, ‘ocean’, ‘beach’, ‘plains’.\\nThe terrains traversed by AutoGPT [28] in each trial is\\n• Trial 1: ‘plains’, ‘dripstone_caves’, ‘savanna’, ‘meadow’;\\n• Trial 2: ‘snowy_taiga’;\\n• Trial 3: ‘plains’, ‘stony_shore’, ‘forest’, ‘ocean’.\\nB.4.3\\nEfficient Zero-Shot Generalization to Unseen Tasks\\nThe results of zero-shot generalization to unseen tasks for the other two tasks are presented in Fig. A.3.\\nSimilar to Fig. 8, VOYAGER consistently solves all tasks, while the baselines are unable to solve any\\n41\\n\\n\\nFigure A.3: Zero-shot generalization to unseen tasks. We visualize the intermediate progress of each\\nmethod on the other two tasks. We do not plot ReAct and Reflexion since they do not make any\\nmeaningful progress.\\ntask within 50 prompting iterations. Our skill library, constructed from lifelong learning, not only\\nenhances VOYAGER’s performance but also provides a boost to AutoGPT [28].\\nB.4.4\\nAccurate Skill Retrieval\\nWe conduct an evaluation of our skill retrieval (309 samples in total) and the results are in Table. A.4.\\nThe top-5 accuracy standing at 96.5% suggests our retrieval process is reliable (note that we include\\nthe top-5 relevant skills in the prompt for synthesizing a new skill).\\nTable A.4: Skill retrieval accuracy.\\nTop-1 Acc\\nTop-2 Acc\\nTop-3 Acc\\nTop-4 Acc\\nTop-5 Acc\\n80.2 ± 3.0\\n89.3 ± 1.8\\n93.2 ± 0.7\\n95.2 ± 1.8\\n96.5 ± 0.3\\nB.4.5\\nRobust to Model Variations\\nIn the main paper, all of Voyager’s experiments are conducted with gpt-4-0314. We additionally\\nrun new experiments with gpt-4-0613 and find that the performance is roughly the same (Fig. A.4).\\nIt demonstrates that Voyager is robust to model variations.\\nFigure A.4: VOYAGER’s performance with GPT-4-0314 and GPT-4-0613.\\n42\",\"difficulty\":\"easy\",\"domain\":\"Multi-Document QA\",\"length\":\"medium\",\"question\":\"Which of the following statements is correct?\",\"sub_domain\":\"Academic\"}","display_format":"text","language":"","answer_status":"published","assets":[],"source_url":"https://huggingface.co/datasets/zai-org/LongBench-v2","history":"initial import","indexing_mode":"noindex","subproblems":[],"grids":[]}