LongBench v2 / 66ec0c4c821e116aacb1994a / Which of the following statements is correct?

Video PreTraining (VPT): Learning to Act by
Watching Unlabeled Online Videos
Abstract
Pretraining on noisy, internet-scale datasets has been heavily studied as a technique
for training models with broad, general capabilities for text, images, and other
modalities.1–6 However, for many sequential decision domains such as robotics,
video games, and computer use, publicly available data does not contain the labels
required to train behavioral priors in the same way. We extend the internet-scale
pretraining paradigm to sequential decision domains through semi-supervised
imitation learning wherein agents learn to act by watching online unlabeled videos.
Specifically, we show that with a small amount of labeled data we can train an
inverse dynamics model accurate enough to label a huge unlabeled source of online
data – here, online videos of people playing Minecraft – from which we can then
train a general behavioral prior. Despite using the native human interface (mouse
and keyboard at 20Hz), we show that this behavioral prior has nontrivial zero-
shot capabilities and that it can be fine-tuned, with both imitation learning and
reinforcement learning, to hard-exploration tasks that are impossible to learn from
scratch via reinforcement learning. For many tasks our models exhibit human-
level performance, and we are the first to report computer agents that can craft
diamond tools, which can take proficient humans upwards of 20 minutes (24,000
environment actions) of gameplay to accomplish.
1
Introduction
Work in recent years has demonstrated the efficacy of pretraining large and general foundation
models7 on noisy internet-scale datasets for use in downstream tasks in natural language1–4 and
computer vision.5,6,8 For sequential decision domains (e.g. robotics, game playing, and computer
usage) where agents must repeatedly act within an environment, a wealth of data also exists on the
web; however, most of this data is in the form of unlabeled video (i.e. without the actions taken
at each frame), making it much less straightforward to train a behavioral prior in these domains
than it is in e.g. natural language. In a few rare settings, such as Chess, Go, and StarCraft, there
∗This was a large effort by a dedicated team. Each author made huge contributions on many fronts over long
time periods. All members were full time on the project for over six months. BB, IA, PZ, and JC were on the
original VPT project team and were thus involved for even longer (over a year). Aside from those original team
members, author order is random. It was also randomized between IA and PZ.
†OpenAI
‡University of British Columbia
arXiv:2206.11795v1  [cs.LG]  23 Jun 2022


already exist large datasets with action labels from various online platforms that researchers have
used for imitation learning.9,10 When large labeled datasets do not exist, the canonical strategy
for training capable agents is reinforcement learning (RL),11 which can be sample inefficient and
expensive for hard-exploration problems.12–18 Many virtual tasks, e.g. navigating websites, using
Photoshop, booking flights, etc., can be very hard to learn with RL and do not have large, commonly
available sources of labeled data.19,20 In this paper, we seek to extend the paradigm of training
large, general-purpose foundation models to sequential decision domains by utilizing freely available
internet-scale unlabeled video datasets with a simple semi-supervised imitation learning method. We
call this method Video PreTraining (VPT) and demonstrate its efficacy in the domain of Minecraft.
Existing semi-supervised imitation learning methods aim to learn with few or no explicit action labels;
however, they generally rely on the policy’s ability to explore the environment throughout training,
making them susceptible to exploration bottlenecks.21–25 Furthermore, most prior semi-supervised
imitation learning work was tested in the relatively low data regime; because we experiment with far
more data (∼70k hours of unlabeled video), we hypothesize that we can achieve good performance
with a much simpler method, a trend that has proven true for pretraining in other modalities such
as text.1 In particular, given a large but unlabeled dataset, we propose generating pseudo-labels by
gathering a small amount of labeled data to train an inverse dynamics model (IDM) that predicts
the action taken at each timestep in a video. Behavioral cloning (BC) can require a large amount
of data because the model must learn to infer intent and the distribution over future behaviors from
only past observations. In contrast, the inverse dynamics modeling task is simpler because it is
non-causal, meaning it can look at both past and future frames to infer actions. In most settings,
environment mechanics are far simpler than the breadth of human behavior that can take place within
the environment, suggesting that non-causal IDMs could require far less data to train than causal BC
models. Using pseudo-labels generated from the IDM, we then train a model to mimic the distribution
of behavior in the previously unlabeled dataset with standard behavioral cloning at scale, which does
not require any model rollouts and thus does not suffer from any potential exploration bottlenecks
in the environment. Finally, we show we can fine-tune this model to downstream tasks with either
behavioral cloning or reinforcement learning.
Figure 1:
Example Minecraft
crafting GUI. Agents use the
mouse and keyboard to navigate
menus and drag and drop items.
We chose to test our method in Minecraft because (a) it is one
of the most actively played games in the world26 and thus has
a wealth of commonly available video data online, (b) it is a
fairly open-ended sandbox game with an extremely wide variety
of potential things to do, build, and collect, making our results
more applicable to real-world applications such as computer us-
age, which also tends to be varied and open-ended, and (c) it
has already garnered interest by the RL community as a research
domain due to its complexity and correspondingly difficult ex-
ploration challenges.27–31 In this work we use the native human
interface for Minecraft so that we can (1) most accurately model
the human behavior distribution and reduce domain shift between
video data and the environment, (2) make data collection easier by allowing our human contractors to
play the game without modification, and (3) eliminate the need to hand-engineer a custom interface
for models to interact with the environment. This choice means that our models play at 20 frames
per second and must use a mouse and keyboard interface to interact with human GUIs for crafting,
smelting, trading, etc., including dragging items to specific slots or navigating the recipe book with
the mouse cursor (Fig. 1). Compared to prior work in Minecraft that uses a lower frame rate and
constructs crafting and attacking macros,30,32–34 using the native human interface drastically increases
the environment’s exploration difficulty, making most simple tasks near impossible with RL from
scratch. Even the simple task of gathering a single wooden log while already facing a tree takes 60
consecutive attack actions with the human interface, meaning the chance for a naive random policy to
succeed is 1
2
60. While this paper shows results in Minecraft only, the VPT method is general and
could be applied to any domain.
In Section 4 we show that the VPT foundation model has nontrivial zero-shot performance, accom-
plishing tasks impossible to learn with RL alone, such as crafting planks and crafting tables (tasks
requiring a human proficient in Minecraft a median of 50 seconds or ∼970 consecutive actions).
Through fine-tuning with behavioral cloning to smaller datasets that target more specific behavior
distributions, our agent is able to push even further into the technology tree, crafting stone tools
2


(taking a human a median of 2.3 minutes or ∼2790 actions). Finally, fine-tuning via RL produces
the most dramatic improvements: our agent is able to craft diamond tools, an unprecedented result
in Minecraft made even more challenging by using the native human interface. This task requires
a proficient human a median upwards of 20 minutes or ∼24000 actions. The main contributions
of this work are (1) we are the first to show promising results applying semi-supervised imitation
learning to extremely large, noisy, and freely available video datasets for sequential decision domains,
(2) we show that such pretraining plus fine-tuning enables agents to solve tasks that were otherwise
impossible to learn, (3) we show that labeled contractor data is far more efficiently used within
the VPT method than it would be by directly training a foundation model from it and (4) we open
source our contractor data, trained model weights, and Minecraft environment for future research
into learning to act via semi-supervised imitation learning at scale.
2
Preliminaries and Related Work
Imitation learning methods35–38 seek to construct a policy that accurately models the distribution of
behavior in some dataset D = {(oi, ai)}, i ∈{1...N} of action-observation pairs. In order to roll
out these policies in an environment, they must be causal, meaning they condition on observations
from the current timestep t and past timesteps only, i.e. π ∼p(at|o1...ot). Imitation learning is
simplest when demonstrations are labeled with corresponding actions. Imitating labeled trajectories
has seen success in aerial vehicles,39,40 self-driving cars,41,42 board games,9,43 and video games.10,44
When labeled demonstrations are not available, standard behavioral cloning will not work; however,
there is a large body of work in imitating behavior from unlabeled demonstrations.22 For instance,
GAIL23 constructs an adversarial objective incentivizing the trained policy to exhibit behaviors
indistinguishable from those in the target dataset. Edwards et al. 45 propose to first learn a latent
policy using unlabeled demonstrations and then map the learned latent actions to real actions with
a small amount of environment interaction. Peng et al. 46 first use motion-capture methods to track
agent positions in videos and then train RL agents to match these waypoints. Similarly, Behbahani
et al. 47 and Aytar et al. 48 task a RL agent to match waypoints; however, they construct waypoints that
are embeddings from unsupervised feature learning models. Pathak et al. 49 and Nair et al. 50 train
goal conditioned policies to take actions that advance the current state towards expert-provided goal
states expressed as high dimensional visual waypoints. Most similar to our own work, Torabi et al. 24
simultaneously train (1) an inverse dynamics model (IDM),51 which aims to uncover the underlying
action between timesteps given observations of past and future timesteps, e.g. pIDM(at|ot, ot+1), and
(2) a behavioral cloning (BC) model on trajectories of observations labeled with the IDM. Data to
train the IDM is collected by rolling out the BC model in the target environment such that both
models improve in tandem. However, at any point in training if there are sequences in the dataset that
the IDM performs poorly on, it requires that the BC model perform those or similar sequences in
order for the IDM to improve and correctly label them. Therefore, if the BC model does not explore
efficiently, it could severely slow down learning. In order to avoid this potential issue we opted for a
simpler two-stage approach: we first train an IDM on a small number of labeled trajectories collected
from human contractors (they play the game as would normally as we record their keypresses and
mouse movements). Because human contractors reach most relevant parts of the state space, we can
hold the IDM fixed throughout BC training.
Compared to most previous work in semi-supervised imitation learning, we experiment in the much
more complex and open-ended environment of Minecraft. Minecraft is a voxel-based 3D video
game that, due its popularity and wide variety of mechanics, has attracted a vast amount of RL
research.27,28,30–34,52–60 A large body of work focuses on small, custom-made Minecraft worlds
with tasks such as navigation,53,60 block placing,54,55 instruction following,58,59 combat,56 and
others.28,31,57 Work operating in the massive, randomly generated environments of Minecraft itself
has included hill climbing,52 automated curriculum learning30 and, most closely related to the RL
experiments presented in Sec. 4.4, diamond mining.27,32–34 However, to the best of our knowledge,
there is no published work that operates in the full, unmodified human action space, which includes
drag-and-drop inventory management and item crafting.
3


Collecting “Clean” Data
Training the VPT Foundation Model
via Behavioral Cloning
Training the Inverse Dynamics Model (IDM)
~270k hours
unlabeled
video
~70k hours
unlabeled
video
~2k hours
video
labeled with
actions
Filter for “clean”
video segments
Search for relevant
Minecraft videos
via keywords
Contractors
collect data 
Label videos
with IDM 
~70k hours
video
IDM-labeled
with actions
Train non-causal IDM
Train causal
VPT Foundation Model
a
d
space
w
a
d
space
w
Figure 2: Video Pretraining (VPT) Method Overview.
3
Methods
Inverse Dynamics Models (IDM)
VPT, illustrated in Figure 2, requires we first collect a small
amount of labeled contractor data with which to train an inverse dynamics model pIDM(at|o1...T ),
which seeks to minimize the negative log-likelihood of an action at timestep t given a trajectory of T
observations ot : t ∈[1...T]. In contrast to an imitation learning policy, the IDM can be non-causal,
meaning its prediction for at can be a function of both past and future events, i.e. ot′>t. Compared to
the behavioral cloning objective of modeling the distribution of human intent given past frames only,
we hypothesize that inverting environment dynamics is easier and more data efficient to learn. Indeed,
Sec. 4.1 will show that the IDM objective is much easier to learn, and furthermore Sec. 4.6 will show
that with very little labeled data (as few as 100 hours) we can train a fairly accurate IDM. This IDM
can be used to label online videos, providing the large amount of data required for the harder task of
behavioral cloning. See appendices D and B for IDM training and data collection details.
Data Filtering
We gather a large dataset of Minecraft videos by searching the web for related
keywords (Appendix A). Online videos often (1) include overlaid artifacts, such as a video feed
of the player’s face, channel logos, watermarks, etc., (2) are collected from platforms other than
a computer with different gameplay, or (3) are from different game modes, e.g. in Minecraft we
only want "survival mode" where players start from scratch and must gather or craft all their items.
We call data “clean” if it does not contain visual artifacts and is from survival mode, and call all
other data “unclean.” With enough data, a large enough model, and enough training compute, a BC
model trained on both unclean and clean videos would likely still perform well in a clean Minecraft
environment. However, for simplicity and training compute efficiency, we choose to filter out unclean
segments of video (note that a video may contain both clean and unclean segments). We do this by
training a model to filter out unclean segments using a small dataset (8800) of images sampled from
online videos labeled by contractors as clean or unclean (Appendix A.2).
VPT Foundation Model
We train a foundation model with standard behavioral cloning, i.e. mini-
mizing the negative log-likelihood of actions predicted by the IDM on clean data. For a particular
trajectory of length T we minimize
min
θ
X
t∈[1...T ]
−log πθ(at|o1, . . . , ot), where at ∼pIDM(at|o1, . . . , ot, . . . , oT )
(1)
As we will see in the following sections, this model exhibits nontrivial zero-shot behavior and can be
fine-tuned with both imitation learning and RL to perform even more complex skills.
4
Results
4.1
Performance of the Inverse Dynamics Model
The IDM architecture is comprised primarily of a temporal convolution layer, a ResNet62 image
processing stack, and residual unmasked attention layers, from which the IDM simultaneously
predicts keypresses and mouse movements (see Appendix D for IDM architecture and training
details). A key hypothesis behind our work is that IDMs can be trained with a relatively small amount
of labeled data. While more data improves both mouse movement and keypress predictions, our best
4


Figure 3: (Left) IDM keypress accuracy and mouse movement R2 (explained variance61) as a
function of dataset size. (Right) IDM vs. behavioral cloning data efficiency.
IDM trains on only 1962 hours of data (compared to the ∼70k hours of clean data we collected from
the internet) and achieves 90.6% keypress accuracy and a 0.97 R2 for mouse movements evaluated
on a held-out validation set of contractor-labeled data (Figure 3 left).
Figure 3 (right) validates our hypothesis that IDMs are far more data efficient than BC models, likely
because inverting environment mechanics is far easier than modeling the entire distribution of human
behavior. The IDM is two orders of magnitude more data efficient than a BC model trained on the
same data and improves more quickly with more data. This evidence supports the hypothesis that it is
more effective to use contractor data within the VPT pipeline by training an IDM than it is to train a
foundation model from contractor data directly (Sections 4.5 and 4.6 provide additional evidence).
4.2
VPT Foundation Model Training and Zero-Shot Performance
Figure 4: (Left) Training and validation loss on the web_clean internet dataset with IDM pseudo-
labels, and loss on the main IDM contractor dataset, which has ground-truth labels but is out-of-
distribution (see text). (Right) Amount a given item was collected per episode averaged over 2500
60-minute survival episodes as a function of training epoch, shaded with the standard error of the
mean. Basic mining refers to collection of dirt, gravel, or sand (all materials that can be gathered
without tools). Logs are obtained by repeatedly hitting trees for three seconds, a difficult feat for an
RL agent to achieve as we show in Sec. 4.4. Planks can be crafted from logs, and crafting tables
crafted from planks. Crafting requires using in-game crafting GUIs, and proficient humans take a
median of 50 seconds (970 consecutive actions) to make a crafting table.
We now explore the emergent behavior learned by a behavioral cloning policy trained on an extremely
large, but noisy, internet dataset labeled with our IDM. To collect the unlabeled internet dataset,
we searched for publicly available videos of Minecraft play with search terms such as “minecraft
survival for beginners.” These searches resulted in ∼270k hours of video, which we filtered down to
“clean” video segments yielding an unlabeled dataset of ∼70k hours, which we refer to as web_clean
(Appendix A has further details on data scraping and filtering). We then generated pseudo-labels
for web_clean with our best IDM (Section 3) and then trained the VPT foundation model with
behavioral cloning. Preliminary experiments suggested that our model could benefit from 30 epochs
of training and that a 0.5 billion parameter model was required to stay in the efficient learning
regime63 for that training duration (Appendix H), which took ∼9 days on 720 V100 GPUs.
We evaluate our models by measuring validation loss (Fig. 4, left) and rolling them out in the
Minecraft environment. Unless otherwise noted, in all environment evaluations we spawn agents in a
standard survival mode game where they play for 60 minutes, i.e. 72000 consecutive actions, and we
plot the mean and shade the standard error of the mean for various game statistics such as crafting
and collection rates (Fig. 4, right). The VPT foundation model quickly learns to chop down trees
to collect logs, a task we found near impossible for an RL agent to achieve with the native human
interface (Sec. 4.4). It also learns to craft those logs into wooden planks and then use those planks
5


to craft a crafting table, which are required to unlock most other technology in the game and take a
human proficient in Minecraft approximately 50 seconds (970 consecutive actions) to collect. While
these behaviors are fairly complex in the native human action space, the VPT foundation model crafts
these items at a rate far below that of our proficient contractors, e.g. on average our contractors craft
5.44 crafting tables in 60 minutes of play versus 0.19 for the foundation model. The model also crafts
a non-negligible amount of wooden sticks, which are required to make wooden tools; collects various
flowers and crafts dyes from them; kills zombies that appear during the night; hunts wild animals;
collects various berries and mushrooms and eats them; and finds game-generated villages from which
to collect various rare items from chests. The model also learned to navigate uneven terrain, swim,
and pillar jump, which involves the agent repeatedly jumping and quickly placing a block below itself
such that it climbs upward by making a pillar.(iv)
While training and validation loss decrease healthily over training (Fig. 4, left), loss on our contractor
dataset (which the VPT model does not train on) begins increasing after 7 epochs. Contractor data
could be out-of-distribution because our contractors may have a different distribution of play or
because there is some impactful visual domain shift compared to videos from the web. While one
could have expected this would be predictive of declining evaluation performance, we do not see
notable game statistics from the VPT foundation model rollouts (Figure 4, right) decrease over
training, and in the next section we show that BC fine-tuning performance continually improves as the
VPT foundation model trains. We provide more insight into this curious phenomenon in Appendix H.
4.3
Fine-Tuning with Behavioral Cloning
Foundation models are designed to have a broad behavior profile and be generally capable across a
wide variety of tasks. To incorporate new knowledge or allow them to specialize on a narrower task
distribution, it is common practice to fine-tune these models to smaller, more specific datasets.1 The
VPT foundation model trained on the broad web_clean dataset had nontrivial zero-shot performance;
it was able to craft a crafting table yet unable to go past this in the technology tree. As a case
study into BC fine-tuning, we attempt to improve the VPT foundation model’s ability to collect
and craft these “early game” items by fine-tuning to two narrower datasets targeted at Minecraft
behavior within the first few minutes of players starting in a fresh world. In the first dataset,
contractor_house, contractors have 10 minutes to build a basic house from scratch using primarily
wood, sand, and dirt. Collecting contractor data can be difficult and expensive, so we also construct a
dataset earlygame_keyword by searching for videos online with descriptions that match keywords
such as “new world”, “let’s play episode 1”, etc.; this is a subset of web_clean and is labeled with
the IDM. See Appendix B.4 and A.3 for full descriptions of both datasets.
Effect of Foundation Model Quality on BC Fine-Tuning
59x
213x
59x
Figure 5:
(Left) Collection and crafting rates for three policies:
the zero-shot VPT foun-
dation model, and the VPT foundation model BC fine-tuned to the earlygame_keyword or
contractor_house datasets. BC fine-tuning to either dataset improves performance, including (for
the contractor_house dataset) yielding wooden and stone tools. Proficient Minecraft players take
a median of 1.2 minutes (1390 actions) to construct wooden tools and 2.3 minutes (2790 actions)
to construct stone tools. (Right) Collection and crafting rates for VPT foundation model snapshots
throughout training after they are BC fine-tuned to the contractor_house dataset. In general,
crafting-related behaviors increase throughout foundation model training. Fig. 4 defines the other
task terms (logs, planks, crafting tables, and total crafting).
(iv)Sample videos: https://www.youtube.com/playlist?list=PLNAOIb_agjf3U3rSvG_BCWqJ869NdBhcP
6


Fine-tuning to earlygame_keyword results in a large boost compared to the zero-shot foundation
model: 2.5x more crafting tables, 6.1x more planks, 4.3x more logs, and 5.5x more crafting overall
(Fig. 5). However, when fine-tuning to this dataset we did not see any new behaviors emerge,
only a refinement of existing skills. We saw an even bigger improvement when fine-tuning to the
contractor_house dataset: 213x more crafting tables, 59x more wooden planks, 7x more logs,
and 59x more crafting over all. In addition, we saw the emergence of crafting wooden tools, which
requires placing a crafting table on the ground, opening it to reveal a new crafting interface, and then
using it to craft wooden tools. This entire sequence takes a proficient human player a median of 1.2
minutes (1390 consecutive actions) to accomplish. The model goes further and collects cobblestone,
which requires a wooden pickaxe to mine, and crafts stone tools, requiring it to again use a crafting
table; this takes a proficient human player a median of 2.3 minutes (2790 consecutive actions). We
also saw this model more frequently raiding villages that randomly spawn in the game, hunting
animals for food, in addition to many behaviors we saw performed by the foundation model.(v)
Despite the foundation model’s zero-shot rollout performance plateauing 1/3 into training (Fig. 4,
right), fine-tuning performance does continue to increase throughout foundation model training
(Fig. 5, right). Additionally, there is a stark difference in performance when training from scratch vs.
fine-tuning from the VPT foundation model (Fig. 5 right, comparing the left and rightmost points).
4.4
Fine-Tuning with Reinforcement Learning
Figure 6: Typical sequence of items for obtaining a diamond pickaxe. Below each item is the median
time and number of actions contractors required to obtain that item and the percentage of contractors
that got the item within 10 minutes. The median time to obtain a diamond pickaxe is unknown (except
that it is > 20m) because contractors obtained this item in less than 50% of 20-minute episodes.
To demonstrate the efficacy of RL fine-tuning, we chose the challenging goal of obtaining a diamond
pickaxe within 10 minutes starting from a fresh Minecraft survival world. Doing so involves acquiring
a sequence of difficult-to-obtain items that require complex skills like mining, inventory management,
crafting with and without a crafting table, tool use, operating a furnace, and mining at the lowest
depths, where many hazards like enemies and lava exist (Fig. 6). Adding to the difficulty, progress
can be easily lost by dropping items, destroying items, or dying. Obtaining a diamond pickaxe more
often than not takes a proficient human over 20 minutes (24,000 actions).
Agents are rewarded for each item obtained in the sequence, with lower rewards for items that have to
be collected in bulk and higher rewards for items near the end of the sequence. Agents are optimized
with the phasic policy gradient64 RL algorithm for ∼1.3 million episodes (roughly 1.4×1010 frames).
Episodes last for 10 minutes. See Appendix G.1 for reward function and RL training details. Due to
computational constraints, RL experiments use a ∼248 million parameter VPT model (Appendix H).
A major problem when fine-tuning with RL is catastrophic forgetting65,66 because previously learned
skills can be lost before their value is realized. For instance, while our VPT foundation model never
exhibits the entire sequence of behaviors required to smelt iron zero-shot, it did train on examples of
players smelting with furnaces. It therefore may have some latent ability to smelt iron once the many
prerequisites to do so have been performed. To combat the catastrophic forgetting of latent skills
such that they can continually improve exploration throughout RL fine-tuning, we add an auxiliary
Kullback-Leibler (KL) divergence loss between the RL model and the frozen pretrained policy.10
Training from a randomly initialized policy fails to achieve almost any reward, underscoring how
hard an exploration challenge the diamond pickaxe task is for RL in the native human action space
(Fig. 7a). The model never learns to reliably collect logs, typically the first of many steps to obtaining
a diamond pickaxe (Fig. 7b). RL fine-tuning from the VPT foundation model does substantially
better (Fig. 7a), learning everything up to mining iron ore and crafting furnaces. (Fig. 7c). However,
this agent fails at smelting an iron ingot, the next item required to get further into the tech tree, likely
(v)Sample Videos: https://www.youtube.com/playlist?list=PLNAOIb_agjf2yDSs4AqcoyPv4z_eWUiKm
7


0
5
10
15
20
25
Reward
Reward over episodes
RL from Rand. Init. model
RL from VPT Found. model
RL from Early-Game model
No KL-loss
(a)
0
20
40
60
80
100
% episodes
RL from Rand. Init. model
Logs
Planks
Sticks
Crafting T
ables
Wooden Pickaxe
Cobblestone
Stone Pickaxe
Coal
T
orch
Furnace
Iron Ore
Iron Ingot
Iron Pickaxe
Diamonds
Diamond Pickaxe
(b)
0.0
0.2
0.4
0.6
0.8
1.0
1.2
1.4
Episodes
1e6
0
20
40
60
80
100
% episodes
RL from VPT Found. model
(c)
0.0
0.2
0.4
0.6
0.8
1.0
1.2
1.4
Episodes
1e6
0
20
40
60
80
100
% episodes
2.5%
RL from Early-Game model
(d)
Figure 7: RL Fine-tuning results. (a) RL from a randomly initialized model fails to get almost
any reward, RL fine-tuning from the VPT foundation model performs substantially better with a
reward near 13, and RL fine-tuning from the early-game model performs best with a reward of 25.
When training the early-game model without a KL loss to the original policy (No KL-loss) progress
stalls after 100,000 episodes, suggesting that the skills necessary to make further progress have been
catastrophically forgotten. (b) RL from a randomly initialized model occasionally collects sticks by
breaking leaves (an easy but inefficient method of getting sticks that does not require logs or planks)
and never learns to reliably collect logs. (c) RL fine-tuning from the VPT Foundation model learns
everything in the curriculum up to iron ore and making furnaces, but fails to learn to use the furnace to
smelt iron ingots. (d) RL fine-tuning from the early-game model learns to obtain (at human-level) all
items in the sequence towards a diamond pickaxe and crafts a diamond pickaxe in 2.5% of episodes.
because the zero-shot probability that the VPT foundation model smelts an iron ingot is too low, even
when given the prerequisite materials.
Results further improve by first BC fine-tuning the VPT Foundation Model to the
earlygame_keyword dataset (the early-game model, Sec. 4.3) and then fine-tuning with RL
(Fig. 7a), which in preliminary experiments we found to perform better than first fine-tuning to
contractor_house followed by fine-tuning with RL (Appendix G.2). The three-phase training
(pretraining, BC fine-tuning, and then RL fine-tuning) succeeds in learning extremely difficult tasks:
it achieves over 80% reliability on iron pickaxes, almost 20% reliability on collecting diamonds, and
2.5% reliability on obtaining a diamond pickaxe (Fig. 7d). For comparison, human players given
the objective of obtaining a diamond pickaxe collect these items in 57%, 15%, and 12% of episodes,
respectively, meaning our model is human-level for crafting iron pickaxes and mining diamonds.
Others have managed to obtain diamonds with ∼0.1% reliability in 15 minutes32,33 but always with a
simplified action space designed to ease exploration. To the best of our knowledge, we are the first to
report non-zero success rates on crafting a diamond pickaxe. Qualitatively, the model developed
useful skills for diamond mining, such as efficient mining patterns, cave exploration, returning to
previously placed objects like crafting tables, and advanced techniques like using wooden pickaxes
as fuel when moving on to iron tools.(vi)
Finally, we validated the importance of the KL loss to the pretrained model during RL fine-tuning.
The treatment without a KL loss obtains only items early in the sequence (logs, planks, sticks, and
crafting tables) limiting its reward (Fig. 7a). This failure to progress further into the sequence is
likely because, while the initial skills of chopping logs and crafting planks are being learned with RL,
subsequent skills like crafting a wooden pickaxe are lost due to catastrophic forgetting.
4.5
Data Scaling Properties of the Foundation Model
In this section we validate a core hypothesis behind this work: that it is far more effective to use
labeled contractor data to train an IDM within the VPT method than it is to directly train a BC
foundation model from that same small contractor dataset. If we could cheaply collect a labeled
contractor dataset of a similar order of magnitude as web_clean, then this would not be important;
however, collecting that scale of data would have cost millions of dollars. Figure 8 compares
foundation models trained on increasing orders of magnitude of data from 1 hour up to the full ∼70k
web_clean dataset. Foundation models trained up to and including 1k hours are trained on the IDM
(vi)Videos found at https://www.youtube.com/playlist?list=PLNAOIb_agjf3e_UKweM5pQUSfTw8r-Wfc
8


Trained on
Contractor Data
Trained on IDM
Labeled Web Data
Figure 8: (Left) Zero-shot rollout performance of foundation models trained on varying amounts
of data. Models to the left of the dashed black line (points ≤1k hours) were trained on contractor
data (ground-truth labels), and models to the right were trained on IDM pseudo-labeled subsets
of web_clean. Due to compute limitations, this analysis was performed with smaller (71 million
parameter) models except for the final point, which is the 0.5 billion parameter VPT foundation
model. (Right) The corresponding performance of each model after BC fine-tuning each model to
the contractor_house dataset.
contractor data, and those trained on 5k hours and above are trained on subsets of web_clean, which
does not contain any IDM contractor data. Scaling training data increases log collection, mining, and
crafting capabilities. The zero-shot model only begins to start crafting crafting tables at over 5000
hours of training data. When fine-tuning each foundation model to contractor_house, we see that
crafting rates for crafting tables and wooden tools increase by orders of magnitude when using the
entire ∼70k hour web_clean dataset. We furthermore only see the emergence of crafting stone tools
at the largest data scale.
4.6
Effect of Inverse Dynamics Model Quality on Behavioral Cloning
Figure 9: Zero-shot performance of BC models
trained from scratch on the earlygame_keyword
dataset labeled with IDMs that were trained on
increasing amounts of contractor data.
This section investigates how downstream
BC performance is affected by IDM qual-
ity.
We train IDMs on increasingly larger
datasets and use each to independently label
the earlygame_keyword dataset (this smaller
dataset was chosen due to a limited compute bud-
get). We then train a BC model from scratch on
each dataset and report game statistics for each
model as a function of IDM contractor dataset
size (Fig. 9).
IDMs trained on at least 10 hours of data are
required for any crafting, and the crafting rate
increases quickly up until 100 hours of data, after which there are few to no gains and differences are
likely due to noise. Similarly, crafting tables are only crafted after 50 or more hours of IDM data, and
again gains plateau after 100 hours. While in all previous experiments we use our best IDM trained
on 1962 hours of data, these results suggest we could reduce that number to as low as 100 hours.
5
Discussion and Conclusion
The results presented in this paper help pave the path to utilizing the wealth of unlabeled data on the
web for sequential decision domains. Compared to generative video modeling or contrastive methods
that would only yield representational priors, VPT offers the exciting possibility of directly learning
to act during pretraining and using these learned behavioral priors as extremely effective exploration
priors for RL. VPT could even be a better general representation learning method even when the
downstream task is not learning to act in that domain—for example, fine-tuning to explain what is
happening in a video—because arguably the most important information in any given scene would be
present in features trained to correctly predict the distribution over future human actions. We leave
this intriguing direction to future work.
Future work could improve results with more data (we estimate we could collect >1M hours) and
larger, better-tuned models. Furthermore, all the models in this work condition on past observations
only; we cannot ask the model to perform specific tasks. Appendix I presents preliminary experiments
on conditioning our models on closed captions (text transcripts of speech in videos), showing they
9


become weakly steerable; we believe this a rich direction for future research. Also, loss was not
consistently correlated with downstream evaluation metrics (Sec. 4.2), which often made progress
slow and hard-won. Another fruitful future direction would be to investigate the correlation between
various training metrics and downstream evaluations. Finally, while we do not anticipate any direct
negative societal impacts from the models trained in this work, as VPT improves and expands to other
domains it will be important to assess and mitigate harms that emerge with other forms of pretraining
on internet datasets, such as emulating inappropriate behavior.67
In conclusion, VPT extends the paradigm of training large and general purpose behavioral priors from
freely available internet-scale data to sequential decision domains. Our models exhibited impressive
zero-shot behavior and, when fine-tuned with RL, achieved an unprecedented result of crafting a
diamond pickaxe in Minecraft (all the more difficult given the human interface). We further showed
that contractor data is far better used within the VPT pipeline than to train a foundation model directly
and that only a small amount of contractor data (about $2000 USD) was required to unlock massive
amounts of unlabeled online data for use in BC. Finally, learning with the human keyboard and mouse
interface is highly general and allows losslessly modeling the entire distribution of human behavior.
While we only experiment in Minecraft, we believe that VPT provides a general recipe for training
behavioral priors in hard, yet generic, action spaces in any domain that has a large amount of freely
available unlabeled data, such as computer usage.
References
[1] Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal,
Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language models are
few-shot learners. Advances in neural information processing systems, 33:1877–1901, 2020.
[2] Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. Bert: Pre-training of
deep bidirectional transformers for language understanding. arXiv preprint arXiv:1810.04805,
2018.
[3] Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike
Lewis, Luke Zettlemoyer, and Veselin Stoyanov. Roberta: A robustly optimized bert pretraining
approach. arXiv preprint arXiv:1907.11692, 2019.
[4] Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena,
Yanqi Zhou, Wei Li, and Peter J Liu. Exploring the limits of transfer learning with a unified
text-to-text transformer. arXiv preprint arXiv:1910.10683, 2019.
[5] Dhruv Mahajan, Ross Girshick, Vignesh Ramanathan, Kaiming He, Manohar Paluri, Yixuan Li,
Ashwin Bharambe, and Laurens Van Der Maaten. Exploring the limits of weakly supervised
pretraining. In Proceedings of the European conference on computer vision (ECCV), pages
181–196, 2018.
[6] Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal,
Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual
models from natural language supervision. In International Conference on Machine Learning,
pages 8748–8763. PMLR, 2021.
[7] Rishi Bommasani, Drew A Hudson, Ehsan Adeli, Russ Altman, Simran Arora, Sydney von
Arx, Michael S Bernstein, Jeannette Bohg, Antoine Bosselut, Emma Brunskill, et al. On the
opportunities and risks of foundation models. arXiv preprint arXiv:2108.07258, 2021.
[8] Xiaohua Zhai, Alexander Kolesnikov, Neil Houlsby, and Lucas Beyer. Scaling vision transform-
ers. CoRR, abs/2106.04560, 2021. URL https://arxiv.org/abs/2106.04560.
[9] David Silver, Aja Huang, Chris J Maddison, Arthur Guez, Laurent Sifre, George Van Den Driess-
che, Julian Schrittwieser, Ioannis Antonoglou, Veda Panneershelvam, Marc Lanctot, et al. Mas-
tering the game of go with deep neural networks and tree search. Nature, 529(7587):484–489,
2016.
[10] Oriol Vinyals, Igor Babuschkin, Wojciech M Czarnecki, Michaël Mathieu, Andrew Dudzik, Jun-
young Chung, David H Choi, Richard Powell, Timo Ewalds, Petko Georgiev, et al. Grandmaster
level in starcraft ii using multi-agent reinforcement learning. Nature, 575(7782):350–354, 2019.
10


[11] Richard S Sutton and Andrew G Barto. Reinforcement learning: An introduction. MIT press,
2018.
[12] Christopher Berner, Greg Brockman, Brooke Chan, Vicki Cheung, Przemysław D˛
ebiak, Christy
Dennison, David Farhi, Quirin Fischer, Shariq Hashme, Chris Hesse, et al. Dota 2 with large
scale deep reinforcement learning. arXiv preprint arXiv:1912.06680, 2019.
[13] Bowen Baker, Ingmar Kanitscheider, Todor Markov, Yi Wu, Glenn Powell, Bob McGrew,
and Igor Mordatch.
Emergent tool use from multi-agent autocurricula.
arXiv preprint
arXiv:1909.07528, 2019.
[14] Max Jaderberg, Wojciech M Czarnecki, Iain Dunning, Luke Marris, Guy Lever, Antonio Garcia
Castaneda, Charles Beattie, Neil C Rabinowitz, Ari S Morcos, Avraham Ruderman, et al.
Human-level performance in 3d multiplayer games with population-based reinforcement learn-
ing. Science, 364(6443):859–865, 2019.
[15] Adrià Puigdomènech Badia, Bilal Piot, Steven Kapturowski, Pablo Sprechmann, Alex Vitvit-
skyi, Zhaohan Daniel Guo, and Charles Blundell. Agent57: Outperforming the atari human
benchmark. In International Conference on Machine Learning, pages 507–517. PMLR, 2020.
[16] Marc Bellemare, Sriram Srinivasan, Georg Ostrovski, Tom Schaul, David Saxton, and Remi
Munos.
Unifying count-based exploration and intrinsic motivation.
Advances in neural
information processing systems, 29, 2016.
[17] Yuri Burda, Harrison Edwards, Amos Storkey, and Oleg Klimov. Exploration by random
network distillation. arXiv preprint arXiv:1810.12894, 2018.
[18] Adrien Ecoffet, Joost Huizinga, Joel Lehman, Kenneth O Stanley, and Jeff Clune. First return,
then explore. Nature, 590(7847):580–586, 2021.
[19] Peter C Humphreys, David Raposo, Toby Pohlen, Gregory Thornton, Rachita Chhaparia, Alistair
Muldal, Josh Abramson, Petko Georgiev, Alex Goldin, Adam Santoro, et al. A data-driven
approach for learning to control computers. arXiv preprint arXiv:2202.08137, 2022.
[20] Tianlin Shi, Andrej Karpathy, Linxi Fan, Jonathan Hernandez, and Percy Liang. World of
bits: An open-domain platform for web-based agents. In Doina Precup and Yee Whye Teh,
editors, Proceedings of the 34th International Conference on Machine Learning, volume 70 of
Proceedings of Machine Learning Research, pages 3135–3144. PMLR, 06–11 Aug 2017. URL
https://proceedings.mlr.press/v70/shi17a.html.
[21] Andrew Y Ng, Stuart J Russell, et al. Algorithms for inverse reinforcement learning. In Icml,
volume 1, page 2, 2000.
[22] Faraz Torabi, Garrett Warnell, and Peter Stone. Recent advances in imitation learning from
observation. arXiv preprint arXiv:1905.13566, 2019.
[23] Jonathan Ho and Stefano Ermon. Generative adversarial imitation learning. Advances in neural
information processing systems, 29, 2016.
[24] Faraz Torabi, Garrett Warnell, and Peter Stone. Behavioral cloning from observation. arXiv
preprint arXiv:1805.01954, 2018.
[25] YuXuan Liu, Abhishek Gupta, Pieter Abbeel, and Sergey Levine. Imitation from observation:
Learning to imitate behaviors from raw video via context translation. In 2018 IEEE International
Conference on Robotics and Automation (ICRA), pages 1118–1125. IEEE, 2018.
[26] Twinfinite
Staff.
Most
played
games
in
2021,
ranked
by
peak
concur-
rent
players.
Twinfinite.
URL
https://twinfinite.net/2021/12/
most-played-games-in-2020-ranked-by-peak-concurrent-players/.
[27] William H Guss, Brandon Houghton, Nicholay Topin, Phillip Wang, Cayden Codel, Manuela
Veloso, and Ruslan Salakhutdinov. Minerl: A large-scale dataset of minecraft demonstrations.
arXiv preprint arXiv:1907.13440, 2019.
11


[28] Chen Tessler, Shahar Givony, Tom Zahavy, Daniel Mankowitz, and Shie Mannor. A deep
hierarchical approach to lifelong learning in minecraft. In Proceedings of the AAAI Conference
on Artificial Intelligence, volume 31, 2017.
[29] Christian Scheller, Yanick Schraner, and Manfred Vogel. Sample efficient reinforcement
learning through learning from demonstrations in minecraft. In NeurIPS 2019 Competition and
Demonstration Track, pages 67–76. PMLR, 2020.
[30] Ingmar Kanitscheider, Joost Huizinga, David Farhi, William Hebgen Guss, Brandon Houghton,
Raul Sampedro, Peter Zhokhov, Bowen Baker, Adrien Ecoffet, Jie Tang, et al. Multi-task
curriculum learning in a complex, visual, hard-exploration domain: Minecraft. arXiv preprint
arXiv:2106.14876, 2021.
[31] Junhyuk Oh, Valliappa Chockalingam, Honglak Lee, et al. Control of memory, active perception,
and action in minecraft. In International Conference on Machine Learning, pages 2790–2799.
PMLR, 2016.
[32] Vihang P Patil, Markus Hofmarcher, Marius-Constantin Dinu, Matthias Dorfer, Patrick M Blies,
Johannes Brandstetter, Jose A Arjona-Medina, and Sepp Hochreiter. Align-rudder: Learning
from few demonstrations by reward redistribution. arXiv preprint arXiv:2009.14108, 2020.
[33] Alexey Skrynnik, Aleksey Staroverov, Ermek Aitygulov, Kirill Aksenov, Vasilii Davydov, and
Aleksandr I Panov. Forgetful experience replay in hierarchical reinforcement learning from
demonstrations. arXiv preprint arXiv:2006.09939, 2020.
[34] Zichuan Lin, Junyou Li, Jianing Shi, Deheng Ye, Qiang Fu, and Wei Yang.
Juewu-mc:
Playing minecraft with sample-efficient hierarchical reinforcement learning. arXiv preprint
arXiv:2112.04907, 2021.
[35] Dean A Pomerleau. Alvinn: An autonomous land vehicle in a neural network. Advances in
neural information processing systems, 1, 1988.
[36] Stefan Schaal. Is imitation learning the route to humanoid robots? Trends in cognitive sciences,
3(6):233–242, 1999.
[37] Brenna D Argall, Sonia Chernova, Manuela Veloso, and Brett Browning. A survey of robot
learning from demonstration. Robotics and autonomous systems, 57(5):469–483, 2009.
[38] Ahmed Hussein, Mohamed Medhat Gaber, Eyad Elyan, and Chrisina Jayne. Imitation learning:
A survey of learning methods. ACM Computing Surveys (CSUR), 50(2):1–35, 2017.
[39] Claude Sammut, Scott Hurst, Dana Kedzier, and Donald Michie. Learning to fly. In Machine
Learning Proceedings 1992, pages 385–393. Elsevier, 1992.
[40] Alessandro Giusti, Jérôme Guzzi, Dan C Cire¸
san, Fang-Lin He, Juan P Rodríguez, Flavio
Fontana, Matthias Faessler, Christian Forster, Jürgen Schmidhuber, Gianni Di Caro, et al. A
machine learning approach to visual perception of forest trails for mobile robots. IEEE Robotics
and Automation Letters, 1(2):661–667, 2015.
[41] Mariusz Bojarski, Davide Del Testa, Daniel Dworakowski, Bernhard Firner, Beat Flepp, Prasoon
Goyal, Lawrence D Jackel, Mathew Monfort, Urs Muller, Jiakai Zhang, et al. End to end learning
for self-driving cars. arXiv preprint arXiv:1604.07316, 2016.
[42] Felipe Codevilla, Matthias Müller, Antonio López, Vladlen Koltun, and Alexey Dosovitskiy.
End-to-end driving via conditional imitation learning. In 2018 IEEE international conference
on robotics and automation (ICRA), pages 4693–4700. IEEE, 2018.
[43] Rémi Coulom. Computing “elo ratings” of move patterns in the game of go. ICGA journal, 30
(4):198–208, 2007.
[44] Todd Hester, Matej Vecerik, Olivier Pietquin, Marc Lanctot, Tom Schaul, Bilal Piot, Dan Horgan,
John Quan, Andrew Sendonaris, Ian Osband, et al. Deep q-learning from demonstrations. In
Proceedings of the AAAI Conference on Artificial Intelligence, volume 32, 2018.
12


[45] Ashley Edwards, Himanshu Sahni, Yannick Schroecker, and Charles Isbell. Imitating latent
policies from observation. In International conference on machine learning, pages 1755–1763.
PMLR, 2019.
[46] Xue Bin Peng, Angjoo Kanazawa, Jitendra Malik, Pieter Abbeel, and Sergey Levine. Sfv:
Reinforcement learning of physical skills from videos. ACM Transactions On Graphics (TOG),
37(6):1–14, 2018.
[47] Feryal Behbahani, Kyriacos Shiarlis, Xi Chen, Vitaly Kurin, Sudhanshu Kasewa, Ciprian Stirbu,
Joao Gomes, Supratik Paul, Frans A Oliehoek, Joao Messias, et al. Learning from demonstration
in the wild. In 2019 International Conference on Robotics and Automation (ICRA), pages
775–781. IEEE, 2019.
[48] Yusuf Aytar, Tobias Pfaff, David Budden, Thomas Paine, Ziyu Wang, and Nando De Fre-
itas. Playing hard exploration games by watching youtube. Advances in neural information
processing systems, 31, 2018.
[49] Deepak Pathak, Parsa Mahmoudieh, Guanghao Luo, Pulkit Agrawal, Dian Chen, Yide Shentu,
Evan Shelhamer, Jitendra Malik, Alexei A. Efros, and Trevor Darrell. Zero-shot visual imitation.
In ICLR, 2018.
[50] Ashvin Nair, Dian Chen, Pulkit Agrawal, Phillip Isola, Pieter Abbeel, Jitendra Malik, and Sergey
Levine. Combining self-supervised learning and imitation for vision-based rope manipulation.
pages 2146–2153, 05 2017. doi: 10.1109/ICRA.2017.7989247.
[51] Duy Nguyen-Tuong, Jan Peters, Matthias Seeger, and Bernhard Schölkopf. Learning inverse
dynamics: a comparison. In European symposium on artificial neural networks, number CONF,
2008.
[52] David Abel, Alekh Agarwal, Fernando Diaz, Akshay Krishnamurthy, and Robert E Schapire.
Exploratory gradient boosting for reinforcement learning in complex domains. arXiv preprint
arXiv:1603.04119, 2016.
[53] Dilip Arumugam, Jun Ki Lee, Sophie Saskin, and Michael L Littman. Deep reinforcement
learning from policy-dependent human feedback. arXiv preprint arXiv:1902.04257, 2019.
[54] Alexander Trott, Stephan Zheng, Caiming Xiong, and Richard Socher. Keeping your dis-
tance: Solving sparse reward tasks using self-balancing shaped rewards. Advances in Neural
Information Processing Systems, 32, 2019.
[55] Stephan Alaniz. Deep reinforcement learning with model learning and monte carlo tree search
in minecraft. arXiv preprint arXiv:1803.08456, 2018.
[56] Hiroto Udagawa, Tarun Narasimhan, and Shim-Young Lee. Fighting zombies in minecraft with
deep reinforcement learning. Technical report, Technical report, Technical report, Stanford
University, 2016.
[57] Tianmin Shu, Caiming Xiong, and Richard Socher. Hierarchical and interpretable skill acquisi-
tion in multi-task reinforcement learning. arXiv preprint arXiv:1712.07294, 2017.
[58] Junhyuk Oh, Satinder Singh, Honglak Lee, and Pushmeet Kohli. Zero-shot task generalization
with multi-task deep reinforcement learning. In International Conference on Machine Learning,
pages 2661–2670. PMLR, 2017.
[59] Zhengxiang Shi, Yue Feng, and Aldo Lipani. Learning to execute or ask clarification questions.
arXiv preprint arXiv:2204.08373, 2022.
[60] Tambet Matiisen, Avital Oliver, Taco Cohen, and John Schulman. Teacher–student curriculum
learning. IEEE transactions on neural networks and learning systems, 31(9):3732–3740, 2019.
[61] Robert George Douglas Steel, James Hiram Torrie, et al. Principles and procedures of statistics.
Principles and procedures of statistics., 1960.
13


[62] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image
recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition,
pages 770–778, 2016.
[63] Jared Kaplan, Sam McCandlish, Tom Henighan, Tom B Brown, Benjamin Chess, Rewon Child,
Scott Gray, Alec Radford, Jeffrey Wu, and Dario Amodei. Scaling laws for neural language
models. arXiv preprint arXiv:2001.08361, 2020.
[64] Karl W Cobbe, Jacob Hilton, Oleg Klimov, and John Schulman. Phasic policy gradient. In
Marina Meila and Tong Zhang, editors, Proceedings of the 38th International Conference on
Machine Learning, volume 139 of Proceedings of Machine Learning Research, pages 2020–
2027. PMLR, 18–24 Jul 2021. URL https://proceedings.mlr.press/v139/cobbe21a.
html.
[65] Dhireesha Kudithipudi, Mario Aguilar-Simon, Jonathan Babb, Maxim Bazhenov, Douglas
Blackiston, Josh Bongard, Andrew P Brna, Suraj Chakravarthi Raja, Nick Cheney, Jeff Clune,
et al. Biological underpinnings for lifelong learning machines. Nature Machine Intelligence, 4
(3):196–210, 2022.
[66] James Kirkpatrick, Razvan Pascanu, Neil Rabinowitz, Joel Veness, Guillaume Desjardins,
Andrei A Rusu, Kieran Milan, John Quan, Tiago Ramalho, Agnieszka Grabska-Barwinska, et al.
Overcoming catastrophic forgetting in neural networks. Proceedings of the national academy of
sciences, 114(13):3521–3526, 2017.
[67] Emily M Bender, Timnit Gebru, Angelina McMillan-Major, and Shmargaret Shmitchell. On
the dangers of stochastic parrots: Can language models be too big???. In Proceedings of the
2021 ACM Conference on Fairness, Accountability, and Transparency, pages 610–623, 2021.
[68] Fabian Pedregosa, Gaël Varoquaux, Alexandre Gramfort, Vincent Michel, Bertrand Thirion,
Olivier Grisel, Mathieu Blondel, Peter Prettenhofer, Ron Weiss, Vincent Dubourg, et al. Scikit-
learn: Machine learning in python. the Journal of machine Learning research, 12:2825–2830,
2011.
[69] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez,
Łukasz Kaiser, and Illia Polosukhin. Attention is all you need. Advances in neural information
processing systems, 30, 2017.
[70] Jimmy Lei Ba, Jamie Ryan Kiros, and Geoffrey E Hinton. Layer normalization. arXiv preprint
arXiv:1607.06450, 2016.
[71] Yann A LeCun, Léon Bottou, Genevieve B Orr, and Klaus-Robert Müller. Efficient backprop.
In Neural networks: Tricks of the trade, pages 9–48. Springer, 2012.
[72] Diederik P Kingma and Jimmy Ba. Adam: A method for stochastic optimization. arXiv preprint
arXiv:1412.6980, 2014.
[73] Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan,
Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, Alban Desmaison, Andreas
Kopf, Edward Yang, Zachary DeVito, Martin Raison, Alykhan Tejani, Sasank Chilamkurthy,
Benoit Steiner, Lu Fang, Junjie Bai, and Soumith Chintala. Pytorch: An imperative style, high-
performance deep learning library. In H. Wallach, H. Larochelle, A. Beygelzimer, F. d'Alché-
Buc, E. Fox, and R. Garnett, editors, Advances in Neural Information Processing Systems 32,
pages 8024–8035. Curran Associates, Inc., 2019. URL http://papers.neurips.cc/paper/
9015-pytorch-an-imperative-style-high-performance-deep-learning-library.
pdf.
[74] Yann N Dauphin, Razvan Pascanu, Caglar Gulcehre, Kyunghyun Cho, Surya Ganguli, and
Yoshua Bengio. Identifying and attacking the saddle point problem in high-dimensional non-
convex optimization. Advances in neural information processing systems, 27, 2014.
[75] Jason Yosinski, Jeff Clune, Yoshua Bengio, and Hod Lipson. How transferable are features in
deep neural networks? Advances in neural information processing systems, 27, 2014.
14


[76] Zihang Dai, Zhilin Yang, Yiming Yang, Jaime Carbonell, Quoc V Le, and Ruslan Salakhutdinov.
Transformer-xl: Attentive language models beyond a fixed-length context. arXiv preprint
arXiv:1901.02860, 2019.
[77] John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal
policy optimization algorithms. arXiv preprint arXiv:1707.06347, 2017.
[78] John Schulman, Philipp Moritz, Sergey Levine, Michael Jordan, and Pieter Abbeel. High-
dimensional continuous control using generalized advantage estimation.
arXiv preprint
arXiv:1506.02438, 2015.
[79] Ronald J Williams. Simple statistical gradient-following algorithms for connectionist reinforce-
ment learning. Machine learning, 8(3):229–256, 1992.
[80] Volodymyr Mnih, Adria Puigdomenech Badia, Mehdi Mirza, Alex Graves, Timothy Lillicrap,
Tim Harley, David Silver, and Koray Kavukcuoglu. Asynchronous methods for deep reinforce-
ment learning. In International conference on machine learning, pages 1928–1937. PMLR,
2016.
[81] Marcin Andrychowicz, Filip Wolski, Alex Ray, Jonas Schneider, Rachel Fong, Peter Welinder,
Bob McGrew, Josh Tobin, OpenAI Pieter Abbeel, and Wojciech Zaremba. Hindsight experience
replay. Advances in neural information processing systems, 30, 2017.
[82] Tom Schaul, Daniel Horgan, Karol Gregor, and David Silver. Universal value function ap-
proximators. In International conference on machine learning, pages 1312–1320. PMLR,
2015.
[83] Open Ended Learning Team, Adam Stooke, Anuj Mahajan, Catarina Barros, Charlie Deck,
Jakob Bauer, Jakub Sygnowski, Maja Trebacz, Max Jaderberg, Michael Mathieu, et al. Open-
ended learning leads to generally capable agents. arXiv preprint arXiv:2107.12808, 2021.
[84] Jelena Luketina, Nantas Nardelli, Gregory Farquhar, Jakob Foerster, Jacob Andreas, Edward
Grefenstette, Shimon Whiteson, and Tim Rocktäschel.
A survey of reinforcement learn-
ing informed by natural language. In Proceedings of the Twenty-Eighth International Joint
Conference on Artificial Intelligence, IJCAI-19, pages 6309–6317. International Joint Confer-
ences on Artificial Intelligence Organization, 7 2019. doi: 10.24963/ijcai.2019/880. URL
https://doi.org/10.24963/ijcai.2019/880.
[85] DeepMind Interactive Agents Team, Josh Abramson, Arun Ahuja, Arthur Brussee, Federico
Carnevale, Mary Cassin, Felix Fischer, Petko Georgiev, Alex Goldin, Tim Harley, et al. Creating
multimodal interactive agents with imitation and self-supervised learning. arXiv preprint
arXiv:2112.03763, 2021.
[86] Aditya Ramesh, Prafulla Dhariwal, Alex Nichol, Casey Chu, and Mark Chen. Hierarchical
text-conditional image generation with clip latents. arXiv preprint arXiv:2204.06125, 2022.
[87] Dong Yu and Li Deng. Automatic speech recognition, volume 1. Springer, 2016.
[88] Daulet Nurmanbetov. rpunct, May 25 2021. URL https://github.com/Felflare/rpunct.
accessed 2022-04-22.
[89] Arvind Neelakantan, Tao Xu, Raul Puri, Alec Radford, Jesse Michael Han, Jerry Tworek,
Qiming Yuan, Nikolas Tezak, Jong Wook Kim, Chris Hallacy, et al. Text and code embeddings
by contrastive pre-training. arXiv preprint arXiv:2201.10005, 2022.
Acknowledgements
We thank the following people for helpful discussions and support: Bob McGrew, Ken Stanley,
Joel Lehman, Ilya Sutskever, Wojciech Zaremba, Ingmar Kanitscheider, David Farhi, Glenn Powell,
Jonathan Gordon, and the OpenAI supercomputing team, especially Christian Gibson, Ben Chess,
and Christopher Berner.
15


Supplementary Information
A
Collecting Internet Data
A.1
Initial Unclean Dataset Curation
Our goal was to curate a video dataset of Minecraft gameplay from the survival game mode. Addition-
ally, we prefer the data come from game modes as close as possible to our evaluation environment,
meaning preferably coming from Minecraft version 1.16, being on a computer (which uses a mouse
and keyboard vs. video game controllers with keypads and other buttons), being single- (vs. multi-)
player, and having the default look of the game (vs. modifications that alter that style, such as to
make it look realistic). To try to accomplish these goals, we collect a dataset by performing keyword
searches of publicly available videos on the internet. A list of search queries we used are given in
Table 1.
minecraft survival longplay
minecraft gameplay no webcam
minecraft gameplay survival mode
minecraft survival tutorial
minecraft survival guide
minecraft survival let’s play
minecraft survival for beginners
minecraft beginners guide
ultimate minecraft starter guide
minecraft survival guide 1.16
minecraft how to start a new survival world
minecraft survival fresh start
minecraft survival let’s play episode 1
let’s play minecraft episode 1
minecraft survival 101
minecraft survival learning to play
how to play minecraft survival
how to play minecraft
minecraft survival basic
minecraft survival for noobs
minecraft survival for dummies
how to play minecraft for beginners
minecraft survival tutorial series
minecraft survival new world
minecraft survival a new beginning
minecraft survival episodio 1
minecraft survival epizod 1
minecraft survival 1. bölüm
i made a new minecraft survival world
Table 1: Search terms used for generating the initial web dataset.
For videos that have metadata available, we perform an additional step of metadata-based filtering
to eliminate videos that do not fit our target distribution. In this step, we look for a list of blacklist
keywords in the video title and description and reject videos that contain these terms. The blacklist
keywords we use are: {ps3, ps4, ps5, xbox 360, playstation, timelapse, multiplayer, minecraft pe,
pocket edition, skyblock, realistic minecraft, how to install, how to download, realmcraft, animation}.
This process yielded us ∼270k hours of unlabeled data, which we filter down to only a “clean” subset
as described in the next section.
A.2
Training a Model to Filter out Unclean Video Segments
We restrict the scope of this work to the Minecraft Survival game mode and therefore limit our
training dataset to clips that are obtained from this mode that are relatively free from visual artifacts.
16


To do so, we asked contractors to label a set of random video frames (images) from Minecraft videos
(N=8800). These images were from a random subset of the videos we collected toward the beginning
of the project (Section A.1).
A.2.1
Label Collection
We asked 5 workers on Amazon Mechanical Turk (mTurk) that we selected with a sample qualification
task to label random screen capture images to be used in training the classifier. A sample worker
interface that the workers saw on mTurk is given in Figure 10.
We asked workers to label videos as being in one of the following three categories (see Figure 11 for
visual examples of each class):
1. Minecraft Survival Mode - No Artifacts: Video frames (images) that correspond
to the Minecraft Survival game mode that do not contain any non-game visual artifacts (e.g.
subscribe buttons, channel logos, advertisements, picture-in-picture of the narrator, etc.).
2. Minecraft Survival Mode - with Artifacts:
Video frames (images) of the
Minecraft Survival game mode that include such visual artifacts.
3. None of the Above: Video frames (images) that are not from the Minecraft survival
game mode, including those from other Minecraft game modes such as creative mode or
even other games/topics entirely.
The full set of instructions workers received are as follows (note that we also included multiple image
examples from each category in the worker instructions, similar to the sample subset provided in
Figure 11):
Please help us identify screenshots that belong only to the survival mode in Minecraft. Everything
else (Minecraft creative mode, other games, music videos, etc.) should be marked as None of the
above. Survival mode is identified by the info at the bottom of the screen:
• a health bar (row of hearts)
• a hunger bar (row of chicken drumsticks)
• a bar showing items held
Survival Mode
Valid survival mode videos have health/hunger bars and an item hotbar at the bottom of the screen.
Creative Mode
Creative mode only has an item hotbar and should be classified as None of the Above.
Label Descriptions
• Minecraft Survival Mode - No Artifacts: These images will be clean screenshots
from the Minecraft survival mode gameplay without any noticeable artifacts.
• Minecraft Survival Mode - with Artifacts: These images will be valid survival
mode screenshots, but with some added artifacts. Typical artifacts may include image
overlays (a logo/brand), text annotations, a picture-in-picture of the player, etc.
• None of the Above: Use this category when the image is not a valid Minecraft survival
screenshot. It may be a non-Minecraft frame or from a different game mode. In non-survival
game modes such as the creative mode, the health/hunger bars will be missing from the
image, the item hotbar may or may not be still present.
In total, we spent $319.96 on human labeling experiments on mTurk, of which 159.98wasdirectlypaidtoworkers.TheremainingamountwasspenttowardsAmazonplatformfees.Theworkersreceived0.01 per labeled image, at an hourly compensation of $7.20 (based on an estimated labeling
time of 5 seconds/image – in our internal sample run of the same task, we found the average labeling
time to be < 3 seconds).
Since we perform rigorous keyword and metadata based filtering of videos (as described in A.1) from
which we served sample images to be labeled, serving offensive content to workers was extremely
17


low risk and no such images were detected during our manual checks. We only collected labels
during our experiment, and the workers were fully anonymized via the mTurk platform, therefore no
personally identifiable information (PII) was collected.
Figure 10: Amazon Mechanical Turk worker interface showing an example labeling task
Figure 11: (Left) Sample image for Class 1: Minecraft Survival Mode - No Artifacts.
(Middle) Sample image for Class 2: Minecraft Survival Mode - with Artifacts – Image
contains annotations and picture-in-picture of the narrator. (Right) Sample image for Class 3: None
of the Above – Image is missing the hotbar as well as health and armor bars, indicating that it was
not captured during survival mode gameplay
A.2.2
SVM Training
With the image labels collected as described in the previous section, we trained a classifier to extract
video segments that consist of frames from the Minecraft Survival Mode - No Artifacts
category. Given a set of labeled images, we obtain embeddings for each image using the RN50x64
ResNet CLIP Model.6 This is a ResNet-based CLIP model that is scaled up to have approximately
64x the compute of a ResNet-50. We then train a Support Vector Machine (SVM) using the RBF
kernel to obtain a frame classifier. We use the Scikit-learn68 SVM implementation with the parameter
configuration given in Table 2.
Finally, we apply the classifier to frames of raw video sequences at a rate of 3 frames/second. We filter
for videos that consist of at least 80% "clean" frames at this stage (Classes Minecraft Survival
Mode - with Artifacts and None of the Above are both considered not clean). From this set,
we apply a median filter (with a kernel size of 7) to the labels and segment videos by splitting the
"clean" segments that are at least 5s in duration. The result of this is our final web_clean dataset.
A.3
early_game Dataset
The early_game dataset is a ∼3000 hour subset of web_clean targeted at “early game” Minecraft
behavior, i.e. instances where players start in a fresh world with no items. We obtain the metadata
text that accompanies the videos in web_clean and determine whether any of the following regular
expressions match:
18


CLIP Model Specification
RN50x64 (see text)
CLIP Input Image Resolution
448x448x3
CLIP Embedding Feature Length
1024
SVM Parameters
Kernel
rbf
C
20
Gamma
scale
Sample Size
Class 1
2200
Class 2
2200
Class 3
4400
Table 2: Feature Extraction Details and SVM Configuration. The parameters are for the SVM
implementation in Scikit-learn68.
• (ep|episode|eps|day|session|sesh|chapter|chap
.|series|part|parte|pt|round|day|tâ
.p|bölüm|episodio|epizod|pizod)(
)*(\.1|#1|1|\.01|#01|01|one[ˆ0-9]|$)
• start
• beginning
• (new|fresh|clean).*(world|game|play)
• from scratch
From this set of videos, we take only the first 5 minutes of each video.
B
Contractor Data
B.1
Recording Contractor Play
Our contractors use a custom Minecraft recorder that we built that records their actions and game video
feeds as they play. The recorder is implemented using the MCP-Reborn (github.com/Hexeption/MCP-
Reborn) modding package. To ensure that the recorder environment is as close as possible to
the Minecraft environment used for RL rollouts and evaluations (Appendix C), we use the same
underlying game engine for both. The recorder is a Java app that runs in a window mode, with
constant resolution of 1280x760. Brightness is set to 0 (the "gloomy" setting in Minecraft), which
is the default setting. Other graphics settings (field of view, GUI scale) are fixed to the values used
in the Minecraft environment (C.1); we explicitly prevented users from changing graphics settings.
Unlike the environment, the recorder allows all keyboard key presses and continuous (as opposed to
binned) mouse actions. On every game step (or “tick”) the frame buffer used to display the game
window is downsized to 640x360 and written into a video file. In-game actions are recorded in a
separate JSONL file (a text file where each line is a JSON-formatted string). All recordings are
chunked into 5 minute clips: after each 5 minute segment of contractor game play the recorder
automatically uploads the video file, the JSONL file with actions, as well as a Minecraft state file.
To ensure that contractors cannot corrupt each other’s data, we provided every contractor with an
individual cloud bucket, as well as with credentials giving write access only to that bucket. Credentials
also included adjective-adjective-noun names (e.g. grumpy-amethyst-chipmunk), generated with the
namegenerator python package to ensure contractor anonymity when we publish the data.
B.2
Contractor Contract
We recruited contractors by posting the following offer on the UpWork freelancing platform.
“We are collecting data for training AI models in Minecraft. You’ll need to install
java, download the modified version of Minecraft (that collects and uploads your
play data), and play Minecraft survival mode! Paid per hour of gameplay. Prior
experience in Minecraft not necessary. We do not collect any data that is unrelated
to Minecraft from your computer.”
19


We had the applications open for a day, and then randomly selected 10 applicants for the first round of
contractors. Later in the project, as we needed more data and as some contractors asked to terminate
their contracts, we added more applicants from the original pool as well as referrals from the currently
working contractors. The contractors were paid 20perhour(minusUpworkplatformfeesandapplicabletaxes).Alloftheresultspresentedinthispaperarebasedonabout4,500hoursofdata(includingdatarecordedtogatherstatisticsofhumanplaythatwasnotusedfortraining),whichcostusaround90,000. Over the course of the project, we collected some data we did not use due to
bugs in the recorder and for some ideas we ultimately did not pursue. In total, we spent about 160kforcontractorcompensationoverthecourseoftheproject.However,aswediscussinSec.4.6,wecouldlikelyobtainmostofourresultswithanIDMtrainedusingonly2000 worth of data, i.e. the
foundation VPT model, BC fine-tuning to the earlygame_keyword dataset, and the RL fine-tuning
results. Collecting the contractor_house dataset cost about 8000.BecauseweusedtheIDMtrainedonabout2000hoursofcontractordata,theactualcostofcontractordataforthoseresultswasaround40,000.
In early stages of the project, we were planning to use contractor data solely for the purpose of training
the IDM. As such, no specific tasks were given, other than “play the survival mode of Minecraft like
you normally would.” Later in the project, we requested that contractors perform specific tasks in
Minecraft, such as:
• Collect as many units of wood as possible, using only wooden or stone tools (treechop)
• Start a new world every 30 minutes of game play
• Build a basic house in 10 minutes using only dirt, wood, sand, and either wooden or stone
tools (contractor_house, more details below in Appendix B.4).
• Starting from a new world and an empty inventory, find resources and craft a diamond
pickaxe in 20 minutes (obtain_diamond_pickaxe). This dataset was used to obtain
statistics for how long it takes humans on average to complete this task (and the subtasks
required to complete it) when obtaining a diamond pickaxe is their goal.
Since we only recorded in-game events and videos, the data does not include personally identifiable
information. That being said, the contractors could theoretically use Minecraft’s open-world property
to generate personally identifiable information and/or offensive content (e.g. by using Minecraft
blocks to write their name or offensive messages, then finding a spot from which the message would
be visible). In practice, we have not seen any attempts to do so in the contractor videos that we
watched. Of course, we train our BC models on videos from the internet of people playing Minecraft,
and if such behavior is in those videos our model could also potentially learn it, although we expect
such behavior is rare enough that our model would not be likely to reproduce it.
B.3
Data for the Inverse Dynamics Model.
Since the IDM’s task is to infer actions given the video, any labelled data is appropriate for IDM
training. In practice, we included general gameplay as well as the treechop task data described
in the previous section, which amounted to a total of 1962 hours. Due to collecting datasets like
contractor_house only at late stages of the project, they were not included in IDM training.
B.4
contractor_house.
The contractor_house contains about 420 hours of data. We asked contractors to build a basic
house in 10 minutes, using only basic dirt, wood, and sand, blocks. Each trajectory starts in a newly
generated world and a timer forcibly ends a trajectory after a 20 minute time limit. For this task, many
contractors chose to begin their trajectories by crafting basic tools and building blocks, specifically
it was common for the first 2 minutes to be spent crafting a wooden pickaxe and then mining stone
for an assortment of stone tools before gathering more building blocks and beginning to create their
structure.
C
Minecraft environment details
Our Minecraft training environment is a hybrid between MineRL27 and the MCP-Reborn
(github.com/Hexeption/MCP-Reborn) Minecraft modding package. Unlike the regular Minecraft
20


Figure 12: (Left) Sample of a Minecraft frame in the original resolution (640x360) with an in-game
GUI open. The mouse cursor can be seen in the center of the image. This particular GUI shows the
player’s inventory and can be used to craft very basic items. (Middle) We downsample images to
128x128 for computational reasons. Shown is a downsampled observation with an in-game GUI for
crafting. This is the resolution used by our models. (Right) A 128x128 observation as seen by our
models without in-game GUI. The health, hunger, hotbar overlays, and agent hand can be seen in the
lower part of the image.
game, in which the server (or the "world") always runs at 20Hz and the client runs as fast as rendering
can complete (typically at 60-100Hz), in our version the client and server run in the same thread
at the same frequency. This allows us to run the environment slower or faster than real time, while
avoiding artifacts like missing chunks of the world. The action and observation spaces are similar
to those of MineRL environments and are described in more detail in the following subsections.
The environment also returns diagnostic information, such as in-game stats, contents of the agent’s
inventory, whether any in-game GUI is open, etc., which we use for tracking and recording but not as
inputs to the models. The episode length is 10 minutes for RL experiments and 60 minutes for BC
model evaluations. The agent can "die" in a number of ways, such as staying under water for too long
and drowning, being killed by hostile mobs, or falling from a tall structure. We do not terminate the
episode on agent "death". Instead, just as for humans in the regular Minecraft game, the agent drops
all its items when it dies and respawns at a random spot close to the initial spawning spot in the same
Minecraft world. The policy state is not masked on death, so the model can remember the fact that it
has died and act accordingly.
C.1
Observation space
The environment observations are simply the raw pixels from the Minecraft game that a human
would see. Unlike MineRL, we do not remove overlays like the hotbar, health indicators, and the
animation of a moving hand shown in response to the attack or “use” actions. The field of view is
70 degrees, which corresponds to the Minecraft default. GUI scale (a parameter controlling the size
of the in-game GUI) is set to 2, and brightness is set to 2 (which is not a Minecraft default, but is
very frequently used in online videos). The rendering resolution is 640x360, which is downsampled
to 128x128 before being input to the models. We empirically found 128x128 to be the smallest
resolution for which in-game GUI elements are still discernible, and then chose that to minimize
compute costs. Whenever an in-game GUI is open, we additionally render an image of a mouse
cursor at the appropriate mouse position to match what a human player’s operating system does (Fig.
12).
C.2
Action space
Our action space includes almost all actions directly available to human players, such as keypresses,
mouse movements, and clicks. The specific binary actions we include are shown in Table 3.
One difference between the human action space and our agent’s is that we disallow typing arbitrary
letters, which is only useful for entering text into the search bar of the crafting recipe book. Humans
can either do that or browse the recipe book with the mouse, the latter of which our agent can still
do. However, because we do allow the agent to press letters that are also shortcuts for actions (e.g.
outside of the GUI, the "W" key triggers the forward action) agents are able to press a few keys
within the GUI (W, A, S, D, E, Q) that produce letters if the recipe book search bar is selected. We
have not seen agents attempt to search the recipe book with these letters. Instead, our agents navigate
the recipe book with the mouse or craft by dragging items around the crafting window.
21


Action
Human action
Description
forward
W key
Move forward.
back
S key
Move backward.
left
A key
Strafe left.
right
D key
Strafe right.
jump
space key
Jump.
inventory
E key
Open or close inventory and the 2x2 crafting grid.
sneak
shift key
Move carefully in current direction of motion. In the
GUI it acts as a modifier key: when used with attack
it moves item from/to the inventory to/from the hot-
bar, and when used with craft it crafts the maximum
number of items possible instead of just 1.
sprint
ctrl key
Move fast in the current direction of motion.
attack
left mouse button
Attack; In GUI, pick up the stack of items or place the
stack of items in a GUI cell; when used as a double
click (attack - no attack - attack sequence), collect all
items of the same kind present in inventory as a single
stack.
use
right mouse button
Place the item currently held or use the block the player
is looking at. In GUI, pick up the stack of items or place
a single item from a stack held by mouse.
drop
Q key
Drop a single item from the stack of items the player
is currently holding. If the player presses ctrl-Q then
it drops the entire stack. In the GUI, the same thing
happens except to the item the mouse is hovering over.
hotbar.[1-9]
keys 1 – 9
Switch active item to the one in a given hotbar cell.
Table 3: Binary actions included in the action space. https://minecraft.fandom.com/wiki/
Controls has more detailed descriptions of each action.
In addition to the binary (on/off) keypress actions, our action space also includes mouse movements.
As with human gameplay, when in-game GUIs are not open, mouse X and Y actions change the
agent’s yaw and pitch, respectively. When a GUI is open, camera actions move the mouse cursor.
Mouse movements are relative (i.e. they move the mouse or camera relative to the current position,
and thus their effect depends on the current position).
Inventory interaction in Minecraft requires fine-grained mouse movements to achieve tasks such as
crafting and smelting, while mining and navigating the world can be achieved with coarser mouse
action. To be able to achieve both with the same action space, we implemented mouse movements
as a set of discrete actions with foveated binning along each axis (Fig. 13), which in preliminary
experiments we found to improve crafting performance.
D
Inverse Dynamics Model Training Details
D.1
IDM Architecture
The IDM model has approximately 0.5 billion trainable weights. The input to the IDM is 128
consecutive image frames (128 frames of video), each of which has dimensions 128 × 128 × 3. The
IDM is tasked with predicting the action at each frame. All image pixel values are first divided
by 255.0 such that they lie within the range [0, 1]. The first layer of the IDM is a 3-D convolution
with 128 learnable filters with a temporal kernel width of 5 and spatial kernel widths of 1. This
convolution is non-causal, meaning that embeddings at time index t are functions of pixel values at
times t −2, t −1, t, t + 1, and t + 2. We found this layer to be extremely important in IDM training
22


−80 −60 −40 −20
0
20
40
60
80
Mouse movement (pixels)
−10
−5
0
5
10
Camera angle (deg)
0
2
4
6
8
10
Camera bin
Figure 13: Relative camera angle or mouse movement in pixels vs. action bin. The same binning is
used for both X and Y coordinates. The binning is foveated, meaning that binning is more fine-grained
for smaller movements and more coarse-grained for larger movements. There are 11 bins for each
axis (X and Y). The center of each bin (indicated with green circles) is used when un-discretizing
movements (that is, when converting from an action expressed as a bin to a camera angle or mouse
movement).
as it incorporates neighboring temporal information immediately, and we show results comparing
IDM performance with and without this layer in Figure 14. This comparison was made on the default
(1962-hour) IDM dataset.
0
5
10
15
20
25
30
35
Training Progress (Epoch)
0.5
1.0
1.5
2.0
2.5
3.0
Loss
With 3D Conv
No 3D Conv
0
5
10
15
20
25
30
35
Training Progress (Epoch)
0.0
0.2
0.4
0.6
0.8
Keypress Accuracy
With 3D Conv
No 3D Conv
0
5
10
15
20
25
30
35
Training Progress (Epoch)
6
5
4
3
2
1
0
1
Mouse R2
With 3D Conv
No 3D Conv
Figure 14: Effect of 3-D Convolution in the IDM Architecture.
This initial temporal convolutional layer is followed by a ResNet62 image processing network. In this
part of the model, no extra temporal information is shared between neighboring frames; however,
since each frame was first processed with the temporal convolution, some temporal information
is present at this stage. The ResNet image processing network is comprised of three subsequent
stacks with widths W = {64, 128, 128}. Each stack is comprised of, in order, (1) an initial 3x3
convolutional layer with 1-pixel zero padding at the embedding boundary (such that the outgoing
embedding dimensions are the same as the incoming embedding dimension) with W output channels,
(2) a 3x3 max pooling with stride 2 and padding 1 such that the embedding width and height are
halved, and (3) two classic ResNet blocks as defined in He et al. 62 with each layer also having W
output channels.
The output of the ResNet stack is flattened into a 1-dimensional vector of size 217 = 131072 (one
vector for each frame in the video) such that at this stage there are 128 vectors of size 131072. Each
vector is independently processed with two frame-wise dense layers with 256 output activations and
then 4096 output activations, respectively. The result is then fed through 4 subsequent non-causal
(umasked) residual transformer69 blocks. Each block first has an unmasked attention layer, i.e. frames
may attend to future frames, with 32 attention heads of dimension 128 each and a surrounding residual
connection that skips this layer. The embedding is then passed through a frame-wise dense layer
with output dimension 16384 and another with output dimension returning to 4096; a single residual
connection skips past this pair of frame-wise dense layers (not skipping past each layer separately,
but skipping the pair). All dense layers have their weights tied through time, so each frame in the
video is processed with the same weights.
Finally, independent dense layer heads for each action are pulled from the final embedding – a 2
class on/off categorical parameterized with a softmax for each available key as well as a 11-way
23


categorical for both the discretized horizontal and vertical mouse movements (See Appendix C.2 for
details on the action space).
Each dense layer or convolutional layer in the network is preceded by a layernorm70 and followed by
a ReLU non-linearity. Weights are initialized with Fan-In initialization71 and biases are initialized to
zero.
D.2
IDM Training
The total loss for the network is the sum of each independent action prediction loss (one for each
key and one for both mouse directions). Each independent loss is the negative log-likelihood of the
correct action. We use the ADAM72 optimizer with a linear learning rate decay. We use an initial
learning rate of 0.003, a batch size of 128 (where each item in the batch is a video sequence of 128
frames), and a weight decay of 0.01. Hyperparameters were tuned in preliminary experiments. The
IDM is trained on our contractor collected dataset for 20 epochs. This took 4 days on 32 A100 GPUs.
We add data augmentation to each video segment; augmentations are randomly sampled once per
segment such they are temporally consistent. Using the Pytorch73 transforms library, we adjust the
hue by a random factor between -0.2 and 0.2, saturation between 0.8 and 1.2, brightness between
0.8 and 1.2, and contrast between 0.8 and 1.2. We also randomly rotate the image between -2 and 2
degrees, scale it by a random factor between 0.98 and 1.02, shear it between -2 and 2 degrees, and
translate it between -2 and 2 pixels in both the x and y dimensions.
Due the large computational cost of running all of the experiments in this paper, training results
are from one run of training (for IDM, BC, and RL training): this non-ideal situation is mitigated
because deep learning training tends to be low variance74,75 and because we often have data points
from sweeps (e.g. on dataset size) that suggest overall trends.
D.3
Generating Pseudo Labels with the IDM
Section 4.1 shows that inverse dynamics modeling is a much easier task than behavioral cloning
because IDMs can be non-causal. The IDM is trained to simultaneously predict all 128 actions for
each video sequence, so the IDM will effectively be causal for frames at the end of the video clip
because future frames are not included in the sequence. For this reason, we apply the IDM over a
video using a sliding window with stride 64 frames and only use the pseudo-label prediction for
frames 32 to 96 (the center 64 frames). By doing this, the IDM prediction at the boundary of the
video clip is never used except for the first and last frames of a full video.
E
Foundation Model Behavioral Cloning
E.1
Foundation Model Architecture
The behavioral cloning model architecture is the same as the IDM architecture described in Appendix
D.1 except that we modify the architecture so that it is causal (i.e. cannot see the future when making
predictions). This means the BC architecture does not have the initial non-causal convolution the
IDM has (this layer is omitted completely). Furthermore, the residual transformer layers are now
causally masked (as is standard in language modeling) and we do Transformer-XL-style76 training
where frames can attend to keys and values from past batches within the same video. We also use a
Transformer-XL-style relative attention position embedding.
E.2
Null Action Filtering
The most common action humans take is the null action (no keypresses or mouse movements), which
accounts for 35% of all actions they take. Among other reasons, a player may take the null action to
wait for something in the game to finish, to pause between actions, or to take a break to grab a glass
of water. Early on in the project we found that the BC model would take a much larger fraction than
35% of null actions, often upwards of 95%. In order to prevent this behavior we removed frames with
null actions from the dataset. We compare a few different treatments: we filter nulls if there have
been 1, 3, or 21 frames of consecutive null actions, and include a treatment that does not perform
any null filtering. Null action filtering generally helps, increasing all crafting rates (Figure 15 left).
24


basic mining
logs
planks
crafting tables
total crafting
null actions
10 2
100
102
104
Effect of Null Action Filtering
None
All
Groups of 3
Groups of 21
basic mining
logs
planks
crafting tables
total crafting
null actions
10 3
10 2
10 1
100
101
102
103
Joint Hierarchical vs Factored Action Spaces
Factored
Hierarchical Joint
Figure 15: (Left) Effect of Null Action Filtering during training. We compare environment metrics
and number of sampled null action during rollouts (rightmost group of columns) for the following
treatments: no null action filtering (blue), filtering all null actions (green), filtering only groups of
3 or more null actions (red), and filtering only groups of 21 or more null actions (purple). (Right)
Hierarchical versus Factored Action Spaces.
Filtering only groups of 3 performed slightly better than filtering all null action or groups of 21. Initial
experiments indicated that filtering all null actions was better; however, after further model tuning
and after we had already trained our largest models, we found that filtering only groups of 3 or more
null actions performed best. Due to compute constraints we were not able to redo all experiments
with this setting, but doing so would be a reasonable choice for any future work.
E.3
Joint Hierarchical Action Space
We originally worked with a factored action space, where each keypress could be independently on
or off, and this choice was independent of whether the mouse was being moved. This could cause
issues for modeling the human behavior distribution exactly. Say for a given state, humans either with
50% probability (a) move forward and attack or with 50% probability (b) move left and drop their
item. The best a factored distribution can do is to assign 50% probability to each of the 4 constituent
actions because it chooses to press each button simultaneously and independently. See Appendix C.2
for details on the entire action space.
For this reason, we implemented a joint distribution over actions; however, the full joint distribution
over 20 binary buttons and two mouse movement dimensions discretized into 11 bins each would
result in in 220 × 112 ≈1.2 × 108 possible combinations. This is far too large for many reasons, e.g.
the final layer from the transformer stack with a dimension of 4096 would need to be mapped to each
combination resulting in 4096 × 1.2 × 108 ≈5.2 × 1011 parameters for this final layer alone. In
order to reduce this we noted that many buttons in Minecraft have no effect when simultaneously
pressed; for example, if a player tries to move forward and backward at the same time, they remain in
place. Below we list the the sets of mutually exclusive actions. Furthermore, the inventory button is
exclusive with all other buttons and mouse movement.
Mutually Exclusive Actions
forward, back
left, right
sprint, sneak
hotbar.[1-9]
Even reducing the joint action space to reflect these mutually exclusive combinations still results in a
huge action space when combined with the discretized mouse movements, i.e. 33×10×24×112+1 ≈
5.2 × 105. This calculation results from 33 for the 3 sets of 2 mutually exclusive keys above where
taking neither in the set is an option, ×10 for the 9 hotbar keys or no hotbar keypress, ×24 for the
remaining binary 4 keys: use, drop, attack, and jump, ×112 for mouse movements, and finally
+1 for the inventory button which is mutually exclusive with all other actions. ∼5.2 × 105 is still
quite large so we chose to implement a hierarchical binary action for camera being moved or not. If
this action is on, then there is a secondary discrete action head with 121 classes (the joint distribution
of mouse movements because each discretized mouse direction has 11 bins) that determines where
25


to move the mouse. If the hierarchical action is off, then there is no mouse movement, loss for
the secondary mouse movement action is masked during training, and the secondary action head
need not be sampled during evaluations. While this no longer models the full joint distribution, it is
quite a bit better than the factored action space since dependencies between keypresses as well as
whether or not to move the mouse (although not which mouse movement) are modeled jointly. The
resulting action space has dimension 33 × 10 × 24 × 2 + 1 = 8461 (the 112 dimensional multiplier
for camera movement has been replaced by a multiplier of 2 here, corresponding to a binary action
for whether or not to move the mouse) with an additional 121-dimension head for the joint camera
movements. In the future it would be interesting to implement sequential conditional action spaces to
more completely model the joint distribution.
In Figure 15 (right) we compare environment rollout performance between BC models with the
hierarchical joint action space and with the factored action space. Environment statistics are fairly
comparable; however, we see that the factored action space model samples far more null actions.
This is an important example of the factored action space failing to correctly model the distribution in
the dataset because, due to null action filtering, there are 0 null actions in the dataset these models
train on. Despite this, the factored model samples many null actions because the prediction for each
key is not conditioned on other keypresses.
E.4
Foundation Model Training
The foundation model training is similar to the IDM training, with the exception of labels being
IDM-generated pseudo labels. The hyperparameters used for foundation model training are listed in
Table 4.
Hyperparameter
Value
Learning rate
0.002147
Weight decay
0.0625
Epochs
30
Batch size
880
Table 4: Hyperparameters for foundation model training
F
Behavioral Cloning Fine-Tuning
Behavior cloning fine-tuning is similar to the foundation model training, except we either use a focused
subset of all the videos (early_game dataset, described in A.3) with pseudo labels, or contractor data
(contractor_house dataset, described in B.4) with ground-truth labels. The hyperparameters used
for behavior cloning fine-tuning are listed in Table 5. We used 16 A100 GPUs for about 6 hours when
fine-tuning on contractor_house dataset, and 16 A100 GPUs for about 2 days when fine-tuning
on early_game dataset.
Hyperparameter
Value
Learning rate
0.000181
Weight decay
0.039428
Epochs
2
Batch size
16
Table 5: Hyperparameters for behavior cloning fine-tuning
G
Reinforcement Learning Fine-Tuning
G.1
Reinforcement Learning Fine-Tuning Training Details
RL experiments were performed with the phasic policy gradient (PPG) algorithm,64 an RL algorithm
based on the proximal policy optimization (PPO) algorithm77 that increases sample efficiency by
performing additional passes over the collected data to optimize the value function as well as an
26


auxiliary value function. These algorithms have been described extensively in previous work,64,77
so here we describe them only briefly. A major inefficiency when training on-policy algorithms is
that, to remain on-policy, one can only take a single gradient step before new rollout data needs
to be gathered to continue optimization. To alleviate the potentially destructive effects of taking
multiple optimization steps in a single iteration, PPO prevents the policy from changing too much
in a single step by clipping the loss when the difference between the current policy and the policy
before the update becomes too large.77 We also use generalized advantage estimation (GAE), which
can speed-up credit assignment by looking more than 1 step into the future when determining the
advantage of an action, with the look-ahead being determined by hyperparameter λ.78
PPG improves the sample efficiency of PPO when the policy and value function share the same
network by following different optimization processes for the policy, the value function, and their
shared representation. PPG splits optimization in two phases: a wake phase and a sleep phase. In the
wake phase, the policy and value function are optimized as in normal PPO training, with the only
exception being that every sample is used at most once, which prevents the policy from overfitting on
these samples. In the sleep phase PPG optimizes the value function and an auxiliary value function
(which is optimized with the exact same loss as the regular value function, but its output is never used
during training), while keeping a Kullback-Leibler (KL) divergence loss to the policy before the start
of the sleep phase to ensure that the policy does not change. Because the policy is not optimized
in this step, PPG does allow samples to be reused multiple times in this phase. The assumption
behind optimizing the value function during the sleep phase is that value function optimization is
less sensitive to being trained multiple times on the same sample. Optimizing the auxiliary value
function does not directly affect either the value function or the policy, but it can improve the shared
representation of both functions (the assumption being that predicting the value-function requires
encoding all features that are important for distinguishing states). The coefficients for the three
losses (value function loss, auxiliary value function loss, and KL loss) are listed in Table 6. In our
experiments a single iteration consists of two sleep cycles and one wake cycle.
Because the value and auxiliary value functions are not optimized during BC pre-training, they
are initialized at the start of RL fine-tuning. Each value function is implemented as a single, fully
connected layer on top of the last residual transformer block of the pretrained model (Appendix D.1).
The weights of the auxiliary value function are randomly initialized while the weights of the regular
value function are initialized with zero weights, which appeared to prevent destructive updates early
in training that could happen with a randomly initialized value function. To prevent the value-function
loss from having gradients that depend greatly on the magnitude of the reward, we normalize the
value-function target by subtracting the mean and dividing by the standard deviation, which are
estimated through an exponentially weighted moving average.
To prevent catastrophically forgetting the skills of the pretrained network when RL fine-tuning, we
apply an auxiliary KL divergence loss between the RL model and the frozen pretrained policy.10 This
loss is defined as:
Lklpt = ρKL(πpt, πθ)
(2)
Where πθ is the the policy being trained, πpt is the frozen pretrained policy, KL(πpt, πθ) is the
Kullback-Leibler divergence between the policy being trained and the pretrained policy, and ρ is a
coefficient to weight this loss relative to other losses.
In the fine-tuning experiments, this KL divergence loss replaces the common entropy maximization
loss, which is often added to RL experiments to encourage exploration.79,80 The idea behind entropy
maximization is that, when all actions appear to have equal value, such as when the agent has not
learned about the next reward, it should maximize its entropy to increase the chance that it discovers
the next reward. Blindly exploring by maximizing entropy is effective when the state and action
spaces are sufficiently small or the reward is sufficiently dense, but becomes infeasible when the
state and action spaces are large and rewards are sparse, which is the case in the diamond-pickaxe
task. Instead of blindly exploring through uniform-random actions, we assume that the pretrained
policy has an action distribution that is much more likely to take sequences of actions that lead
to interestingly new states, and thus, in states where the agent assigns equal value to each of its
actions, it should mimic the action-distribution of the pretrained policy instead of a uniform-random
action distribution. In experiments with a randomly initialized policy we do include the entropy
maximization loss with a coefficient of 0.01, which has been an effective setting in other Minecraft
work.30 Empirically, we found that a high coefficient ρ for this KL divergence loss would prevent
the agent from properly optimizing the reward function while a low coefficient ρ was ineffective at
27


Hyperparameter
Value
Learning rate:
2 × 10−5
Weight decay:
0.04
Batch size:
40
Batches per iteration:
48
Context length:
128
Discount factor (γ):
0.999
GAE λ:
0.95
PPO clip:
0.2
Max Grad norm:
5
Max Staleness:
2
PPG sleep cycles:
2
PPG sleep value-function coefficient:
0.5
PPG sleep auxiliary value-function coefficient:
0.5
PPG sleep KL coefficient:
1.0
PPG sleep max Sample Reuse:
6
KL divergence coefficient ρ:
0.2
Coefficient ρ decay:
0.9995
Table 6: Hyperparameters for RL experiments. These are the hyperparameters for all treatments with
two exceptions. First, when fine-tuning from the early-game model without a KL divergence loss,
in addition to the KL divergence loss being set to 0, the learning rate was set to 3 × 10−6 (the best
setting out of a sweep over 5 different learning rates), as we found that performance was substantially
lower with the standard learning rate of 2 × 10−5 and the agent did not even learn to collect logs.
We suspect that the reason that the learning rate needed to be lowered when fine-tuning without a
KL loss is that the KL loss prevents making optimization steps that change the policy too much in a
single step, especially in early iterations when the value function has not been optimized yet, and
the KL loss thus makes it possible to optimize with a higher learning rate. Second, when running
RL from a randomly initialized policy there is no KL divergence loss or KL divergence decay, but
instead we use an entropy bonus of 0.01, which reportedly worked well in previous work.30
protecting the learned skills of the pretrained policy and preventing catastrophic forgetting. As such,
we start with a relatively high coefficient ρ and decay it by a fixed factor after each iteration (Table 6).
This method protects policy skills in early iterations while guaranteeing that the policy can eventually
maximize the reward function, regardless of how different its behavior has to be to do so relative to
the pretrained policy.
For the reward function we estimated the rough quantities of each item that a human player might
gather when trying to craft a diamond pickaxe, and we reward the model for gathering up to that
quantity for each item. We started these estimates by iterating over the technology tree backward from
a diamond pickaxe and adding the requirements for each item to the reward function (e.g. first we
added a diamond pickaxe to the reward function, then we added the 3 diamonds and 2 sticks required
for crafting a diamond pickaxe, then we added the 1 iron pickaxe required for mining diamonds,
and so on). Then we added coal and torches to the reward function, with coal being useful as fuel
when smelting iron and for crafting torches while the torches themselves improve visibility and
prevent enemies from spawning. Finally, we reward the model for bringing additional logs (5 logs are
required to craft all items in the reward function, but we reward up to 8 logs), which can be used as
fuel or crafted into a crafting table or sticks if the agent runs out. In practice the agent rarely collects
the additional logs, places the torches, or uses coal as fuel when smelting, but the reward function
was based on human expectations on what would be useful to execute this task, rather than designed
around how an RL model behaves after training. Finally, to encourage the agent to keep mining
diamonds and crafting diamond pickaxes after it has crafted its first diamond pickaxe, we did not put
a limit on the number of diamonds or diamond pickaxes that would be rewarded.
The rewards for the different items are separated into 4 tiers, roughly depending on how late a player
would usually get the relevant item. The first tier consists of all wooden and stone items and has a
base reward of 1, the second tier consists of all items requiring coal with a base reward of 2, the third
tier consists of all items requiring iron with a base reward of 4, and the final tier is diamond with a
base reward of 8. Thus items later in the sequence of items towards a diamond pickaxe generally
28


Item
Quantity rewarded
Reward per item
Log
8
1/8
Planks
20
1/20
Stick
16
1/16
Crafting table
1
1
Wooden pickaxe
1
1
Cobblestone
11
1/11
Stone pickaxe
1
1
Furnace
1
1
Coal
5
2/5
Torch
16
1/8
Iron ore
3
4/3
Iron ingot
3
4/3
Iron pickaxe
1
4
Diamond
inf
8/3
Diamond pickaxe
inf
8
Table 7: Reward per item and total quantity rewarded.
give a higher reward. To make sure that the agent does not over-value items that are supposed to
be gathered in bulk (e.g. the agent is rewarded for up to 20 planks but only up to 1 crafting table,
which can cause the agent to focus on planks at the expense of creating a crafting table), we divide
the base reward of each item by the total quantity that the agent gets rewarded for (for the purpose
of determining the reward, the total quantity for diamonds is 3 and the total quantity for diamond
pickaxes is 1, even though we did not put a limit on the number of these items being rewarded). For
example, the agent is rewarded for 3 iron ore, which has a base reward of 4 for being in the iron tier
and up to 3 blocks of iron ore are rewarded, thus the reward per block of iron ore is 4/3. The quantity
and reward for each item are listed in Table 7.
While every item in the sequence towards a diamond pickaxe is rewarded, the reward function is still
sparse and, in some cases, even deceptive. The sparsity comes from the fact that it can take thousands
of actions to find the next reward, even after the agent has acquired all the necessary prerequisites
(e.g. human players often take more than 10,000 actions to find a diamond after crafting an iron
pickaxe). The reward function can be deceptive when the most efficient method for getting one item
can make it far more difficult to get the next item. For example, a good strategy for the agent to
craft a stone pickaxe quickly is to mine (i.e. spend a few seconds to pick up) its crafting table after
crafting a wooden pickaxe, such that the agent has immediate access to a crafting table as soon as it
has collected enough cobblestone. However, the fastest way to get a reward for gathering cobblestone
is to mine down immediately after crafting a wooden pickaxe, while leaving the crafting table behind.
Thus following the optimal strategy for gathering cobblestone makes it more difficult to learn to craft
a stone pickaxe.
Experiments ran for approximately 6 days (144 hours) on 80 GPUs (for policy optimization) and
56,719 CPUs (mostly for collecting rollouts from Minecraft). In this time the algorithm performed
roughly 4,000 optimization iterations and collected roughly 1.4 million Minecraft episodes consisting
of 12,000 frames each, for a total of 16.8 billion frames.
G.2
Reinforcement Learning Fine-Tuning Additional Data
Additional figures that are helpful for understanding the main results of the RL fine-tuning experiments
are presented in this section. First, we show the items-over-training figure when RL fine-tuning from
the early-game model without a KL loss (Fig. 16). When training without a KL loss, the model
only learns to obtain the four items that the early-game model is capable of getting zero-shot, which
are logs, planks, sticks, and crafting tables. Second, we present preliminary experiments in which
we directly compare RL fine-tuning from the house-building model and RL fine-tuning from the
early-game model (Fig. 17). These experiments differ from the main experiments in that, for both
treatments shown here, the KL loss coefficient was set to 0.4, the learning rate was set to 6 × 10−5,
and the reward for each item was 1/quantity for all items (i.e. items closer to the diamond pickaxe
did not have an increased reward). While RL fine-tuning from the house-building model initially
29


worked better than RL fine-tuning from the early-game model, fine-tuning from the early-game model
worked better after 800,000 episodes and showed signs of smelting iron ingots, which is why the
early-game model was chosen for the main experiments.
0.0
0.2
0.4
0.6
0.8
1.0
1.2
1.4
Episodes
1e6
0
20
40
60
80
100
% episodes
No KL-loss
Figure 16: Items obtained when RL fine-tuning from the early-game model without a KL loss. The
model learns to obtain all items that the early-game model can craft zero-shot, which are logs, planks,
sticks, and a crafting table. In contrast to the treatment with a KL-penalty, it does not learn any items
beyond these initial four, likely because skills that are not performed zero-shot, and for which the
model thus does not initially see any reward, are catastrophically forgotten while the first four items
are learned.
0
100000
200000
300000
400000
500000
600000
700000
800000
Episodes
0
2
4
6
8
Reward
Reward over episodes
RL from Early-Game model
RL from House-Builing model
0
100000
200000
300000
400000
500000
600000
700000
800000
Episodes
0.000
0.005
0.010
0.015
0.020
0.025
0.030
0.035
Iron Ingots Obtained
Iron Ingots Obtained Per Episodes
RL from Early-Game model
RL from House-Builing model
Figure 17: Preliminary experiments when RL fine-tuning from the early-game model compared to
RL fine-tuning from the house-building model. (Left) While reward initially increases faster when
fine-tuning from the house-building model, fine-tuning form the early-game model eventually obtains
a slightly higher reward. (Right) RL fine-tuning from the early-game model has a higher likelihood
of smelting an iron-ingot, which is why the early-game model was chosen for future RL fine-tuning
experiments.
H
Foundation Model Scaling
In early experiments we found that increasing model size led to models staying in the efficient
learning regime longer into training.63 Here we compare the 0.5B model described in Section 4.2 to
both a 248M and 71M parameter model. Both of these models are trained for 15 epochs as compared
to the 30 epochs the 0.5B model trained for. These models have the same architecture as the 0.5B
model but each layer in the 248M parameter model has 1/2 the width and each layer in the 71M
parameter model 1/3 the width. The 71M model was trained with an initial learning rate of 0.001586,
batch size of 480, and weight decay of 0.044506. The 248M model had an initial learning rate of
0.001831, batch size of 640, and weight decay of 0.051376.
In Figure 18 we show validation loss on web_clean with IDM pseudo-labels, loss on the contractor
dataset used to train the IDM with ground truth labels collected during contractor play, and zero-shot
environment performance for the 71M, 248M, and 0.5B models. While larger models have better
validation loss on web_clean, these results do not tell the clear story that the 0.5B model is better
than its smaller counterparts. The 71M model has the lowest contractor dataset loss while having the
highest web_clean loss, and it also has the best zero-shot environment performance. In fact, we see
that the 71M model even had non-zero wooden tool crafting (Fig. 18 bottom left). The 248M model
also appears to be better at crafting than the 0.5B, and also has lower contractor dataset loss.
While the zero-shot results suggest smaller models are better, fine-tuning tells another story. When
fine-tuning to contractor_house, model size rank ordering reverses and now the 0.5B model
performs best both in validation loss (Fig. 19 left) and in environment performance (Fig. 19 right)
30


10 3
10 2
10 1
100
101
~Compute
2.5
4
6
10
Web Clean Loss
Loss Web Clean Validation Dataset
71M
248M
0.5B
10 3
10 2
10 1
100
101
~Compute
2.5
4
6
10
IDM Contractor Dataset Loss
Loss on IDM Contractor Dataset
71M
248M
0.5B
basic
mining
logs
planks
crafting
tables
total
crafting
10 2
10 1
100
101
Collected or Crafted
Zero-Shot Performance vs Model Size
71M
248M
0.5B
0
2
4
6
8
10
12
14
16
Training Progress (Epoch)
0
10 3
10 2
10 1
100
101
Crafting or Collection
Zero-Shot Performance over Training
(71M Parameter Model)
basic mining
logs
planks
crafting tables
total crafting
wooden tools
0
2
4
6
8
10
12
14
16
Training Progress (Epoch)
0
10 3
10 2
10 1
100
101
Zero-Shot Performance over Training
(248M Parameter Model)
0
5
10
15
20
25
Training Progress (Epoch)
0
10 3
10 2
10 1
100
101
Zero-Shot Performance over Training
(0.5B Parameter Model)
Figure 18: Training and Zero-Shot Performance versus Model Scale. In the first two plots the x-axis
is compute normalized to that used by the 71M parameter model, such that after 15 epochs of training
the 71M model has used 1 "compute". The 248M parameter model and the 71M model are trained on
the same amount of data (15 epochs), and the 0.5B parameter model is trained on 30 epochs of data.
(Top Left) Loss on the web_clean validation dataset. (Top Middle) Loss on the IDM contractor
dataset; note that these models were trained only on web_clean and not on any contractor data.
(Top Right) Zero-shot environment rollout performance at the end of training. (Bottom) Zero-shot
environment rollout performance over training for the 71M model (bottom left), 248M model (bottom
middle), and 0.5B model (bottom right).
basic
mining
logs
planks crafting
tables
total
crafting
wooden
tools
stone
tools
10 3
10 2
10 1
100
101
102
103
104
Collected or Crafted
Fine-Tuning to Contractor House Dataset
71M
248M
0.5B
0.0
0.5
1.0
1.5
2.0
2.5
3.0
Fine-Tuning Epoch
2.2
2.4
2.6
2.8
3.0
contractor_house Validation Loss
Loss on contractor_house
71M
248M
0.5B
0.0
0.5
1.0
1.5
2.0
2.5
3.0
Fine-Tuning Epoch
2.4
2.5
2.6
2.7
2.8
2.9
3.0
Loss on IDM Contractor Dataset
Loss on full IDM Contractor Dataset
71M
248M
0.5B
Figure 19: contractor_house fine-tuning performance versus model size. (Left) Loss on the
contractor_house holdout validation set. (Middle) Loss on the full contractor dataset collected
to train the IDM; this dataset is disjoint from contractor_house. (Right) Environment rollout
performance at the end of fine-tuning.
followed by the 248M model and then the 71M model. Environment model rollouts are performed
using the same game engine that we use to collect contractor data, which could be visually distinct
from videos taken from the web. It is plausible that the larger models overfocus on the visual
peculiarities in web data during pretraining since they have worse contractor data loss (Fig.18 top
middle), and this causes them to perform more poorly in the environment zero-shot. However, we
hypothesize that because the contractor_house dataset we fine-tune to is collected from our game
engine, the larger models that are a better overall Minecraft prior (as indicated by lower web_clean
validation loss in Fig.18 top left) can quickly shift their low level features to perform better on data
coming from our game engine, resulting in better environment rollout performance. This hypothesis
is further supported by Fig. 19 (middle) showing loss on the contractor dataset collected for IDM
training, which has no overlap with contractor_house. After just a few steps of fine-tuning to
contractor_house, all models quickly improve in loss on the full IDM contractor dataset, with
larger models now performing best. While not conclusive, we believe this investigation provides
some intuition for future studies of model scaling for sequential decision making problems.
31


I
Text Conditioning
Goal-conditioned policies81,82 make it possible for a single agent to perform a wide variety of
goals in a single environment, which is particularly relevant in open-ended environments such as
Minecraft. In recent work, goal specification has increasingly taken the form of domain specific
languages83, or even natural language84,85. The benefits of language-conditioned agents can be
tremendous, especially natural-language-conditioned agents, as their goal space contains a wide
variety of potentially very complex tasks. Text conditional models have shown an amazing ability to
perform tasks zero-shot (or learn them few-shot) including generalizing in impressive ways via the
compositional and combinatorial possibilities allowed by natural language (e.g. GPT1 and DALL·E
286). We hypothesize that we should expect similar capabilities to emerge with natural-language-
conditioned virtual agents, if they are similarly trained on enormous amounts of data (that goes from
a natural language description to a sequence of actions that completes the specified goal). In this
section we take preliminary steps toward that future. Our preliminary experiments provide evidence
that it is possible to pretrain a natural-language-conditioned model for Minecraft using the general
approach presented in this paper (VPT) plus conditioning on the speech that often accompanies
videos.
In online videos, the human actor sometimes indicates their intent in their verbal commentary
(e.g. “Let’s go chop some trees to make a wooden axe” or “now let’s learn how to crop photos in
Photoshop”). Conditioning on this closed caption data could produce a steerable pre-trained model:
i.e., it may later be possible to condition the model with text such as “I am going to craft a wooden
pickaxe” or “I am going to build a house,” and have the agent perform those tasks specifically rather
than simply follow typical human behavior (as was investigated in the rest of this paper). An alternate
way to produce a steerable agent is via RL fine-tuning, which we could have done in Section 4.4
by adding a bit indicating the task to be completed, as has been done in prior work30. However,
conditioning on natural language offers many benefits over that approach. First, it is flexible and
powerful, being able to express any task. Second, one does not need to preconceive of the task to
be completed ahead of time. This would allow for general, capable, zero-shot agents like GPT, but
extending those capabilities to embodied tasks such as completing tasks on computers or in simulated
3D worlds. Third, text conditioning can be used even when tasks are difficult to specify via reward
functions (e.g. “Let’s build a house” or–if the agent is capable of doing it–more complex things like
“I will now build a castle surrounded by a moat”). In the limit, VPT+text could conceivably produce
powerful, capable, natural-language-conditional agents with the powers of GPT to meta-learn, follow
instructions, and complete tasks zero or few shot, but in the form of agents that can act in virtual
worlds, complete tasks on computers, and in other similar embodied sequential decision domains.
We do not reach those lofty goals in this work, but we began a first step towards exploring in that
direction.
Many Minecraft videos feature audio commentary from the player. This commentary is sometimes
present in the form of closed captions for the videos, or could be extracted post-hoc using automated
speech recognition (ASR).87 Our dataset features about 17k hours of content with associated closed
captions.
We fine-tuned the 220 million parameter VPT foundation model used in the RL-fine-tuning ex-
periments (chosen vs. 0.5B for the same reason: to reduce compute costs) with an additional
text-conditioning input on the subset of our data for which closed captions are available. To obtain the
conditioning input, we first split videos into 30 second chunks. The same text is associated with every
frame in a given chunk, and is made up of all the closed captions occurring within that chunk, as well
as the line of text preceding and following the chunk (if any). Because the vast majority (around
95%) of our closed caption data lacks capitalization and punctuation, it is punctuated using the rpunct
library88. We then obtain a text embedding vector of length 4,096 from the OpenAI embedding API89,
which is processed by a randomly initialized multi-layer perceptron (MLP) with two hidden layers of
size 2,048. The resulting activations are added for each frame to the pretrained model activations
before the transformer layers (pretransformerActivations += mlp(textEmbedding)). The
model is fine-tuned for four epochs.
Our model shows evidence of steerability. When conditioned on sentences that incite the agent to
explore (such as “I’m going to explore” and “I’m going to find water”) the agent travels significantly
farther from its spawn point (Figure 20a). Additionally, we can steer the agent to preferentially collect
32


dig
dirt
explore
house
seed
water
wood
Conditioning
120
130
140
150
Travel distance (blocks)
Travel Distance with Conditioning
(a)
dig
dirt
explore
house
seed
water
wood
Conditioning
0.4
0.5
0.6
0.7
0.8
0.9
Wheat seeds collected
Seed Collection with Conditioning
(b)
dig
dirt
explore
house
seed
water
wood
Conditioning
1.00
1.25
1.50
1.75
2.00
2.25
2.50
2.75
Oak logs collected
Log Collection with Conditioning
(c)
dig
dirt
explore
house
seed
water
wood
Conditioning
5
6
7
8
9
10
Dirt collected
Dirt Collection with Conditioning
(d)
Figure 20: Evidence for conditioning. In each plot, the variants expected to stand out are shown in
bold. The strings corresponding to each variant are shown in Table 8. Statistics are measured over 5
minute episodes. (a) Distance traveled by the agent . Both “explore” and “water” text strings should
encourage a steerable agent to move more than when doing other tasks, which is what occurs. Grass
(which is needed to get seeds) is not present in all biomes, which is likely why the “seed” condition
produces more travel (as the agent sometimes needs to move to a biome with grass). The travel
distance is the Euclidean distance from the spawn point to the farthest point the agent reached during
the episode on the horizontal (x-z) plane. (b) Collection of wheat seeds. The “seed” variant collects
substantially more than other variants, as expected of a steerable agent. (c) Collection of oak (the
most common type of wood) logs. The “wood” variant collects significantly more oak logs, as is to
be expected of a steerable agent (we speculate that the “water” variant collects less because there are
no trees in water). (d) Collection of dirt. The “dirt” and “dig” variants collect a large amount, and are
the variants that are (indirectly in the case of “dig”) conditioned to collect dirt. It is easy to mistakenly
aim at the ground rather than at grass or trees when collecting seeds or wood, which likely explains
the slightly higher amount of dirt collected by these variants. In all cases, the error bars are 95%
confidence intervals of the mean, over 1,000 episodes per conditioning variant. Treatments for which
the bars in each bar plot do not overlap are statistically significantly different at a p < 0.05 level.
Variant name
String
dig
I’m going to dig as far as possible
dirt
I’m going to collect dirt
explore
I’m going to explore
house
I’m going to make a house
seed
I’m going to collect seeds
water
I’m going to find water
wood
I’m going to chop wood
Table 8: Strings corresponding to each conditioning variant.
33


early game items such as seeds, wood, and dirt by conditioning with text such as “I’m going to collect
seeds/chop wood/collect dirt” (Figure 20b,c,d).
While our results show some level of steerability, more work is required to increase it. For example,
we were not able to successfully steer agents to gather flowers or to hunt, both of which are possible
in the early game, but less common (and, in the case of hunting animals, much more difficult) than
gathering dirt, wood, or seeds. Likewise, an experiment in which the agent is presented with a
crafting window and various resources, and conditioned to craft a given item (e.g. “I’m going to
craft a wooden axe”) failed to show that the conditioning had a significant effect on which items got
crafted. Instead, it seemed the agent was more influenced by the prior, unconditional probability
of what human players would craft next given the resources available, which is not too surprising
since in Minecraft, especially in the early game, there is a relatively consistent path to gathering
resources in a specific order go produce more powerful tools (Fig. 6). For example, if the agent had
the resources to make a stone pickaxe and we asked it instead to make a (weaker) wooden pickaxe, it
often would make the stone pickaxe anyway. Finally, looking at videos of agent behaviors failed to
convince us that the “house” conditioning causes the agents to take more steps towards building a
house than other variants.
Thus, our results show that it is possible to train a somewhat steerable natural-language-conditioned
agent. However, its steerability is still too weak to be practically useful, and it is far from what we
believe could be accomplished with more research, data, and training compute. Another exciting
research direction is to have the model predict future text as well as just the next action.
34


VOYAGER: An Open-Ended Embodied Agent
with Large Language Models
Guanzhi Wang1 2 #, Yuqi Xie3, Yunfan Jiang4∗, Ajay Mandlekar1∗,
Chaowei Xiao1 5, Yuke Zhu1 3, Linxi “Jim” Fan1† #, Anima Anandkumar1 2†
1NVIDIA, 2Caltech, 3UT Austin, 4Stanford, 5UW Madison
∗Equal contribution
†Equal advising
# Corresponding authors
https://voyager.minedojo.org
Abstract
We introduce VOYAGER, the first LLM-powered embodied lifelong learning agent
in Minecraft that continuously explores the world, acquires diverse skills, and
makes novel discoveries without human intervention. VOYAGER consists of three
key components: 1) an automatic curriculum that maximizes exploration, 2) an
ever-growing skill library of executable code for storing and retrieving complex
behaviors, and 3) a new iterative prompting mechanism that incorporates environ-
ment feedback, execution errors, and self-verification for program improvement.
VOYAGER interacts with GPT-4 via blackbox queries, which bypasses the need for
model parameter fine-tuning. The skills developed by VOYAGER are temporally
extended, interpretable, and compositional, which compounds the agent’s abilities
rapidly and alleviates catastrophic forgetting.
Empirically, VOYAGER shows
strong in-context lifelong learning capability and exhibits exceptional proficiency
in playing Minecraft. It obtains 3.3× more unique items, travels 2.3× longer
distances, and unlocks key tech tree milestones up to 15.3× faster than prior SOTA.
VOYAGER is able to utilize the learned skill library in a new Minecraft world to
solve novel tasks from scratch, while other techniques struggle to generalize.
Figure 1: VOYAGER discovers new Minecraft items and skills continually by self-driven exploration,
significantly outperforming the baselines. X-axis denotes the number of prompting iterations.
1
arXiv:2305.16291v2  [cs.AI]  19 Oct 2023


Mine Wood  Log
Make Crafting Table
Craft Stone Sword
Craft Shield
Make Furnace
Cook Steak
Combat Zombie
     Mine Wood Log
Make Crafting Table
Combat 
Zombie
Mine Diamond
New 
Task
Code as 
Actions
Refine Program
Env Feedback
Execution Errors
Update 
Exploration 
Progress
Skill 
Retrieval
Add New Skill
Automatic Curriculum
Iterative Prompting Mechanism
Skill Library
Environment
Self-Verification
Figure 2: VOYAGER consists of three key components: an automatic curriculum for open-ended
exploration, a skill library for increasingly complex behaviors, and an iterative prompting mechanism
that uses code as action space.
1
Introduction
Building generally capable embodied agents that continuously explore, plan, and develop new skills
in open-ended worlds is a grand challenge for the AI community [1–5]. Classical approaches
employ reinforcement learning (RL) [6, 7] and imitation learning [8–10] that operate on primitive
actions, which could be challenging for systematic exploration [11–15], interpretability [16–18], and
generalization [19–21]. Recent advances in large language model (LLM) based agents harness the
world knowledge encapsulated in pre-trained LLMs to generate consistent action plans or executable
policies [16, 22, 19]. They are applied to embodied tasks like games and robotics [23–27], as well as
NLP tasks without embodiment [28–30]. However, these agents are not lifelong learners that can
progressively acquire, update, accumulate, and transfer knowledge over extended time spans [31, 32].
Let us consider Minecraft as an example. Unlike most other games studied in AI [33, 34, 10],
Minecraft does not impose a predefined end goal or a fixed storyline but rather provides a unique
playground with endless possibilities [23]. Minecraft requires players to explore vast, procedurally
generated 3D terrains and unlock a tech tree using gathered resources. Human players typically start
by learning the basics, such as mining wood and cooking food, before advancing to more complex
tasks like combating monsters and crafting diamond tools. We argue that an effective lifelong learning
agent should have similar capabilities as human players: (1) propose suitable tasks based on its
current skill level and world state, e.g., learn to harvest sand and cactus before iron if it finds itself in
a desert rather than a forest; (2) refine skills based on environmental feedback and commit mastered
skills to memory for future reuse in similar situations (e.g. fighting zombies is similar to fighting
spiders); (3) continually explore the world and seek out new tasks in a self-driven manner.
Towards these goals, we introduce VOYAGER, the first LLM-powered embodied lifelong learning
agent to drive exploration, master a wide range of skills, and make new discoveries continually
without human intervention in Minecraft. VOYAGER is made possible through three key modules
(Fig. 2): 1) an automatic curriculum that maximizes exploration; 2) a skill library for storing
and retrieving complex behaviors; and 3) a new iterative prompting mechanism that generates
executable code for embodied control. We opt to use code as the action space instead of low-level
motor commands because programs can naturally represent temporally extended and compositional
actions [16, 22], which are essential for many long-horizon tasks in Minecraft. VOYAGER interacts
with a blackbox LLM (GPT-4 [35]) through prompting and in-context learning [36–38]. Our approach
bypasses the need for model parameter access and explicit gradient-based training or finetuning.
More specifically, VOYAGER attempts to solve progressively harder tasks proposed by the automatic
curriculum, which takes into account the exploration progress and the agent’s state. The curriculum
is generated by GPT-4 based on the overarching goal of “discovering as many diverse things as
possible”. This approach can be perceived as an in-context form of novelty search [39, 40]. VOYAGER
incrementally builds a skill library by storing the action programs that help solve a task successfully.
2


Inventory (5/36): {'oak_planks': 3, 'stick': 
4, 'crafting_table': 1, 'stone': 3, 
'wooden_pickaxe': 1}
Biome: river
Inventory (4/36): {'sugar_cane': 3, 'paper': 
3, 'fishing_rod': 1, 'wooden_pickaxe': 1}
Nearby entities: pig, cat, villager
Health: 12/20
Hunger: 0/20
Inventory (6/36): {'furnace': 1, 
'stone_pickaxe': 1, 'oak_planks': 7, 
'crafting_table': 1, 'raw_iron': 4, 'coal': 1} 
Time: night
Nearby entities: zombie
Equipment: 'stone_sword', 'shield'
Reasoning: Since you have a wooden pickaxe and some stones, it would be 
beneficial to upgrade your pickaxe to a stone pickaxe for better efficiency.
Task: Craft 1 stone pickaxe.
Reasoning: Since you have a fishing rod in your inventory and you are near a river 
biome, it would be a good opportunity to catch some fish for food and experience. 
Task: Catch 1 fish.
Reasoning: Your hunger is at 0, which means you need to find food to replenish 
your hunger. Since there are pigs nearby, you can kill one to obtain raw porkchops.
Task: Kill 1 pig.
Reasoning: You have raw iron and coal, and you have a furnace. It's time to smelt 
the iron to make iron ingots, which can be used to craft better tools and armor.
Task: Smelt 4 raw iron.
Reasoning: Since it's night and there's a zombie nearby, it's a good opportunity to 
try killing the zombie now that you have a stone sword and a shield equipped.
Task: Kill 1 zombie.
GPT-4
GPT-4
GPT-4
GPT-4
GPT-4
Figure 3: Tasks proposed by the automatic curriculum. We only display the partial prompt for brevity.
See Appendix, Sec. A.3 for the full prompt structure.
Each program is indexed by the embedding of its description, which can be retrieved in similar
situations in the future. Complex skills can be synthesized by composing simpler programs, which
compounds VOYAGER’s capabilities rapidly over time and alleviates catastrophic forgetting in other
continual learning methods [31, 32].
However, LLMs struggle to produce the correct action code consistently in one shot [41]. To address
this challenge, we propose an iterative prompting mechanism that: (1) executes the generated
program to obtain observations from the Minecraft simulation (such as inventory listing and nearby
creatures) and error trace from the code interpreter (if any); (2) incorporates the feedback into GPT-4’s
prompt for another round of code refinement; and (3) repeats the process until a self-verification
module confirms the task completion, at which point we commit the program to the skill library (e.g.,
craftStoneShovel() and combatZombieWithSword()) and query the automatic curriculum for
the next milestone (Fig. 2).
Empirically, VOYAGER demonstrates strong in-context lifelong learning capabilities. It can construct
an ever-growing skill library of action programs that are reusable, interpretable, and generalizable
to novel tasks. We evaluate VOYAGER systematically against other LLM-based agent techniques
(e.g., ReAct [29], Reflexion [30], AutoGPT [28]) in MineDojo [23], an open-source Minecraft AI
framework. VOYAGER outperforms prior SOTA by obtaining 3.3× more unique items, unlocking key
tech tree milestones up to 15.3× faster, and traversing 2.3× longer distances. We further demonstrate
that VOYAGER is able to utilize the learned skill library in a new Minecraft world to solve novel tasks
from scratch, while other methods struggle to generalize.
2
Method
VOYAGER consists of three novel components: (1) an automatic curriculum (Sec. 2.1) that suggests
objectives for open-ended exploration, (2) a skill library (Sec. 2.2) for developing increasingly
complex behaviors, and (3) an iterative prompting mechanism (Sec. 2.3) that generates executable
code for embodied control. Full prompts are presented in Appendix, Sec. A.
2.1
Automatic Curriculum
Embodied agents encounter a variety of objectives with different complexity levels in open-ended
environments. An automatic curriculum offers numerous benefits for open-ended exploration, ensur-
ing a challenging but manageable learning process, fostering curiosity-driven intrinsic motivation
for agents to learn and explore, and encouraging the development of general and flexible problem-
solving strategies [42–44]. Our automatic curriculum capitalizes on the internet-scale knowledge
contained within GPT-4 by prompting it to provide a steady stream of new tasks or challenges. The
curriculum unfolds in a bottom-up fashion, allowing for considerable adaptability and responsiveness
to the exploration progress and the agent’s current state (Fig. 3). As VOYAGER progresses to harder
self-driven goals, it naturally learns a variety of skills, such as “mining a diamond”.
3


Program Description
Skill Library
Top-5 Relevant Skills
Program Generated by GPT-4
Task: Craft Iron Pickaxe
Key
Add
Retrieve
Value
Skill Library
Query
How to craft an iron pickaxe in 
Minecraft?
To craft an iron pickaxe, you 
need to 3 iron ingots and 2 
sticks. Once you have gathered 
the materials, ....
----------------------------------
         Environment Feedback
Mine Wood  Log
Make Crafting Table
Craft Wooden Pickaxe
Craft Stone Sword
Make Furnace
...
Combat Cow
Cook Steak
Craft Iron Axe
Combat Zombie
Smelt Iron Ingot
Craft Stick
Make Crafting Table
Make Furnace
Craft Wooden Pickaxe
GPT-3.5
Embedding
Embedding
GPT-3.5
Figure 4: Skill library. Top: Adding a new skill. Each time GPT-4 generates and verifies a new
skill, we add it to the skill library, represented by a vector database. The key is the embedding vector
of the program description (generated by GPT-3.5), while the value is the program itself. Bottom:
Skill retrieval. When faced with a new task proposed by the automatic curriculum, we first leverage
GPT-3.5 to generate a general suggestion for solving the task, which is combined with environment
feedback as the query context. Subsequently, we perform querying to identify the top-5 relevant skills.
The input prompt to GPT-4 consists of several components:
(1) Directives encouraging diverse behaviors and imposing constraints,
such as
“My ultimate goal is to discover as many diverse things as possible
...
The next task should not be too hard since I may not have the
necessary resources or have learned enough skills to complete it
yet.”;
(2) The agent’s current state, including inventory, equipment, nearby blocks and entities,
biome, time, health and hunger bars, and position;
(3) Previously completed and failed tasks, reflecting the agent’s current exploration progress
and capabilities frontier;
(4) Additional context: We also leverage GPT-3.5 to self-ask questions based on the agent’s
current state and exploration progress and self-answer questions. We opt to use GPT-3.5
instead of GPT-4 for standard NLP tasks due to budgetary considerations.
2.2
Skill Library
With the automatic curriculum consistently proposing increasingly complex tasks, it is essential to
have a skill library that serves as a basis for learning and evolution. Inspired by the generality, inter-
pretability, and universality of programs [45], we represent each skill with executable code that scaf-
folds temporally extended actions for completing a specific task proposed by the automatic curriculum.
The input prompt to GPT-4 consists of the following components:
(1) Guidelines
for
code
generation,
such
as
“Your function will be reused
for building more complex functions.
Therefore, you should make
it generic and reusable.”;
(2) Control primitive APIs, and relevant skills retrieved from the skill library, which are
crucial for in-context learning [36–38] to work well;
(3) The generated code from the last round, environment feedback, execution errors, and
critique, based on which GPT-4 can self-improve (Sec. 2.3);
(4) The agent’s current state, including inventory, equipment, nearby blocks and entities,
biome, time, health and hunger bars, and position;
4


I cannot make stick because I need:  2 more planks
I cannot make stone_shovel because I need:  2 more stick
throw new Error(`No item named ${name}`);
No item named acacia_axe
at line 18:await craftItem(bot, "acacia_axe", 1);
Environment Feedback
Execution Error
GPT-4
GPT-4
Figure 5: Left: Environment feedback. GPT-4 realizes it needs 2 more planks before crafting sticks.
Right: Execution error. GPT-4 realizes it should craft a wooden axe instead of an acacia axe since
there is no acacia axe in Minecraft. We only display the partial prompt for brevity. The full prompt
structure for code generation is in Appendix, Sec. A.4.
(5) Chain-of-thought prompting [46] to do reasoning before code generation.
We iteratively refine the program through a novel iterative prompting mechanism (Sec. 2.3), in-
corporate it into the skill library as a new skill, and index it by the embedding of its description
(Fig. 4, top). For skill retrieval, we query the skill library with the embedding of self-generated task
plans and environment feedback (Fig. 4, bottom). By continuously expanding and refining the skill
library, VOYAGER can learn, adapt, and excel in a wide spectrum of tasks, consistently pushing the
boundaries of its capabilities in the open world.
2.3
Iterative Prompting Mechanism
We introduce an iterative prompting mechanism for self-improvement through three types of feedback:
(1) Environment feedback, which illustrates the intermediate progress of program execution
(Fig. 5, left). For example, “I cannot make an iron chestplate because I need:
7 more iron ingots” highlights the cause of failure in crafting an iron chestplate. We use
bot.chat() inside control primitive APIs to generate environment feedback and prompt
GPT-4 to use this function as well during code generation;
(2) Execution errors from the program interpreter that reveal any invalid operations or syntax
errors in programs, which are valuable for bug fixing (Fig. 5, right);
(3) Self-verification for checking task success. Instead of manually coding success checkers
for each new task proposed by the automatic curriculum, we instantiate another GPT-4
agent for self-verification. By providing VOYAGER’s current state and the task to GPT-4,
we ask it to act as a critic [47–49] and inform us whether the program achieves the task.
In addition, if the task fails, it provides a critique by suggesting how to complete the task
(Fig. 6). Hence, our self-verification is more comprehensive than self-reflection [30] by both
checking success and reflecting on mistakes.
During each round of code generation, we execute the generated program to obtain environment
feedback and execution errors from the code interpreter, which are incorporated into GPT-4’s prompt
for the next round of code refinement. This iterative process repeats until self-verification validates
5


Inventory (8/36): {'oak_planks': 5, 'cobblestone': 2, 
'porkchop': 2, 'wooden_sword': 1, 'coal': 5, 'wooden_pickaxe': 
1, 'oak_log': 3, 'dirt': 9}
Task: Mine 5 coal ores
Inventory (10/36): {'raw_copper': 9, 'copper_ingot': 3, 
'acacia_planks': 1, 'raw_iron': 1, 'stick': 1, 'iron_sword': 1, 
'iron_pickaxe': 1, 'iron_ingot': 3, 'crafting_table': 1, 'furnace': 1}
Task: Craft a spyglass
Inventory (7/36): {'oak_log': 5, 'oak_planks': 1, 
'wooden_pickaxe': 1, 'wooden_sword': 1, 'porkchop': 2, 
'white_wool': 2, 'mutton': 6}
Task: Kill 3 sheep
Inventory (9/36): {'string': 4, 'coal': 1, 'rotten_flesh': 1, 
'iron_sword': 1, 'furnace': 1, 'dirt': 6, 'stone_shovel': 1, 
'wooden_pickaxe': 1, 'granite': 5}
Task: Kill 1 zombie
Reasoning: Mining coal_ore in Minecraft will get coal. You have 5 coal in your 
inventory.
Success: True
Reasoning: To craft a spyglass, you need 2 copper ingots and 1 amethyst shard. 
You have 3 copper ingots, but you don't have any amethyst shards.
Success: False
Critique: Find and mine an amethyst shard underground.
Reasoning: You have 2 white_wool and 6 mutton in your inventory, which indicates 
that you killed 2 sheep. You needed to kill 3 sheep.
Success: False
Critique: Find and kill one more sheep to complete the task.
Reasoning: You have 1 rotten_flesh in your inventory, which means you have killed 
at least 1 zombie.
Success: True
GPT-4
GPT-4
GPT-4
GPT-4
Figure 6: Self-verification examples. We only display the partial prompt for brevity. See Appendix,
Sec. A.5 for the full prompt structure.
the task’s completion, at which point we add this new skill to the skill library and ask the automatic
curriculum for a new objective (Fig. 2). If the agent gets stuck after 4 rounds of code generation, then
we query the curriculum for another task. This iterative prompting approach significantly improves
program synthesis for embodied control, enabling VOYAGER to continuously acquire diverse skills
without human intervention.
3
Experiments
3.1
Experimental Setup
We leverage OpenAI’s gpt-4-0314 [35] and gpt-3.5-turbo-0301 [50] APIs for text completion,
along with text-embedding-ada-002 [51] API for text embedding. We set all temperatures to
0 except for the automatic curriculum, which uses temperature = 0.1 to encourage task diversity. Our
simulation environment is built on top of MineDojo [23] and leverages Mineflayer [52] JavaScript
APIs for motor controls. See Appendix, Sec. B.1 for more details.
3.2
Baselines
Because there is no LLM-based agents that work out of the box for Minecraft, we make our best
effort to select a number of representative algorithms as baselines. These methods are originally
designed only for NLP tasks without embodiment, therefore we have to re-interpret them to be
executable in MineDojo and compatible with our experimental setting:
ReAct [29] uses chain-of-thought prompting [46] by generating both reasoning traces and action
plans with LLMs. We provide it with our environment feedback and the agent states as observations.
Reflexion [30] is built on top of ReAct [29] with self-reflection to infer more intuitive future actions.
We provide it with execution errors and our self-verification module.
AutoGPT [28] is a popular software tool that automates NLP tasks by decomposing a high-level
goal into multiple subgoals and executing them in a ReAct-style loop. We re-implement AutoGPT
by using GPT-4 to do task decomposition and provide it with the agent states, environment feedback,
and execution errors as observations for subgoal execution. Compared with VOYAGER, AutoGPT
lacks the skill library for accumulating knowledge, self-verification for assessing task success, and
automatic curriculum for open-ended exploration.
Note that we do not directly compare with prior methods that take Minecraft screen pixels as input
and output low-level controls [53–55]. It would not be an apple-to-apple comparison, because we rely
on the high-level Mineflayer [52] API to control the agent. Our work’s focus is on pushing the limits
of GPT-4 for lifelong embodied agent learning, rather than solving the 3D perception or sensorimotor
control problems. VOYAGER is orthogonal and can be combined with gradient-based approaches like
6


Table 1: Tech tree mastery. Fractions indicate the number of successful trials out of three total runs.
0/3 means the method fails to unlock a level of the tech tree within the maximal prompting iterations
(160). Numbers are prompting iterations averaged over three trials. The fewer the iterations, the
more efficient the method.
Method
Wooden Tool
Stone Tool
Iron Tool
Diamond Tool
ReAct [29]
N/A (0/
3)
N/A (0/
3)
N/A (0/
3)
N/A (0/
3)
Reflexion [30]
N/A (0/
3)
N/A (0/
3)
N/A (0/
3)
N/A (0/
3)
AutoGPT [28]
92 ± 72 (3/
3)
94 ± 72 (3/
3)
135 ± 103 (3/
3)
N/A (0/
3)
VOYAGER w/o Skill Library
7 ± 2 (3/
3)
9 ± 4 (3/
3)
29 ± 11 (3/
3)
N/A (0/
3)
VOYAGER (Ours)
6 ± 2 (3/
3)
11 ± 2 (3/
3)
21 ± 7 (3/
3)
102 (1/
3)
Figure 7: Map coverage: bird’s eye views of Minecraft maps. VOYAGER is able to traverse 2.3×
longer distances compared to baselines while crossing diverse terrains.
VPT [8] as long as the controller provides a code API. We make a system-level comparison between
VOYAGER and prior Minecraft agents in Table. A.2.
3.3
Evaluation Results
We systematically evaluate VOYAGER and baselines on their exploration performance, tech tree
mastery, map coverage, and zero-shot generalization capability to novel tasks in a new world.
Significantly better exploration.
Results of exploration performance are shown in Fig. 1.
VOYAGER’s superiority is evident in its ability to consistently make new strides, discovering 63
unique items within 160 prompting iterations, 3.3× many novel items compared to its counterparts.
On the other hand, AutoGPT lags considerably in discovering new items, while ReAct and Reflexion
struggle to make significant progress, given the abstract nature of the open-ended exploration goal
that is challenging to execute without an appropriate curriculum.
Consistent tech tree mastery. The Minecraft tech tree tests the agent’s ability to craft and use a
hierarchy of tools. Progressing through this tree (wooden tool →stone tool →iron tool →diamond
tool) requires the agent to master systematic and compositional skills. Compared with baselines,
VOYAGER unlocks the wooden level 15.3× faster (in terms of the prompting iterations), the stone
level 8.5× faster, the iron level 6.4× faster, and VOYAGER is the only one to unlock the diamond level
of the tech tree (Fig. 2 and Table. 1). This underscores the effectiveness of the automatic curriculum,
which consistently presents challenges of suitable complexity to facilitate the agent’s progress.
Extensive map traversal. VOYAGER is able to navigate distances 2.3× longer compared to baselines
by traversing a variety of terrains, while the baseline agents often find themselves confined to local
areas, which significantly hampers their capacity to discover new knowledge (Fig. 7).
7


Table 2: Zero-shot generalization to unseen tasks. Fractions indicate the number of successful
trials out of three total attempts. 0/3 means the method fails to solve the task within the maximal
prompting iterations (50). Numbers are prompting iterations averaged over three trials. The fewer
the iterations, the more efficient the method.
Method
Diamond Pickaxe
Golden Sword
Lava Bucket
Compass
ReAct [29]
N/A (0/
3)
N/A (0/
3)
N/A (0/
3)
N/A (0/
3)
Reflexion [30]
N/A (0/
3)
N/A (0/
3)
N/A (0/
3)
N/A (0/
3)
AutoGPT [28]
N/A (0/
3)
N/A (0/
3)
N/A (0/
3)
N/A (0/
3)
AutoGPT [28] w/ Our Skill Library
39 (1/
3)
30 (1/
3)
N/A (0/
3)
30 (2/
3)
VOYAGER w/o Skill Library
36 (2/
3)
30 ± 9 (3/
3)
27 ± 9 (3/
3)
26 ± 3 (3/
3)
VOYAGER (Ours)
19 ± 3 (3/
3)
18 ± 7 (3/
3)
21 ± 5 (3/
3)
18 ± 2 (3/
3)
Figure 8: Zero-shot generalization to unseen tasks. We visualize the intermediate progress of each
method on two tasks. See Appendix, Sec. B.4.3 for the other two tasks. We do not plot ReAct and
Reflexion since they do not make any meaningful progress.
Efficient zero-shot generalization to unseen tasks. To evaluate zero-shot generalization, we clear
the agent’s inventory, reset it to a newly instantiated world, and test it with unseen tasks. For both
VOYAGER and AutoGPT, we utilize GPT-4 to break down the task into a series of subgoals. Table. 2
and Fig. 8 show VOYAGER can consistently solve all the tasks, while baselines cannot solve any task
within 50 prompting iterations. What’s interesting to note is that our skill library constructed from
lifelong learning not only enhances VOYAGER’s performance but also gives a boost to AutoGPT.
This demonstrates that the skill library serves as a versatile tool that can be readily employed by other
methods, effectively acting as a plug-and-play asset to enhance performance.
3.4
Ablation Studies
We ablate 6 design choices (automatic curriculum, skill library, environment feedback, execution
errors, self-verification, and GPT-4 for code generation) in VOYAGER and study their impact on
exploration performance (see Appendix, Sec. B.3 for details of each ablated variant). Results are
shown in Fig. 9. We highlight the key findings below:
• Automatic curriculum is crucial for the agent’s consistent progress. The discovered item
count drops by 93% if the curriculum is replaced with a random one, because certain tasks
may be too challenging if attempted out of order. On the other hand, a manually designed
curriculum requires significant Minecraft-specific expertise, and does not take into account
the agent’s live situation. It falls short in the experimental results compared to our automatic
curriculum.
• VOYAGER w/o skill library exhibits a tendency to plateau in the later stages. This
underscores the pivotal role that the skill library plays in VOYAGER. It helps create more
complex actions and steadily pushes the agent’s boundaries by encouraging new skills to be
built upon older ones.
8


Figure 9: Left: Ablation studies for the automatic curriculum, skill library, and GPT-4. GPT-3.5
means replacing GPT-4 with GPT-3.5 for code generation. VOYAGER outperforms all the alternatives,
demonstrating the critical role of each component. Right: Ablation studies for the iterative
prompting mechanism. VOYAGER surpasses all the other options, thereby highlighting the essential
significance of each type of feedback in the iterative prompting mechanism.
Figure 10: VOYAGER builds 3D structures with human feedback. The progress of building designs
that integrate human input is demonstrated from left to right.
• Self-verification is the most important among all the feedback types. Removing the
module leads to a significant drop (−73%) in the discovered item count. Self-verification
serves as a critical mechanism to decide when to move on to a new task or reattempt a
previously unsuccessful task.
• GPT-4 significantly outperforms GPT-3.5 in code generation and obtains 5.7× more
unique items, as GPT-4 exhibits a quantum leap in coding abilities. This finding corroborates
recent studies in the literature [56, 57].
3.5
Multimodal Feedback from Humans
VOYAGER does not currently support visual perception, because the available version of GPT-4 API
is text-only at the time of this writing. However, VOYAGER has the potential to be augmented by
multimodal perception models [58, 59] to achieve more impressive tasks. We demonstrate that given
human feedback, VOYAGER is able to construct complex 3D structures in Minecraft, such as a Nether
Portal and a house (Fig. 10). There are two ways to integrate human feedback:
(1) Human as a critic (equivalent to VOYAGER’s self-verification module): humans provide
visual critique to VOYAGER, allowing it to modify the code from the previous round. This
feedback is essential for correcting certain errors in the spatial details of a 3D structure that
VOYAGER cannot perceive directly.
(2) Human as a curriculum (equivalent to VOYAGER’s automatic curriculum module): humans
break down a complex building task into smaller steps, guiding VOYAGER to complete them
incrementally. This approach improves VOYAGER’s ability to handle more sophisticated 3D
construction tasks.
9


4
Limitations and Future Work
Cost. The GPT-4 API incurs significant costs. It is 15× more expensive than GPT-3.5. Nevertheless,
VOYAGER requires the quantum leap in code generation quality from GPT-4 (Fig. 9), which GPT-3.5
and open-source LLMs cannot provide [60].
Inaccuracies. Despite the iterative prompting mechanism, there are still cases where the agent gets
stuck and fails to generate the correct skill. The automatic curriculum has the flexibility to reattempt
this task at a later time. Occasionally, self-verification module may also fail, such as not recognizing
spider string as a success signal of beating a spider.
Hallucinations. The automatic curriculum occasionally proposes unachievable tasks. For example, it
may ask the agent to craft a “copper sword" or “copper chestplate", which are items that do not exist
within the game. Hallucinations also occur during the code generation process. For instance, GPT-4
tends to use cobblestone as a fuel input, despite being an invalid fuel source in the game. Additionally,
it may call functions absent in the provided control primitive APIs, leading to code execution errors.
We are confident that improvements in the GPT API models as well as novel techniques for finetuning
open-source LLMs will overcome these limitations in the future.
5
Related work
Decision-making Agents in Minecraft.
Minecraft is an open-ended 3D world with incredibly
flexible game mechanics supporting a broad spectrum of activities. Built upon notable Minecraft
benchmarks [23, 61–65], Minecraft learning algorithms can be divided into two categories: 1)
Low-level controller: Many prior efforts leverage hierarchical reinforcement learning to learn from
human demonstrations [66–68]. Kanitscheider et al. [14] design a curriculum based on success rates,
but its objectives are limited to curated items. MineDojo [23] and VPT [8] utilize YouTube videos
for large-scale pre-training. DreamerV3 [69], on the other hand, learns a world model to explore
the environment and collect diamonds. 2) High-level planner: Volum et al. [70] leverage few-shot
prompting with Codex [41] to generate executable policies, but they require additional human
interaction. Recent works leverage LLMs as a high-level planner in Minecraft by decomposing
a high-level task into several subgoals following Minecraft recipes [55, 53, 71], thus lacking full
exploration flexibility. Like these latter works, VOYAGER also uses LLMs as a high-level planner by
prompting GPT-4 and utilizes Mineflayer [52] as a low-level controller following Volum et al. [70].
Unlike prior works, VOYAGER employs an automatic curriculum that unfolds in a bottom-up manner,
driven by curiosity, and therefore enables open-ended exploration.
Large Language Models for Agent Planning.
Inspired by the strong emergent capabilities of
LLMs, such as zero-shot prompting and complex reasoning [72, 37, 38, 36, 73, 74], embodied agent
research [75–78] has witnessed a significant increase in the utilization of LLMs for planning purposes.
Recent efforts can be roughly classified into two groups. 1) Large language models for robot
learning: Many prior works apply LLMs to generate subgoals for robot planning [27, 27, 25, 79, 80].
Inner Monologue [26] incorporates environment feedback for robot planning with LLMs. Code as
Policies [16] and ProgPrompt [22] directly leverage LLMs to generate executable robot policies.
VIMA [19] and PaLM-E [59] fine-tune pre-trained LLMs to support multimodal prompts. 2)
Large language models for text agents: ReAct [29] leverages chain-of-thought prompting [46] and
generates both reasoning traces and task-specific actions with LLMs. Reflexion [30] is built upon
ReAct [29] with self-reflection to enhance reasoning. AutoGPT [28] is a popular tool that automates
NLP tasks by crafting a curriculum of multiple subgoals for completing a high-level goal while
incorporating ReAct [29]’s reasoning and acting loops. DERA [81] frames a task as a dialogue
between two GPT-4 [35] agents. Generative Agents [82] leverages ChatGPT [50] to simulate human
behaviors by storing agents’ experiences as memories and retrieving those for planning, but its agent
actions are not executable. SPRING [83] is a concurrent work that uses GPT-4 to extract game
mechanics from game manuals, based on which it answers questions arranged in a directed acyclic
graph and predicts the next action. All these works lack a skill library for developing more complex
behaviors, which are crucial components for the success of VOYAGER in lifelong learning.
Code Generation with Execution.
Code generation has been a longstanding challenge in
NLP [41, 84, 85, 73, 37], with various works leveraging execution results to improve program
10


synthesis. Execution-guided approaches leverage intermediate execution outcomes to guide program
search [86–88]. Another line of research utilizes majority voting to choose candidates based on their
execution performance [89, 90]. Additionally, LEVER [91] trains a verifier to distinguish and reject
incorrect programs based on execution results. CLAIRIFY [92], on the other hand, generates code
for planning chemistry experiments and makes use of a rule-based verifier to iteratively provide
error feedback to LLMs. VOYAGER distinguishes itself from these works by integrating environment
feedback, execution errors, and self-verification (to assess task success) into an iterative prompting
mechanism for embodied control.
6
Conclusion
In this work, we introduce VOYAGER, the first LLM-powered embodied lifelong learning agent,
which leverages GPT-4 to explore the world continuously, develop increasingly sophisticated skills,
and make new discoveries consistently without human intervention. VOYAGER exhibits superior
performance in discovering novel items, unlocking the Minecraft tech tree, traversing diverse terrains,
and applying its learned skill library to unseen tasks in a newly instantiated world. VOYAGER serves
as a starting point to develop powerful generalist agents without tuning the model parameters.
7
Broader Impacts
Our research is conducted within Minecraft, a safe and harmless 3D video game environment. While
VOYAGER is designed to be generally applicable to other domains, such as robotics, its application to
physical robots would require additional attention and the implementation of safety constraints by
humans to ensure responsible and secure deployment.
8
Acknowledgements
We are extremely grateful to Ziming Zhu, Kaiyu Yang, Rafał Kocielnik, Colin White, Or Sharir, Sahin
Lale, De-An Huang, Jean Kossaifi, Yuncong Yang, Charles Zhang, Minchao Huang, and many other
colleagues and friends for their helpful feedback and insightful discussions. This work is done during
Guanzhi Wang’s internship at NVIDIA. Guanzhi Wang is supported by the Kortschak fellowship in
Computing and Mathematical Sciences at Caltech.
References
[1] Eric Kolve, Roozbeh Mottaghi, Winson Han, Eli VanderBilt, Luca Weihs, Alvaro Herrasti,
Daniel Gordon, Yuke Zhu, Abhinav Gupta, and Ali Farhadi. Ai2-thor: An interactive 3d
environment for visual ai. arXiv preprint arXiv: Arxiv-1712.05474, 2017.
[2] Manolis Savva, Jitendra Malik, Devi Parikh, Dhruv Batra, Abhishek Kadian, Oleksandr
Maksymets, Yili Zhao, Erik Wijmans, Bhavana Jain, Julian Straub, Jia Liu, and Vladlen
Koltun. Habitat: A platform for embodied AI research. In 2019 IEEE/CVF International
Conference on Computer Vision, ICCV 2019, Seoul, Korea (South), October 27 - November 2,
2019, pages 9338–9346. IEEE, 2019.
[3] Yuke Zhu, Josiah Wong, Ajay Mandlekar, and Roberto Martín-Martín. robosuite: A mod-
ular simulation framework and benchmark for robot learning. arXiv preprint arXiv: Arxiv-
2009.12293, 2020.
[4] Fei Xia, William B. Shen, Chengshu Li, Priya Kasimbeg, Micael Tchapmi, Alexander Toshev,
Li Fei-Fei, Roberto Martín-Martín, and Silvio Savarese. Interactive gibson benchmark (igibson
0.5): A benchmark for interactive navigation in cluttered environments. arXiv preprint arXiv:
Arxiv-1910.14442, 2019.
[5] Bokui Shen, Fei Xia, Chengshu Li, Roberto Martín-Martín, Linxi Fan, Guanzhi Wang, Claudia
Pérez-D’Arpino, Shyamal Buch, Sanjana Srivastava, Lyne P. Tchapmi, Micael E. Tchapmi, Kent
Vainio, Josiah Wong, Li Fei-Fei, and Silvio Savarese. igibson 1.0: a simulation environment for
interactive tasks in large realistic scenes. arXiv preprint arXiv: Arxiv-2012.02924, 2020.
11


[6] Jens Kober, J Andrew Bagnell, and Jan Peters. Reinforcement learning in robotics: A survey.
The International Journal of Robotics Research, 32(11):1238–1274, 2013.
[7] Kai Arulkumaran, Marc Peter Deisenroth, Miles Brundage, and Anil Anthony Bharath. Deep
reinforcement learning: A brief survey. IEEE Signal Processing Magazine, 34(6):26–38, 2017.
[8] Bowen Baker, Ilge Akkaya, Peter Zhokhov, Joost Huizinga, Jie Tang, Adrien Ecoffet, Brandon
Houghton, Raul Sampedro, and Jeff Clune. Video pretraining (vpt): Learning to act by watching
unlabeled online videos. arXiv preprint arXiv: Arxiv-2206.11795, 2022.
[9] DeepMind Interactive Agents Team, Josh Abramson, Arun Ahuja, Arthur Brussee, Federico
Carnevale, Mary Cassin, Felix Fischer, Petko Georgiev, Alex Goldin, Mansi Gupta, Tim
Harley, Felix Hill, Peter C Humphreys, Alden Hung, Jessica Landon, Timothy Lillicrap, Hamza
Merzic, Alistair Muldal, Adam Santoro, Guy Scully, Tamara von Glehn, Greg Wayne, Nathaniel
Wong, Chen Yan, and Rui Zhu. Creating multimodal interactive agents with imitation and
self-supervised learning. arXiv preprint arXiv: Arxiv-2112.03763, 2021.
[10] Oriol Vinyals, Igor Babuschkin, Junyoung Chung, Michael Mathieu, Max Jaderberg, Wo-
jciech M Czarnecki, Andrew Dudzik, Aja Huang, Petko Georgiev, Richard Powell, et al.
Alphastar: Mastering the real-time strategy game starcraft ii. DeepMind blog, 2, 2019.
[11] Adrien Ecoffet, Joost Huizinga, Joel Lehman, Kenneth O. Stanley, and Jeff Clune. Go-explore:
a new approach for hard-exploration problems. arXiv preprint arXiv: Arxiv-1901.10995, 2019.
[12] Joost Huizinga and Jeff Clune. Evolving multimodal robot behavior via many stepping stones
with the combinatorial multiobjective evolutionary algorithm.
Evolutionary computation,
30(2):131–164, 2022.
[13] Rui Wang, Joel Lehman, Aditya Rawal, Jiale Zhi, Yulun Li, Jeffrey Clune, and Kenneth O.
Stanley. Enhanced POET: open-ended reinforcement learning through unbounded invention of
learning challenges and their solutions. In Proceedings of the 37th International Conference on
Machine Learning, ICML 2020, 13-18 July 2020, Virtual Event, volume 119 of Proceedings of
Machine Learning Research, pages 9940–9951. PMLR, 2020.
[14] Ingmar Kanitscheider, Joost Huizinga, David Farhi, William Hebgen Guss, Brandon Houghton,
Raul Sampedro, Peter Zhokhov, Bowen Baker, Adrien Ecoffet, Jie Tang, Oleg Klimov, and Jeff
Clune. Multi-task curriculum learning in a complex, visual, hard-exploration domain: Minecraft.
arXiv preprint arXiv: Arxiv-2106.14876, 2021.
[15] Michael Dennis, Natasha Jaques, Eugene Vinitsky, Alexandre M. Bayen, Stuart Russell, Andrew
Critch, and Sergey Levine. Emergent complexity and zero-shot transfer via unsupervised
environment design. In Hugo Larochelle, Marc’Aurelio Ranzato, Raia Hadsell, Maria-Florina
Balcan, and Hsuan-Tien Lin, editors, Advances in Neural Information Processing Systems 33:
Annual Conference on Neural Information Processing Systems 2020, NeurIPS 2020, December
6-12, 2020, virtual, 2020.
[16] Jacky Liang, Wenlong Huang, Fei Xia, Peng Xu, Karol Hausman, Brian Ichter, Pete Florence,
and Andy Zeng. Code as policies: Language model programs for embodied control. arXiv
preprint arXiv: Arxiv-2209.07753, 2022.
[17] Shao-Hua Sun, Te-Lin Wu, and Joseph J. Lim. Program guided agent. In 8th International
Conference on Learning Representations, ICLR 2020, Addis Ababa, Ethiopia, April 26-30, 2020.
OpenReview.net, 2020.
[18] Zelin Zhao, Karan Samel, Binghong Chen, and Le Song. Proto: Program-guided transformer for
program-guided tasks. In Marc’Aurelio Ranzato, Alina Beygelzimer, Yann N. Dauphin, Percy
Liang, and Jennifer Wortman Vaughan, editors, Advances in Neural Information Processing
Systems 34: Annual Conference on Neural Information Processing Systems 2021, NeurIPS
2021, December 6-14, 2021, virtual, pages 17021–17036, 2021.
[19] Yunfan Jiang, Agrim Gupta, Zichen Zhang, Guanzhi Wang, Yongqiang Dou, Yanjun Chen,
Li Fei-Fei, Anima Anandkumar, Yuke Zhu, and Linxi (Jim) Fan. Vima: General robot manipu-
lation with multimodal prompts. ARXIV.ORG, 2022.
12


[20] Mohit Shridhar, Lucas Manuelli, and Dieter Fox. Cliport: What and where pathways for robotic
manipulation. arXiv preprint arXiv: Arxiv-2109.12098, 2021.
[21] Linxi Fan, Guanzhi Wang, De-An Huang, Zhiding Yu, Li Fei-Fei, Yuke Zhu, and Animashree
Anandkumar. SECANT: self-expert cloning for zero-shot generalization of visual policies. In
Marina Meila and Tong Zhang, editors, Proceedings of the 38th International Conference on
Machine Learning, ICML 2021, 18-24 July 2021, Virtual Event, volume 139 of Proceedings of
Machine Learning Research, pages 3088–3099. PMLR, 2021.
[22] Ishika Singh, Valts Blukis, Arsalan Mousavian, Ankit Goyal, Danfei Xu, Jonathan Tremblay,
Dieter Fox, Jesse Thomason, and Animesh Garg. Progprompt: Generating situated robot task
plans using large language models. arXiv preprint arXiv: Arxiv-2209.11302, 2022.
[23] Linxi Fan, Guanzhi Wang, Yunfan Jiang, Ajay Mandlekar, Yuncong Yang, Haoyi Zhu, Andrew
Tang, De-An Huang, Yuke Zhu, and Anima Anandkumar. Minedojo: Building open-ended
embodied agents with internet-scale knowledge. arXiv preprint arXiv: Arxiv-2206.08853, 2022.
[24] Andy Zeng, Adrian Wong, Stefan Welker, Krzysztof Choromanski, Federico Tombari, Aveek
Purohit, Michael Ryoo, Vikas Sindhwani, Johnny Lee, Vincent Vanhoucke, and Pete Florence.
Socratic models: Composing zero-shot multimodal reasoning with language. arXiv preprint
arXiv: Arxiv-2204.00598, 2022.
[25] Michael Ahn, Anthony Brohan, Noah Brown, Yevgen Chebotar, Omar Cortes, Byron David,
Chelsea Finn, Keerthana Gopalakrishnan, Karol Hausman, Alex Herzog, Daniel Ho, Jasmine
Hsu, Julian Ibarz, Brian Ichter, Alex Irpan, Eric Jang, Rosario Jauregui Ruano, Kyle Jeffrey,
Sally Jesmonth, Nikhil J Joshi, Ryan Julian, Dmitry Kalashnikov, Yuheng Kuang, Kuang-Huei
Lee, Sergey Levine, Yao Lu, Linda Luu, Carolina Parada, Peter Pastor, Jornell Quiambao,
Kanishka Rao, Jarek Rettinghouse, Diego Reyes, Pierre Sermanet, Nicolas Sievers, Clayton Tan,
Alexander Toshev, Vincent Vanhoucke, Fei Xia, Ted Xiao, Peng Xu, Sichun Xu, and Mengyuan
Yan. Do as i can, not as i say: Grounding language in robotic affordances. arXiv preprint arXiv:
Arxiv-2204.01691, 2022.
[26] Wenlong Huang, Fei Xia, Ted Xiao, Harris Chan, Jacky Liang, Pete Florence, Andy Zeng,
Jonathan Tompson, Igor Mordatch, Yevgen Chebotar, Pierre Sermanet, Noah Brown, Tomas
Jackson, Linda Luu, Sergey Levine, Karol Hausman, and Brian Ichter. Inner monologue:
Embodied reasoning through planning with language models. arXiv preprint arXiv: Arxiv-
2207.05608, 2022.
[27] Wenlong Huang, Pieter Abbeel, Deepak Pathak, and Igor Mordatch. Language models as zero-
shot planners: Extracting actionable knowledge for embodied agents. In Kamalika Chaudhuri,
Stefanie Jegelka, Le Song, Csaba Szepesvári, Gang Niu, and Sivan Sabato, editors, International
Conference on Machine Learning, ICML 2022, 17-23 July 2022, Baltimore, Maryland, USA,
volume 162 of Proceedings of Machine Learning Research, pages 9118–9147. PMLR, 2022.
[28] Significant-gravitas/auto-gpt: An experimental open-source attempt to make gpt-4 fully au-
tonomous., 2023.
[29] Shunyu Yao, Jeffrey Zhao, Dian Yu, Nan Du, Izhak Shafran, Karthik Narasimhan, and Yuan
Cao. React: Synergizing reasoning and acting in language models. arXiv preprint arXiv:
Arxiv-2210.03629, 2022.
[30] Noah Shinn, Beck Labash, and Ashwin Gopinath. Reflexion: an autonomous agent with
dynamic memory and self-reflection. arXiv preprint arXiv: Arxiv-2303.11366, 2023.
[31] German Ignacio Parisi, Ronald Kemker, Jose L. Part, Christopher Kanan, and Stefan Wermter.
Continual lifelong learning with neural networks: A review. Neural Networks, 113:54–71, 2019.
[32] Liyuan Wang, Xingxing Zhang, Hang Su, and Jun Zhu. A comprehensive survey of continual
learning: Theory, method and application. arXiv preprint arXiv: Arxiv-2302.00487, 2023.
[33] Volodymyr Mnih, Koray Kavukcuoglu, David Silver, Alex Graves, Ioannis Antonoglou, Daan
Wierstra, and Martin Riedmiller. Playing atari with deep reinforcement learning. arXiv preprint
arXiv: Arxiv-1312.5602, 2013.
13


[34] OpenAI, :, Christopher Berner, Greg Brockman, Brooke Chan, Vicki Cheung, Przemysław
D˛
ebiak, Christy Dennison, David Farhi, Quirin Fischer, Shariq Hashme, Chris Hesse, Rafal Józe-
fowicz, Scott Gray, Catherine Olsson, Jakub Pachocki, Michael Petrov, Henrique P. d. O. Pinto,
Jonathan Raiman, Tim Salimans, Jeremy Schlatter, Jonas Schneider, Szymon Sidor, Ilya
Sutskever, Jie Tang, Filip Wolski, and Susan Zhang. Dota 2 with large scale deep reinforcement
learning. arXiv preprint arXiv: Arxiv-1912.06680, 2019.
[35] OpenAI. Gpt-4 technical report. arXiv preprint arXiv: Arxiv-2303.08774, 2023.
[36] Jason Wei, Yi Tay, Rishi Bommasani, Colin Raffel, Barret Zoph, Sebastian Borgeaud, Dani
Yogatama, Maarten Bosma, Denny Zhou, Donald Metzler, Ed H. Chi, Tatsunori Hashimoto,
Oriol Vinyals, Percy Liang, Jeff Dean, and William Fedus. Emergent abilities of large language
models. arXiv preprint arXiv: Arxiv-2206.07682, 2022.
[37] Tom B. Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal,
Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel
Herbert-Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel M.
Ziegler, Jeffrey Wu, Clemens Winter, Christopher Hesse, Mark Chen, Eric Sigler, Mateusz
Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam McCandlish, Alec
Radford, Ilya Sutskever, and Dario Amodei. Language models are few-shot learners. In Hugo
Larochelle, Marc’Aurelio Ranzato, Raia Hadsell, Maria-Florina Balcan, and Hsuan-Tien Lin,
editors, Advances in Neural Information Processing Systems 33: Annual Conference on Neural
Information Processing Systems 2020, NeurIPS 2020, December 6-12, 2020, virtual, 2020.
[38] Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena,
Yanqi Zhou, Wei Li, and Peter J. Liu. Exploring the limits of transfer learning with a unified
text-to-text transformer. J. Mach. Learn. Res., 21:140:1–140:67, 2020.
[39] Benjamin Eysenbach, Abhishek Gupta, Julian Ibarz, and Sergey Levine. Diversity is all you
need: Learning skills without a reward function. In 7th International Conference on Learning
Representations, ICLR 2019, New Orleans, LA, USA, May 6-9, 2019. OpenReview.net, 2019.
[40] Edoardo Conti, Vashisht Madhavan, Felipe Petroski Such, Joel Lehman, Kenneth O. Stanley,
and Jeff Clune. Improving exploration in evolution strategies for deep reinforcement learning via
a population of novelty-seeking agents. In Samy Bengio, Hanna M. Wallach, Hugo Larochelle,
Kristen Grauman, Nicolò Cesa-Bianchi, and Roman Garnett, editors, Advances in Neural
Information Processing Systems 31: Annual Conference on Neural Information Processing
Systems 2018, NeurIPS 2018, December 3-8, 2018, Montréal, Canada, pages 5032–5043, 2018.
[41] Mark Chen, Jerry Tworek, Heewoo Jun, Qiming Yuan, Henrique Ponde de Oliveira Pinto,
Jared Kaplan, Harri Edwards, Yuri Burda, Nicholas Joseph, Greg Brockman, Alex Ray, Raul
Puri, Gretchen Krueger, Michael Petrov, Heidy Khlaaf, Girish Sastry, Pamela Mishkin, Brooke
Chan, Scott Gray, Nick Ryder, Mikhail Pavlov, Alethea Power, Lukasz Kaiser, Mohammad
Bavarian, Clemens Winter, Philippe Tillet, Felipe Petroski Such, Dave Cummings, Matthias
Plappert, Fotios Chantzis, Elizabeth Barnes, Ariel Herbert-Voss, William Hebgen Guss, Alex
Nichol, Alex Paino, Nikolas Tezak, Jie Tang, Igor Babuschkin, Suchir Balaji, Shantanu Jain,
William Saunders, Christopher Hesse, Andrew N. Carr, Jan Leike, Josh Achiam, Vedant Misra,
Evan Morikawa, Alec Radford, Matthew Knight, Miles Brundage, Mira Murati, Katie Mayer,
Peter Welinder, Bob McGrew, Dario Amodei, Sam McCandlish, Ilya Sutskever, and Wojciech
Zaremba. Evaluating large language models trained on code. arXiv preprint arXiv: Arxiv-
2107.03374, 2021.
[42] Rui Wang, Joel Lehman, Jeff Clune, and Kenneth O. Stanley. Paired open-ended trailblazer
(poet): Endlessly generating increasingly complex and diverse learning environments and their
solutions. arXiv preprint arXiv: Arxiv-1901.01753, 2019.
[43] Rémy Portelas, Cédric Colas, Lilian Weng, Katja Hofmann, and Pierre-Yves Oudeyer. Auto-
matic curriculum learning for deep RL: A short survey. In Christian Bessiere, editor, Proceedings
of the Twenty-Ninth International Joint Conference on Artificial Intelligence, IJCAI 2020, pages
4819–4825. ijcai.org, 2020.
14


[44] Sébastien Forestier, Rémy Portelas, Yoan Mollard, and Pierre-Yves Oudeyer. Intrinsically
motivated goal exploration processes with automatic curriculum learning. The Journal of
Machine Learning Research, 23(1):6818–6858, 2022.
[45] Kevin Ellis, Catherine Wong, Maxwell Nye, Mathias Sable-Meyer, Luc Cary, Lucas Morales,
Luke Hewitt, Armando Solar-Lezama, and Joshua B. Tenenbaum. Dreamcoder: Growing
generalizable, interpretable knowledge with wake-sleep bayesian program learning. arXiv
preprint arXiv: Arxiv-2006.08381, 2020.
[46] Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Ed Chi, Quoc Le, and Denny
Zhou. Chain of thought prompting elicits reasoning in large language models. arXiv preprint
arXiv: Arxiv-2201.11903, 2022.
[47] Volodymyr Mnih, Adrià Puigdomènech Badia, Mehdi Mirza, Alex Graves, Timothy P. Lillicrap,
Tim Harley, David Silver, and Koray Kavukcuoglu. Asynchronous methods for deep rein-
forcement learning. In Maria-Florina Balcan and Kilian Q. Weinberger, editors, Proceedings
of the 33nd International Conference on Machine Learning, ICML 2016, New York City, NY,
USA, June 19-24, 2016, volume 48 of JMLR Workshop and Conference Proceedings, pages
1928–1937. JMLR.org, 2016.
[48] John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal
policy optimization algorithms. arXiv preprint arXiv: Arxiv-1707.06347, 2017.
[49] Timothy P. Lillicrap, Jonathan J. Hunt, Alexander Pritzel, Nicolas Heess, Tom Erez, Yuval
Tassa, David Silver, and Daan Wierstra. Continuous control with deep reinforcement learning.
In Yoshua Bengio and Yann LeCun, editors, 4th International Conference on Learning Repre-
sentations, ICLR 2016, San Juan, Puerto Rico, May 2-4, 2016, Conference Track Proceedings,
2016.
[50] Introducing chatgpt, 2022.
[51] New and improved embedding model, 2022.
[52] PrismarineJS. Prismarinejs/mineflayer: Create minecraft bots with a powerful, stable, and high
level javascript api., 2013.
[53] Kolby Nottingham, Prithviraj Ammanabrolu, Alane Suhr, Yejin Choi, Hanna Hajishirzi, Sameer
Singh, and Roy Fox. Do embodied agents dream of pixelated sheep?: Embodied decision
making using language guided world modelling. ARXIV.ORG, 2023.
[54] Shaofei Cai, Zihao Wang, Xiaojian Ma, Anji Liu, and Yitao Liang. Open-world multi-task
control through goal-aware representation learning and adaptive horizon prediction. arXiv
preprint arXiv: Arxiv-2301.10034, 2023.
[55] Zihao Wang, Shaofei Cai, Anji Liu, Xiaojian Ma, and Yitao Liang. Describe, explain, plan and
select: Interactive planning with large language models enables open-world multi-task agents.
arXiv preprint arXiv: Arxiv-2302.01560, 2023.
[56] Sébastien Bubeck, Varun Chandrasekaran, Ronen Eldan, Johannes Gehrke, Eric Horvitz, Ece
Kamar, Peter Lee, Yin Tat Lee, Yuanzhi Li, Scott Lundberg, Harsha Nori, Hamid Palangi,
Marco Tulio Ribeiro, and Yi Zhang. Sparks of artificial general intelligence: Early experiments
with gpt-4. arXiv preprint arXiv: Arxiv-2303.12712, 2023.
[57] Yiheng Liu, Tianle Han, Siyuan Ma, Jiayue Zhang, Yuanyuan Yang, Jiaming Tian, Hao He,
Antong Li, Mengshen He, Zhengliang Liu, Zihao Wu, Dajiang Zhu, Xiang Li, Ning Qiang,
Dingang Shen, Tianming Liu, and Bao Ge. Summary of chatgpt/gpt-4 research and perspective
towards the future of large language models. arXiv preprint arXiv: Arxiv-2304.01852, 2023.
[58] Shikun Liu, Linxi Fan, Edward Johns, Zhiding Yu, Chaowei Xiao, and Anima Anandkumar.
Prismer: A vision-language model with an ensemble of experts. arXiv preprint arXiv: Arxiv-
2303.02506, 2023.
15


[59] Danny Driess, Fei Xia, Mehdi S. M. Sajjadi, Corey Lynch, Aakanksha Chowdhery, Brian Ichter,
Ayzaan Wahid, Jonathan Tompson, Quan Vuong, Tianhe Yu, Wenlong Huang, Yevgen Chebotar,
Pierre Sermanet, Daniel Duckworth, Sergey Levine, Vincent Vanhoucke, Karol Hausman, Marc
Toussaint, Klaus Greff, Andy Zeng, Igor Mordatch, and Pete Florence. Palm-e: An embodied
multimodal language model. arXiv preprint arXiv: Arxiv-2303.03378, 2023.
[60] Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timo-
thée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, Aurelien Rodriguez,
Armand Joulin, Edouard Grave, and Guillaume Lample. Llama: Open and efficient foundation
language models. arXiv preprint arXiv: Arxiv-2302.13971, 2023.
[61] William H. Guss, Brandon Houghton, Nicholay Topin, Phillip Wang, Cayden Codel, Manuela
Veloso, and Ruslan Salakhutdinov. Minerl: A large-scale dataset of minecraft demonstrations.
In Sarit Kraus, editor, Proceedings of the Twenty-Eighth International Joint Conference on
Artificial Intelligence, IJCAI 2019, Macao, China, August 10-16, 2019, pages 2442–2448.
ijcai.org, 2019.
[62] William H. Guss, Cayden Codel, Katja Hofmann, Brandon Houghton, Noboru Kuno, Stephanie
Milani, Sharada Mohanty, Diego Perez Liebana, Ruslan Salakhutdinov, Nicholay Topin,
Manuela Veloso, and Phillip Wang. The minerl 2019 competition on sample efficient re-
inforcement learning using human priors. arXiv preprint arXiv: Arxiv-1904.10079, 2019.
[63] William H. Guss, Mario Ynocente Castro, Sam Devlin, Brandon Houghton, Noboru Sean Kuno,
Crissman Loomis, Stephanie Milani, Sharada Mohanty, Keisuke Nakata, Ruslan Salakhutdinov,
John Schulman, Shinya Shiroshita, Nicholay Topin, Avinash Ummadisingu, and Oriol Vinyals.
The minerl 2020 competition on sample efficient reinforcement learning using human priors.
arXiv preprint arXiv: Arxiv-2101.11071, 2021.
[64] Anssi Kanervisto, Stephanie Milani, Karolis Ramanauskas, Nicholay Topin, Zichuan Lin, Jun-
you Li, Jianing Shi, Deheng Ye, Qiang Fu, Wei Yang, Weijun Hong, Zhongyue Huang, Haicheng
Chen, Guangjun Zeng, Yue Lin, Vincent Micheli, Eloi Alonso, François Fleuret, Alexander
Nikulin, Yury Belousov, Oleg Svidchenko, and Aleksei Shpilman. Minerl diamond 2021
competition: Overview, results, and lessons learned. arXiv preprint arXiv: Arxiv-2202.10583,
2022.
[65] Matthew Johnson, Katja Hofmann, Tim Hutton, and David Bignell. The malmo platform for
artificial intelligence experimentation. In Subbarao Kambhampati, editor, Proceedings of the
Twenty-Fifth International Joint Conference on Artificial Intelligence, IJCAI 2016, New York,
NY, USA, 9-15 July 2016, pages 4246–4247. IJCAI/AAAI Press, 2016.
[66] Zichuan Lin, Junyou Li, Jianing Shi, Deheng Ye, Qiang Fu, and Wei Yang. Juewu-mc: Playing
minecraft with sample-efficient hierarchical reinforcement learning. arXiv preprint arXiv:
Arxiv-2112.04907, 2021.
[67] Hangyu Mao, Chao Wang, Xiaotian Hao, Yihuan Mao, Yiming Lu, Chengjie Wu, Jianye
Hao, Dong Li, and Pingzhong Tang. Seihai: A sample-efficient hierarchical ai for the minerl
competition. arXiv preprint arXiv: Arxiv-2111.08857, 2021.
[68] Alexey Skrynnik, Aleksey Staroverov, Ermek Aitygulov, Kirill Aksenov, Vasilii Davydov, and
Aleksandr I. Panov. Hierarchical deep q-network from imperfect demonstrations in minecraft.
Cogn. Syst. Res., 65:74–78, 2021.
[69] Danijar Hafner, Jurgis Pasukonis, Jimmy Ba, and Timothy Lillicrap. Mastering diverse domains
through world models. arXiv preprint arXiv: Arxiv-2301.04104, 2023.
[70] Ryan Volum, Sudha Rao, Michael Xu, Gabriel DesGarennes, Chris Brockett, Benjamin
Van Durme, Olivia Deng, Akanksha Malhotra, and Bill Dolan. Craft an iron sword: Dy-
namically generating interactive game characters by prompting large language models tuned on
code. In Proceedings of the 3rd Wordplay: When Language Meets Games Workshop (Wordplay
2022), pages 25–43, Seattle, United States, 2022. Association for Computational Linguistics.
[71] Haoqi Yuan, Chi Zhang, Hongcheng Wang, Feiyang Xie, Penglin Cai, Hao Dong, and Zongqing
Lu. Plan4mc: Skill reinforcement learning and planning for open-world minecraft tasks. arXiv
preprint arXiv: 2303.16563, 2023.
16


[72] Rishi Bommasani, Drew A. Hudson, Ehsan Adeli, Russ Altman, Simran Arora, Sydney von Arx,
Michael S. Bernstein, Jeannette Bohg, Antoine Bosselut, Emma Brunskill, Erik Brynjolfsson,
Shyamal Buch, Dallas Card, Rodrigo Castellon, Niladri Chatterji, Annie Chen, Kathleen Creel,
Jared Quincy Davis, Dora Demszky, Chris Donahue, Moussa Doumbouya, Esin Durmus, Stefano
Ermon, John Etchemendy, Kawin Ethayarajh, Li Fei-Fei, Chelsea Finn, Trevor Gale, Lauren
Gillespie, Karan Goel, Noah Goodman, Shelby Grossman, Neel Guha, Tatsunori Hashimoto,
Peter Henderson, John Hewitt, Daniel E. Ho, Jenny Hong, Kyle Hsu, Jing Huang, Thomas
Icard, Saahil Jain, Dan Jurafsky, Pratyusha Kalluri, Siddharth Karamcheti, Geoff Keeling,
Fereshte Khani, Omar Khattab, Pang Wei Koh, Mark Krass, Ranjay Krishna, Rohith Kuditipudi,
Ananya Kumar, Faisal Ladhak, Mina Lee, Tony Lee, Jure Leskovec, Isabelle Levent, Xiang Lisa
Li, Xuechen Li, Tengyu Ma, Ali Malik, Christopher D. Manning, Suvir Mirchandani, Eric
Mitchell, Zanele Munyikwa, Suraj Nair, Avanika Narayan, Deepak Narayanan, Ben Newman,
Allen Nie, Juan Carlos Niebles, Hamed Nilforoshan, Julian Nyarko, Giray Ogut, Laurel Orr,
Isabel Papadimitriou, Joon Sung Park, Chris Piech, Eva Portelance, Christopher Potts, Aditi
Raghunathan, Rob Reich, Hongyu Ren, Frieda Rong, Yusuf Roohani, Camilo Ruiz, Jack
Ryan, Christopher Ré, Dorsa Sadigh, Shiori Sagawa, Keshav Santhanam, Andy Shih, Krishnan
Srinivasan, Alex Tamkin, Rohan Taori, Armin W. Thomas, Florian Tramèr, Rose E. Wang,
William Wang, Bohan Wu, Jiajun Wu, Yuhuai Wu, Sang Michael Xie, Michihiro Yasunaga,
Jiaxuan You, Matei Zaharia, Michael Zhang, Tianyi Zhang, Xikun Zhang, Yuhui Zhang, Lucia
Zheng, Kaitlyn Zhou, and Percy Liang. On the opportunities and risks of foundation models.
arXiv preprint arXiv: Arxiv-2108.07258, 2021.
[73] Aakanksha Chowdhery, Sharan Narang, Jacob Devlin, Maarten Bosma, Gaurav Mishra, Adam
Roberts, Paul Barham, Hyung Won Chung, Charles Sutton, Sebastian Gehrmann, Parker
Schuh, Kensen Shi, Sasha Tsvyashchenko, Joshua Maynez, Abhishek Rao, Parker Barnes,
Yi Tay, Noam Shazeer, Vinodkumar Prabhakaran, Emily Reif, Nan Du, Ben Hutchinson,
Reiner Pope, James Bradbury, Jacob Austin, Michael Isard, Guy Gur-Ari, Pengcheng Yin,
Toju Duke, Anselm Levskaya, Sanjay Ghemawat, Sunipa Dev, Henryk Michalewski, Xavier
Garcia, Vedant Misra, Kevin Robinson, Liam Fedus, Denny Zhou, Daphne Ippolito, David
Luan, Hyeontaek Lim, Barret Zoph, Alexander Spiridonov, Ryan Sepassi, David Dohan, Shivani
Agrawal, Mark Omernick, Andrew M. Dai, Thanumalayan Sankaranarayana Pillai, Marie Pellat,
Aitor Lewkowycz, Erica Moreira, Rewon Child, Oleksandr Polozov, Katherine Lee, Zongwei
Zhou, Xuezhi Wang, Brennan Saeta, Mark Diaz, Orhan Firat, Michele Catasta, Jason Wei,
Kathy Meier-Hellstern, Douglas Eck, Jeff Dean, Slav Petrov, and Noah Fiedel. Palm: Scaling
language modeling with pathways. arXiv preprint arXiv: Arxiv-2204.02311, 2022.
[74] Hyung Won Chung, Le Hou, Shayne Longpre, Barret Zoph, Yi Tay, William Fedus, Eric Li,
Xuezhi Wang, Mostafa Dehghani, Siddhartha Brahma, Albert Webson, Shixiang Shane Gu,
Zhuyun Dai, Mirac Suzgun, Xinyun Chen, Aakanksha Chowdhery, Sharan Narang, Gaurav
Mishra, Adams Yu, Vincent Zhao, Yanping Huang, Andrew Dai, Hongkun Yu, Slav Petrov,
Ed H. Chi, Jeff Dean, Jacob Devlin, Adam Roberts, Denny Zhou, Quoc V. Le, and Jason Wei.
Scaling instruction-finetuned language models. arXiv preprint arXiv: Arxiv-2210.11416, 2022.
[75] Jiafei Duan, Samson Yu, Hui Li Tan, Hongyuan Zhu, and Cheston Tan. A survey of embodied
AI: from simulators to research tasks. IEEE Trans. Emerg. Top. Comput. Intell., 6(2):230–244,
2022.
[76] Dhruv Batra, Angel X. Chang, Sonia Chernova, Andrew J. Davison, Jia Deng, Vladlen Koltun,
Sergey Levine, Jitendra Malik, Igor Mordatch, Roozbeh Mottaghi, Manolis Savva, and Hao Su.
Rearrangement: A challenge for embodied ai. arXiv preprint arXiv: Arxiv-2011.01975, 2020.
[77] Harish Ravichandar, Athanasios S Polydoros, Sonia Chernova, and Aude Billard. Recent
advances in robot learning from demonstration.
Annual review of control, robotics, and
autonomous systems, 3:297–330, 2020.
[78] Jack Collins, Shelvin Chand, Anthony Vanderkop, and David Howard. A review of physics
simulators for robotic applications. IEEE Access, 9:51416–51431, 2021.
[79] So Yeon Min, Devendra Singh Chaplot, Pradeep Ravikumar, Yonatan Bisk, and R. Salakhutdi-
nov. Film: Following instructions in language with modular methods. International Conference
on Learning Representations, 2021.
17


[80] Valts Blukis, Chris Paxton, Dieter Fox, Animesh Garg, and Yoav Artzi. A persistent spatial
semantic representation for high-level natural language instruction execution. In 5th Annual
Conference on Robot Learning, 2021.
[81] Varun Nair, Elliot Schumacher, Geoffrey Tso, and Anitha Kannan. Dera: Enhancing large
language model completions with dialog-enabled resolving agents. arXiv preprint arXiv:
Arxiv-2303.17071, 2023.
[82] Joon Sung Park, Joseph C. O’Brien, Carrie J. Cai, Meredith Ringel Morris, Percy Liang, and
Michael S. Bernstein. Generative agents: Interactive simulacra of human behavior. arXiv
preprint arXiv: Arxiv-2304.03442, 2023.
[83] Yue Wu, Shrimai Prabhumoye, So Yeon Min, Yonatan Bisk, Ruslan Salakhutdinov, Amos
Azaria, Tom Mitchell, and Yuanzhi Li. Spring: Gpt-4 out-performs rl algorithms by studying
papers and reasoning. arXiv preprint arXiv: 2305.15486, 2023.
[84] Erik Nijkamp, Bo Pang, Hiroaki Hayashi, Lifu Tu, Huan Wang, Yingbo Zhou, Silvio Savarese,
and Caiming Xiong. A conversational paradigm for program synthesis. arXiv preprint arXiv:
Arxiv-2203.13474, 2022.
[85] Hung Le, Yue Wang, Akhilesh Deepak Gotmare, Silvio Savarese, and Steven C. H. Hoi. Coderl:
Mastering code generation through pretrained models and deep reinforcement learning. arXiv
preprint arXiv: Arxiv-2207.01780, 2022.
[86] Xinyun Chen, Chang Liu, and Dawn Song. Execution-guided neural program synthesis. In 7th
International Conference on Learning Representations, ICLR 2019, New Orleans, LA, USA,
May 6-9, 2019. OpenReview.net, 2019.
[87] Xinyun Chen, Dawn Song, and Yuandong Tian. Latent execution for neural program synthesis.
arXiv preprint arXiv: Arxiv-2107.00101, 2021.
[88] Kevin Ellis, Maxwell I. Nye, Yewen Pu, Felix Sosa, Josh Tenenbaum, and Armando Solar-
Lezama. Write, execute, assess: Program synthesis with a REPL. In Hanna M. Wallach, Hugo
Larochelle, Alina Beygelzimer, Florence d’Alché-Buc, Emily B. Fox, and Roman Garnett,
editors, Advances in Neural Information Processing Systems 32: Annual Conference on Neural
Information Processing Systems 2019, NeurIPS 2019, December 8-14, 2019, Vancouver, BC,
Canada, pages 9165–9174, 2019.
[89] Yujia Li, David Choi, Junyoung Chung, Nate Kushman, Julian Schrittwieser, Rémi Leblond,
Tom Eccles, James Keeling, Felix Gimeno, Agustin Dal Lago, Thomas Hubert, Peter Choy,
Cyprien de Masson d’Autume, Igor Babuschkin, Xinyun Chen, Po-Sen Huang, Johannes Welbl,
Sven Gowal, Alexey Cherepanov, James Molloy, Daniel J. Mankowitz, Esme Sutherland Robson,
Pushmeet Kohli, Nando de Freitas, Koray Kavukcuoglu, and Oriol Vinyals. Competition-level
code generation with alphacode. arXiv preprint arXiv: Arxiv-2203.07814, 2022.
[90] Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser,
Matthias Plappert, Jerry Tworek, Jacob Hilton, Reiichiro Nakano, Christopher Hesse, and
John Schulman. Training verifiers to solve math word problems. arXiv preprint arXiv: Arxiv-
2110.14168, 2021.
[91] Ansong Ni, Srini Iyer, Dragomir Radev, Ves Stoyanov, Wen tau Yih, Sida I. Wang, and
Xi Victoria Lin. Lever: Learning to verify language-to-code generation with execution. arXiv
preprint arXiv: Arxiv-2302.08468, 2023.
[92] Marta Skreta, Naruki Yoshikawa, Sebastian Arellano-Rubach, Zhi Ji, Lasse Bjørn Kristensen,
Kourosh Darvish, Alán Aspuru-Guzik, Florian Shkurti, and Animesh Garg. Errors are useful
prompts: Instruction guided task programming with verifier-assisted iterative prompting. arXiv
preprint arXiv: Arxiv-2303.14100, 2023.
18


A
Method
A.1
VOYAGER Algorithm
Pseudocode 1: VOYAGER algorithm.
def
voyager(
environment ,
# environment
that uses code as action
space
curriculum_agent ,
# curriculum
agent for
proposing
the next task
action_agent ,
# action
agent for code
generation
critic_agent ,
# critic
agent for self -verification
skill_manager ,
# skill
manager
for adding new skills and skill
retrieval
):
agent_state = environment.reset ()
while
True:
exploration_progress = (
curriculum_agent . get_exploration_progress (
curriculum_agent . get_completed_tasks (),
curriculum_agent . get_failed_tasks (),
)
)
task = curriculum_agent . propose_next_task (
agent_state , exploration_progress
)
code = None
environment_feedback = None
execution_errors = None
critique = None
success = False
# try at most 4 rounds
before
moving on to the next task
for i in range (4):
skills = skill_manager . retrieve_skills (
task , environment_feedback
)
code = action_agent. generate_code (
task ,
code ,
environment_feedback ,
execution_errors ,
critique ,
skills ,
)
(
agent_state ,
environment_feedback ,
execution_errors ,
) = environment.step(code)
success , critique = critic_agent. check_task_success (
task , agent_state
)
if success:
break
if success:
skill_manager.add_skill(code)
curriculum_agent . add_completed_task (task)
else:
curriculum_agent . add_failed_task (task)
A.2
Prompting
GPT-4 and GPT-3.5 offer users the ability to designate the role of each prompt message among three
options:
19


• System: A high-level instruction that guides the model behavior throughout the conversation.
It sets the overall tone and objective for the interaction.
• User: A detailed instruction that guides the assistant for the next immediate response.
• Assistant: A response message generated the model.
See https://platform.openai.com/docs/guides/chat/introduction for more details.
To save token usage, instead of engaging in multi-round conversations, we concatenate a system
prompt and a user prompt to obtain each assistant’s response.
A.3
Automatic Curriculum
A.3.1
Components in the Prompt
The input prompt to GPT-4 consists of several components:
(1) Directives encouraging diverse behaviors and imposing constraints (so that the proposed
task is achievable and verifiable): See Sec. A.3.4 for the full prompt;
(2) The agent’s current state:
• Inventory: A dictionary of items with counts, for example, {‘cobblestone’: 4, ‘furnace’:
1, ‘stone_pickaxe’: 1, ‘oak_planks’: 7, ‘dirt’: 6, ‘wooden_pickaxe’: 1, ‘crafting_table’:
1, ‘raw_iron’: 4, ‘coal’: 1};
• Equipment: Armors or weapons equipped by the agents;
• Nearby blocks: A set of block names within a 32-block distance to the agent, for
example, ‘dirt’, ‘water’, ‘spruce_planks’, ‘grass_block’, ‘dirt_path’, ‘sugar_cane’,
‘fern’;
• Other blocks that are recently seen: Blocks that are not nearby or in the inventory;
• Nearby entities: A set of entity names within a 32-block distance to the agent, for
example, ‘pig’, ‘cat’, ‘villager’, ‘zombie’;
• A list of chests that are seen by the agent: Chests are external containers where the
agent can deposit items. If a chest is not opened before, its content is “Unknown”.
Otherwise, the items inside each chest are shown to the agent.
• Biome: For example, ‘plains’, ‘flower_forest’, ‘meadow’, ‘river’, ‘beach’, ‘for-
est’, ‘snowy_slopes’, ‘frozen_peaks’, ‘old_growth_birch_forest’, ‘ocean’, ‘sun-
flower_plains’, ‘stony_shore’;
• Time: One of ‘sunrise’, ‘day’, ‘noon’, ‘sunset’, ‘night’, ‘midnight’;
• Health and hunger bars: The max value is 20;
• Position: 3D coordinate (x, y, z) of the agent’s position in the Minecraft world;
(3) Previously completed and failed tasks;
(4) Additional context: See Sec. A.3.2;
(5) Chain-of-thought prompting [46] in response: We request GPT-4 to first reason about the
current progress and then suggest the next task.
A.3.2
Additional Context
We leverage GPT-3.5 to self-ask questions to provide additional context. Each question is paired with
a concept that is used for retrieving the most relevant document from the wiki knowledge base [23].
We feed the document content to GPT-3.5 for self-answering questions. In practice, using a wiki
knowledge base is optional since GPT-3.5 already possesses a good understanding of Minecraft
game mechanics. However, the external knowledge base becomes advantageous if GPT-3.5 is not
pre-trained in that specific domain. See Sec. A.3.4 for the full prompt.
A.3.3
Warm-up Schedule
In practice, we adopt a warm-up schedule to gradually incorporate the agent’s state and the additional
context into the prompt based on how many tasks the agent has completed. This ensures that the
prompt is exposed to increasing amounts of information over the exploration progress and therefore
20


begins with basic skills and progressively advances towards more intricate and diverse ones. The
warm-up setting that we use across all the experiments is shown in Table. A.1.
Table A.1: Warm-up schedule for automatic curriculum.
Information in the prompt
After how many tasks are completed
core inventory (only including log, planks, stick,
crafting table, furnace, dirt, coal, pickaxe, sword,
and axe)
0
equipment
0
nearby blocks
0
position
0
nearby entities
5
full inventory
7
other blocks that are recently seen
10
biome
10
health bar
15
hunger bar
15
time
15
additional context
15
A.3.4
Full Prompt
Prompt 1: Full system prompt for automatic curriculum. The list of question-answer pairs represents
the additional context.
You are a helpful
assistant
that
tells me the next
immediate
task to
do in Minecraft. My ultimate
goal is to discover as many
diverse
things as possible , accomplish as many
diverse
tasks as possible
and become the best
Minecraft
player in the world.
I will give you the
following
information:
Question 1: ...
Answer: ...
Question 2: ...
Answer: ...
Question 3: ...
Answer: ...
...
Biome: ...
Time: ...
Nearby
blocks: ...
Other
blocks
that are
recently
seen: ...
Nearby
entities (nearest to farthest): ...
Health: Higher
than 15 means I’m healthy.
Hunger: Higher
than 15 means I’m not hungry.
Position: ...
Equipment: If I have
better
armor in my inventory , you should ask me
to equip it.
Inventory (xx /36): ...
Chests: You can ask me to deposit or take
items
from
these
chests.
There
also
might be some
unknown chest , you should ask me to open
and check
items
inside the
unknown
chest.
Completed
tasks so far: ...
Failed
tasks
that are too hard: ...
You must
follow the
following
criteria:
1) You should act as a mentor and guide me to the next task
based on
my current
learning
progress.
2) Please be very
specific
about
what
resources I need to collect ,
what I need to craft , or what mobs I need to kill.
21


3) The next task
should
follow a concise
format , such as "Mine [
quantity] [block ]", "Craft [quantity] [item]", "Smelt [quantity] [
item]", "Kill [quantity] [mob]", "Cook [quantity] [food]", "Equip
[item ]" etc. It should be a single
phrase. Do not
propose
multiple
tasks at the same time. Do not
mention
anything
else.
4) The next task
should not be too hard
since I may not have the
necessary
resources or have
learned
enough
skills to complete it
yet.
5) The next task
should be novel and
interesting. I should
look for
rare
resources , upgrade my equipment
and tools
using
better
materials , and
discover
new things. I should not be doing the same
thing
over and over
again.
6) I may
sometimes
need to repeat
some
tasks if I need to collect
more
resources to complete
more
difficult
tasks. Only
repeat
tasks if
necessary.
7) Do not ask me to build or dig
shelter
even if it ’s at night. I want
to explore
the world and
discover
new things. I don ’t want to
stay in one place.
8) Tasks
that
require
information
beyond the player ’s status to verify
should be avoided. For instance , "Placing 4 torches" and "Dig a 2
x1x2 hole" are not ideal
since
they
require
visual
confirmation
from the screen. All the placing , building , planting , and
trading
tasks
should be avoided. Do not
propose
task
starting
with
these
keywords.
You should
only
respond in the format as described
below:
RESPONSE
FORMAT:
Reasoning: Based on the
information I listed above , do reasoning
about
what the next task
should be.
Task: The next task.
Here ’s an example
response:
Reasoning: The
inventory is empty now , chop down a tree to get some
wood.
Task: Obtain a wood log.
Prompt 2: Full system prompt for asking questions. We provide both good and bad examples as
few-shot exemplars.
You are a helpful
assistant
that asks
questions to help me decide the
next
immediate
task to do in Minecraft. My ultimate
goal is to
discover as many
things as possible , accomplish as many
tasks as
possible
and become the best
Minecraft
player in the world.
I will give you the
following
information:
Biome: ...
Time: ...
Nearby
blocks: ...
Other
blocks
that are
recently
seen: ...
Nearby
entities (nearest to farthest): ...
Health: ...
Hunger: ...
Position: ...
Equipment: ...
Inventory (xx /36): ...
Chests: ...
Completed
tasks so far: ...
Failed
tasks
that are too hard: ...
You must
follow the
following
criteria:
1) You should ask at least 5 questions (but no more than 10 questions)
to help me decide the next
immediate
task to do. Each
question
should be followed by the
concept
that the
question is about.
2) Your
question
should be specific to a concept in Minecraft.
Bad
example (the
question is too
general):
22


Question: What is the best way to play
Minecraft?
Concept: unknown
Bad
example (axe is still general , you should
specify
the type of
axe such as wooden axe):
What are the
benefits of using an axe to gather
resources?
Concept: axe
Good
example:
Question: How to make a wooden
pickaxe?
Concept: wooden
pickaxe
3) Your
questions
should be self -contained
and not
require
any
context
.
Bad
example (the
question
requires
the
context of my current
biome):
Question: What are the blocks
that I can find in my current
biome?
Concept: unknown
Bad
example (the
question
requires
the
context of my current
inventory):
Question: What are the
resources
you need the most
currently?
Concept: unknown
Bad
example (the
question
requires
the
context of my current
inventory):
Question: Do you have any gold or emerald
resources?
Concept: gold
Bad
example (the
question
requires
the
context of my nearby
entities
):
Question: Can you see any
animals
nearby
that you can kill for
food?
Concept: food
Bad
example (the
question
requires
the
context of my nearby
blocks):
Question: Is there any water
source
nearby?
Concept: water
Good
example:
Question: What are the blocks
that I can find in the sparse
jungle
?
Concept: sparse
jungle
4) Do not ask
questions
about
building
tasks (such as building a
shelter) since
they are too hard for me to do.
Let ’s say your
current
biome is sparse
jungle. You can ask
questions
like:
Question: What are the items
that I can find in the sparse
jungle?
Concept: sparse
jungle
Question: What are the mobs that I can find in the sparse
jungle?
Concept: sparse
jungle
Let ’s say you see a creeper
nearby , and you have not
defeated a
creeper
before. You can ask a question
like:
Question: How to defeat the
creeper?
Concept: creeper
Let ’s say you last
completed
task is "Craft a wooden
pickaxe ". You can
ask a question
like:
Question: What are the
suggested
tasks
that I can do after
crafting a
wooden
pickaxe?
Concept: wooden
pickaxe
Here are some more
question
and
concept
examples:
Question: What are the ores that I can find in the sparse
jungle?
Concept: sparse
jungle
(the above
concept
should not be "ore" because I need to look up the
page of "sparse
jungle" to find out what ores I can find in the
sparse
jungle)
Question: How can you obtain
food in the sparse
jungle?
Concept: sparse
jungle
23


(the above
concept
should not be "food" because I need to look up the
page of "sparse
jungle" to find out what food I can obtain in the
sparse
jungle)
Question: How can you use the
furnace to upgrade
your
equipment
and
make
useful
items?
Concept: furnace
Question: How to obtain a diamond
ore?
Concept: diamond
ore
Question: What are the
benefits of using a stone
pickaxe
over a wooden
pickaxe?
Concept: stone
pickaxe
Question: What are the tools
that you can craft
using
wood
planks and
sticks?
Concept: wood
planks
You should
only
respond in the format as described
below:
RESPONSE
FORMAT:
Reasoning: ...
Question 1: ...
Concept 1: ...
Question 2: ...
Concept 2: ...
Question 3: ...
Concept 3: ...
Question 4: ...
Concept 4: ...
Question 5: ...
Concept 5: ...
...
Prompt 3: Full system prompt for answering questions. Context represents the optional content from
a wiki knowledge base.
You are a helpful
assistant
that
answer my question
about
Minecraft.
I will give you the
following
information:
Question: ...
You will
answer the
question
based on the
context (only if available
and
helpful) and your own
knowledge of Minecraft.
1) Start
your
answer
with "Answer: ".
2) Answer "Answer: Unknown" if you don ’t know the answer.
A.4
Skill Library
A.4.1
Components in the Prompt
The input prompt to GPT-4 consists of the following components:
(1) Guidelines for code generation: See Sec A.4.2 for the full prompt;
(2) Control primitive APIs implemented by us: These APIs serve a dual purpose: they demon-
strate the usage of Mineflayer APIs, and they can be directly called by GPT-4.
• exploreUntil(bot, direction, maxTime = 60, callback): Allow the agent
to explore in a fixed direction for maxTime. The callback is the stopping condition
implemented by the agent to determine when to stop exploring;
• mineBlock(bot, name, count = 1): Mine and collect the specified number of
blocks within a 32-block distance;
• craftItem(bot, name, count = 1): Craft the item with a crafting table nearby;
• placeItem(bot, name, position): Place the block at the specified position;
• smeltItem(bot, itemName, fuelName, count = 1): Smelt the item with the
specified fuel. There must be a furnace nearby;
24


• killMob(bot, mobName, timeout = 300):
Attack the mob and collect its
dropped item;
• getItemFromChest(bot, chestPosition, itemsToGet): Move to the chest at
the specified position and get items from the chest;
• depositItemIntoChest(bot, chestPosition, itemsToDeposit):
Move to
the chest at the specified position and deposit items into the chest;
(3) Control primitive APIs provided by Mineflayer:
• await bot.pathfinder.goto(goal): Go to a specific position. See below for how
to set the goal;
• new GoalNear(x, y, z, range): Move the bot to a block within the specified
range of the specified block;
• new GoalXZ(x, z): For long-range goals that don’t have a specific Y level;
• new GoalGetToBlock(x, y, z): Not get into the block, but get directly adjacent
to it. Useful for fishing, farming, filling a bucket, and using a bed.;
• new GoalFollow(entity, range): Follow the specified entity within the specified
range;
• new GoalPlaceBlock(position, bot.world, {}): Position the bot in order to
place a block;
• new GoalLookAtBlock(position, bot.world, {}): Path towards a position
where a face of the block at position is visible;
• bot.isABed(bedBlock): Return true if bedBlock is a bed;
• bot.blockAt(position): Return the block at position;
• await bot.equip(item, destination): Equip the item in the specified destina-
tion. destination must be one of “hand”, “head”, “torso”, “legs”, “feet”, “off-hand”;
• await bot.consume(): Consume the item in the bot’s hand. You must equip the
item to consume first. Useful for eating food, drinking potions, etc.;
• await bot.fish(): Let bot fish. Before calling this function, you must first get to a
water block and then equip a fishing rod. The bot will automatically stop fishing when
it catches a fish;
• await bot.sleep(bedBlock): Sleep until sunrise. You must get to a bed block
first;
• await bot.activateBlock(block): This is the same as right-clicking a block in
the game. Useful for buttons, doors, etc. You must get to the block first;
• await bot.lookAt(position): Look at the specified position. You must go near
the position before you look at it. To fill a bucket with water, you must look at it first;
• await bot.activateItem(): This is the same as right-clicking to use the item in
the bot’s hand. Useful for using a bucket, etc. You must equip the item to activate first;
• await bot.useOn(entity): This is the same as right-clicking an entity in the game.
Useful for shearing a sheep. You must get to the entity first;
(4) Retrieved skills from the skill library;
(5) Generated code from the last round;
(6) Environment feedback: The chat log in the prompt;
(7) Execution errors;
(8) Critique from the self-verification module;
(9) The agent’s current state: See Sec. A.3.1 for each element of the agent’s state;
(10) Task proposed by the automatic curriculum;
(11) Task context: We prompt GPT-3.5 to ask for general suggestions about how to solve the
task. In practice, this part is handled by the automatic curriculum since it has a systematic
mechanism for question-answering (Sec. A.3.2);
(12) Chain-of-thought prompting [46] in response: We ask GPT-4 to first explain the reason why
the code from the last round fails, then give step-by-step plans to finish the task, and finally
generate code. See Sec. A.4.2 for the full prompt.
25


A.4.2
Full Prompt
Prompt 4: Full system prompt for code generation.
You are a helpful
assistant
that
writes
Mineflayer
javascript
code to
complete
any
Minecraft
task
specified by me.
Here are some
useful
programs
written
with
Mineflayer
APIs.
/*
Explore
until
find an iron_ore , use Vec3(0, -1, 0) because
iron ores
are
usually
underground
await
exploreUntil(bot , new Vec3(0, -1, 0), 60, () => {
const
iron_ore = bot.findBlock ({
matching: mcData.blocksByName [" iron_ore "].id ,
maxDistance: 32,
});
return
iron_ore;
});
Explore
until
find a pig , use Vec3(1, 0, 1) because
pigs are
usually
on the
surface
let pig = await
exploreUntil(bot , new Vec3(1, 0, 1), 60, () => {
const pig = bot.nearestEntity (( entity) => {
return (
entity.name === "pig" &&
entity.position.distanceTo(bot.entity.position) < 32
);
});
return pig;
});
*/
async
function
exploreUntil (bot , direction , maxTime = 60, callback) {
/*
Implementation of this
function is omitted.
direction: Vec3 , can only
contain
value of
-1, 0 or 1
maxTime: number , the max time for
exploration
callback: function , early
stop
condition , will be called
each
second , exploration
will stop if return
value is not null
Return: null if explore
timeout , otherwise
return the return
value
of callback
*/
}
// Mine 3 cobblestone: mineBlock(bot , "stone", 3);
async
function
mineBlock(bot , name , count = 1) {
const
blocks = bot.findBlocks ({
matching: (block) => {
return
block.name === name;
},
maxDistance: 32,
count: count ,
});
const
targets = [];
for (let i = 0; i < Math.min(blocks.length , count); i++) {
targets.push(bot.blockAt(blocks[i]));
}
await bot.collectBlock.collect(targets , { ignoreNoPath: true });
}
// Craft 8 oak_planks
from 2 oak_log (do the recipe 2 times):
craftItem(bot , "oak_planks", 2);
26


// You must
place a crafting
table
before
calling
this
function
async
function
craftItem(bot , name , count = 1) {
const
item = mcData.itemsByName[name ];
const
craftingTable = bot.findBlock ({
matching: mcData.blocksByName . crafting_table .id ,
maxDistance: 32,
});
await bot.pathfinder.goto(
new
GoalLookAtBlock ( craftingTable .position , bot.world)
);
const
recipe = bot.recipesFor(item.id , null , 1, craftingTable)[0];
await bot.craft(recipe , count , craftingTable );
}
// Place a crafting_table
near the player , Vec3(1, 0, 0) is just an
example , you shouldn ’t always use that: placeItem(bot , "
crafting_table ", bot.entity.position.offset (1, 0, 0));
async
function
placeItem(bot , name , position) {
const
item = bot.inventory. findInventoryItem (mcData.itemsByName[
name ].id);
// find a reference
block
const
faceVectors = [
new Vec3(0, 1, 0),
new Vec3(0, -1, 0),
new Vec3(1, 0, 0),
new Vec3(-1, 0, 0),
new Vec3(0, 0, 1),
new Vec3(0, 0,
-1),
];
let
referenceBlock = null;
let
faceVector = null;
for (const
vector of faceVectors) {
const
block = bot.blockAt(position.minus(vector));
if (block ?. name !== "air") {
referenceBlock = block;
faceVector = vector;
break;
}
}
// You must
first go to the block
position
you want to place
await bot.pathfinder.goto(new
GoalPlaceBlock (position , bot.world ,
{}));
// You must
equip the item
right
before
calling
placeBlock
await bot.equip(item , "hand ");
await bot.placeBlock(referenceBlock , faceVector);
}
// Smelt 1 raw_iron
into 1 iron_ingot
using 1 oak_planks as fuel:
smeltItem(bot , "raw_iron", "oak_planks ");
// You must
place a furnace
before
calling
this
function
async
function
smeltItem(bot , itemName , fuelName , count = 1) {
const
item = mcData.itemsByName[itemName ];
const
fuel = mcData.itemsByName[fuelName ];
const
furnaceBlock = bot.findBlock ({
matching: mcData.blocksByName .furnace.id ,
maxDistance: 32,
});
await bot.pathfinder.goto(
new
GoalLookAtBlock (furnaceBlock.position , bot.world)
);
const
furnace = await bot.openFurnace(furnaceBlock);
for (let i = 0; i < count; i++) {
await
furnace.putFuel(fuel.id , null , 1);
27


await
furnace.putInput(item.id , null , 1);
// Wait 12 seconds
for the
furnace to smelt the item
await bot.waitForTicks (12 * 20);
await
furnace.takeOutput ();
}
await
furnace.close ();
}
// Kill a pig and
collect
the
dropped
item: killMob(bot , "pig", 300);
async
function
killMob(bot , mobName , timeout = 300) {
const
entity = bot.nearestEntity (
(entity) =>
entity.name ===
mobName &&
entity.position.distanceTo(bot.entity.position) < 32
);
await bot.pvp.attack(entity);
await bot.pathfinder.goto(
new
GoalBlock(entity.position.x, entity.position.y, entity.
position.z)
);
}
// Get a torch
from
chest at (30, 65, 100): getItemFromChest (bot , new
Vec3 (30, 65, 100) , {" torch ": 1});
// This
function
will work no matter how far the bot is from the chest
.
async
function
getItemFromChest (bot , chestPosition , itemsToGet) {
await
moveToChest(bot , chestPosition );
const
chestBlock = bot.blockAt( chestPosition );
const
chest = await bot.openContainer (chestBlock);
for (const
name in itemsToGet) {
const
itemByName = mcData.itemsByName[name ];
const
item = chest. findContainerItem (itemByName.id);
await
chest.withdraw(item.type , null , itemsToGet[name ]);
}
await
closeChest(bot , chestBlock);
}
// Deposit a torch
into
chest at (30, 65, 100): depositItemIntoChest (
bot , new Vec3 (30, 65, 100) , {" torch ": 1});
// This
function
will work no matter how far the bot is from the chest
.
async
function
depositItemIntoChest (bot , chestPosition , itemsToDeposit
) {
await
moveToChest(bot , chestPosition );
const
chestBlock = bot.blockAt( chestPosition );
const
chest = await bot.openContainer (chestBlock);
for (const
name in itemsToDeposit ) {
const
itemByName = mcData.itemsByName[name ];
const
item = bot.inventory. findInventoryItem (itemByName.id);
await
chest.deposit(item.type , null , itemsToDeposit [name ]);
}
await
closeChest(bot , chestBlock);
}
// Check the items
inside the chest at (30, 65, 100):
checkItemInsideChest (bot , new Vec3 (30, 65, 100));
// You only need to call this
function
once
without
any action to
finish
task of checking
items
inside the chest.
async
function
checkItemInsideChest (bot , chestPosition ) {
await
moveToChest(bot , chestPosition );
const
chestBlock = bot.blockAt( chestPosition );
await bot.openContainer (chestBlock);
// You must
close the chest
after
opening it if you are asked to
open a chest
28


await
closeChest(bot , chestBlock);
}
await bot.pathfinder.goto(goal); // A very
useful
function. This
function
may change
your main -hand
equipment.
// Following
are some
Goals you can use:
new
GoalNear(x, y, z, range); // Move the bot to a block
within the
specified
range of the
specified
block. ‘x‘, ‘y‘, ‘z‘, and ‘range ‘
are ‘number ‘
new GoalXZ(x, z); // Useful for long -range
goals
that don ’t have a
specific Y level. ‘x‘ and ‘z‘ are ‘number ‘
new
GoalGetToBlock (x, y, z); // Not get into the block , but get
directly
adjacent to it. Useful for fishing , farming , filling
bucket , and beds. ‘x‘, ‘y‘, and ‘z‘ are ‘number ‘
new
GoalFollow(entity , range); // Follow the
specified
entity
within
the
specified
range. ‘entity ‘ is ‘Entity ‘, ‘range ‘ is ‘number ‘
new
GoalPlaceBlock (position , bot.world , {}); // Position
the bot in
order to place a block. ‘position ‘ is ‘Vec3 ‘
new
GoalLookAtBlock (position , bot.world , {}); // Path into a position
where a blockface of the block at position is visible. ‘position ‘
is ‘Vec3 ‘
// These are other
Mineflayer
functions
you can use:
bot.isABed(bedBlock); // Return
true if ‘bedBlock ‘ is a bed
bot.blockAt(position); // Return the block at ‘position ‘. ‘position ‘
is ‘Vec3 ‘
// These are other
Mineflayer
async
functions
you can use:
await bot.equip(item , destination); // Equip the item in the
specified
destination. ‘item ‘ is ‘Item ‘, ‘destination ‘ can only be "hand",
"head", "torso", "legs", "feet", "off -hand"
await bot.consume (); // Consume
the item in the bot ’s hand. You must
equip the item to consume
first. Useful for eating food , drinking
potions , etc.
await bot.fish (); // Let bot fish. Before
calling
this function , you
must
first get to a water
block and then
equip a fishing
rod. The
bot will
automatically
stop
fishing
when it catches a fish
await bot.sleep(bedBlock); // Sleep
until
sunrise. You must get to a
bed block
first
await bot.activateBlock(block); // This is the same as right -clicking
a block in the game. Useful for buttons , doors , using hoes , etc.
You must get to the block
first
await bot.lookAt(position); // Look at the
specified
position. You
must go near the
position
before you look at it. To fill
bucket
with water , you must
lookAt
first. ‘position ‘ is ‘Vec3 ‘
await bot.activateItem (); // This is the same as right -clicking to use
the item in the bot ’s hand. Useful for using buckets , etc. You
must
equip the item to activate
first
await bot.useOn(entity); // This is the same as right -clicking an
entity in the game. Useful for
shearing sheep , equipping
harnesses
, etc. You must get to the entity
first
{ retrieved_skills }
At each
round of conversation , I will give you
Code from the last
round: ...
Execution
error: ...
Chat log: ...
Biome: ...
Time: ...
Nearby
blocks: ...
Nearby
entities (nearest to farthest):
Health: ...
29


Hunger: ...
Position: ...
Equipment: ...
Inventory (xx /36): ...
Chests: ...
Task: ...
Context: ...
Critique: ...
You should
then
respond to me with
Explain (if applicable): Are there any steps
missing in your plan? Why
does the code not
complete
the task? What does the chat log and
execution
error
imply?
Plan: How to complete
the task step by step. You should pay
attention
to Inventory
since it tells
what you have. The task
completeness
check is also
based on your
final
inventory.
Code:
1) Write an async
function
taking the bot as the only
argument.
2) Reuse the above
useful
programs as much as possible.
- Use ‘mineBlock(bot , name , count)‘ to collect
blocks. Do not
use ‘bot.dig ‘ directly.
- Use ‘craftItem(bot , name , count)‘ to craft
items. Do not use
‘bot.craft ‘ directly.
- Use ‘smeltItem(bot , name
count)‘ to smelt
items. Do not use
‘bot.openFurnace ‘ directly.
- Use ‘placeItem(bot , name , position)‘ to place
blocks. Do not
use ‘bot.placeBlock ‘ directly.
- Use ‘killMob(bot , name , timeout)‘ to kill mobs. Do not use ‘
bot.attack ‘ directly.
3) Your
function
will be reused for
building
more
complex
functions. Therefore , you should
make it generic
and
reusable. You
should not make
strong
assumption
about the
inventory (as it may
be changed at a later
time), and
therefore
you should
always
check
whether
you have the
required
items
before
using
them. If not ,
you should
first
collect
the
required
items and reuse the above
useful
programs.
4) Functions in the "Code from the last
round" section
will not be
saved or executed. Do not reuse
functions
listed
there.
5) Anything
defined
outside a function
will be ignored , define all
your
variables
inside
your
functions.
6) Call ‘bot.chat ‘ to show the
intermediate
progress.
7) Use ‘exploreUntil(bot , direction , maxDistance , callback)‘ when
you cannot
find
something. You should
frequently
call this
before
mining
blocks or killing
mobs. You should
select a direction at
random
every
time
instead of constantly
using (1, 0, 1).
8) ‘maxDistance ‘ should
always be 32 for ‘bot.findBlocks ‘ and ‘bot
.findBlock ‘. Do not cheat.
9) Do not write
infinite
loops or recursive
functions.
10) Do not use ‘bot.on ‘ or ‘bot.once ‘ to register
event
listeners.
You
definitely do not need them.
11) Name your
function in a meaningful
way (can infer the task
from the name).
You should
only
respond in the format as described
below:
RESPONSE
FORMAT:
Explain: ...
Plan:
1) ...
2) ...
3) ...
...
Code:
‘‘‘javascript
// helper
functions (only if needed , try to avoid
them)
...
30


// main
function
after the helper
functions
async
function
yourMainFunctionName (bot) {
// ...
}
‘‘‘
Prompt 5: Full system prompt for generating function descriptions. This is used when adding a new
skill to the skill library. We give a one-shot example in the prompt.
You are a helpful
assistant
that
writes a description of the given
function
written in Mineflayer
javascript
code.
1) Do not
mention
the
function
name.
2) Do not
mention
anything
about ‘bot.chat ‘ or helper
functions.
3) There
might be some
helper
functions
before the main function , but
you only need to describe
the main
function.
4) Try to summarize
the
function in no more than 6 sentences.
5) Your
response
should be a single
line of text.
For example , if the
function is:
async
function
mineCobblestone (bot) {
// Check if the wooden
pickaxe is in the inventory , if not , craft
one
let
woodenPickaxe = bot.inventory. findInventoryItem (mcData.
itemsByName [" wooden_pickaxe "].id);
if (! woodenPickaxe) {
bot.chat (" Crafting a wooden
pickaxe .");
await
craftWoodenPickaxe (bot);
woodenPickaxe = bot.inventory. findInventoryItem (mcData.itemsByName
[" wooden_pickaxe "].id);
}
// Equip the wooden
pickaxe if it exists
if (woodenPickaxe) {
await bot.equip(woodenPickaxe , "hand ");
// Explore
until we find a stone
block
await
exploreUntil(bot , new Vec3(1, -1, 1), 60, () => {
const
stone = bot.findBlock ({
matching: mcData.blocksByName [" stone "].id ,
maxDistance: 32
});
if (stone) {
return
true;
}
});
// Mine 8 cobblestone
blocks
using the wooden
pickaxe
bot.chat (" Found a stone
block. Mining 8 cobblestone
blocks .");
await
mineBlock(bot , "stone", 8);
bot.chat (" Successfully
mined 8 cobblestone
blocks .");
// Save the event of mining 8 cobblestone
bot.save (" cobblestone_mined ");
} else {
bot.chat (" Failed to craft a wooden
pickaxe. Cannot
mine
cobblestone .");
}
}
The main
function is ‘mineCobblestone ‘.
Then you would
write:
31


The
function is about
mining 8 cobblestones
using a wooden
pickaxe.
First
check if a wooden
pickaxe is in the
inventory. If not , craft
one. If the wooden
pickaxe is available , equip the wooden
pickaxe
in the hand. Next , explore
the
environment
until
finding a stone
block. Once a stone
block is found , mine a total of 8 cobblestone
blocks
using the wooden
pickaxe.
A.4.3
Examples
Skill library example 1: craftWoodenPlanks.
async
function
craftWoodenPlanks (bot) {
const
logNames = ["oak_log", "birch_log", "spruce_log", "jungle_log"
, "acacia_log", "dark_oak_log ", "mangrove_log"];
const
plankNames = ["oak_planks", "birch_planks", " spruce_planks ", "
jungle_planks", "acacia_planks", " dark_oak_planks ", "
mangrove_planks "];
const
logInInventory = logNames.find(logName => bot.inventory.count(
mcData.itemsByName[logName ].id) > 0);
if (! logInInventory) {
bot.chat("No wooden log in inventory. Mining a wooden log ...");
await
mineWoodLog(bot);
}
const
logIndex = logNames.indexOf( logInInventory );
const
plankName = plankNames[logIndex ];
bot.chat(‘Crafting 4 ${plankName }... ‘);
await
craftItem(bot , plankName , 1);
bot.chat(‘4 ${plankName} crafted .‘);
}
Skill library example 2: mineTenCobbledDeepslateBelowY0.
async
function
mineTenCobbledDeepslateBelowY0 (bot) {
// Equip the iron
pickaxe
const
ironPickaxe = bot.inventory. findInventoryItem (mcData.
itemsByName["iron_pickaxe"].id);
await bot.equip(ironPickaxe , "hand");
// Find
cobbled_deepslate
blocks
below Y=0
const
cobbledDeepslateBlocks = await
exploreUntil(bot , new Vec3(1,
-1, 1), 60, () => {
const
cobbledDeepslate = bot.findBlock ({
matching: mcData.blocksByName [" cobbled_deepslate "].id ,
maxDistance: 32,
position: pos => pos.y < 0
});
return
cobbledDeepslate ;
});
// Mine 10 cobbled_deepslate
blocks
await
mineBlock(bot , " cobbled_deepslate ", 10);
bot.chat("10 cobbled_deepslate
mined
below Y=0.");
}
Skill library example 3: smeltFiveRawIronV2.
async
function
findSuitablePosition (bot) {
const
offsets = [new Vec3(1, 0, 0), new Vec3(-1, 0, 0), new Vec3(0,
0, 1), new Vec3(0, 0,
-1)];
for (const
offset of offsets) {
const
position = bot.entity.position.offset(offset.x, offset.y,
offset.z);
const
block = bot.blockAt(position);
32


if (block.name === "air") {
return
position;
}
}
return
null;
}
async
function
smeltFiveRawIron (bot) {
// Check if there is a furnace in the
inventory
const
furnaceItem = bot.inventory. findInventoryItem (mcData.
itemsByName.furnace.id);
// If not , craft a furnace
using the
available
cobblestone
if (! furnaceItem) {
await
craftFurnace(bot);
}
// Find a suitable
position to place the
furnace
const
furnacePosition = await
findSuitablePosition (bot);
if (! furnacePosition) {
bot.chat("Could not find a suitable
position to place the
furnace.
");
return;
}
// Place the
furnace at the
suitable
position
await
placeItem(bot , "furnace", furnacePosition );
// Smelt 5 raw iron
using the
available
coal as fuel
await
smeltItem(bot , "raw_iron", "coal", 5);
bot.chat("5 raw iron
smelted.");
}
Skill library example 4: fillBucketWithWater.
async
function
fillBucketWithWater (bot) {
// Find a water
block
nearby
const
waterBlock = await
exploreUntil (bot , new Vec3(1, 0, 1), 60, ()
=> {
const
water = bot.findBlock ({
matching: mcData.blocksByName .water.id ,
maxDistance: 32
});
return
water;
});
if (! waterBlock) {
bot.chat("Could not find
water.");
return;
}
// Go to the water
block
const
adjacentBlock= waterBlock.position.offset (0, 1, 0);
await bot.pathfinder.goto(
new
GoalGetToBlock (adjacentBlock .x, adjacentBlock .y,
adjacentBlock.z)
);
// Look at the water
block
await bot.lookAt(waterBlock.position , true);
// Equip the bucket
const
bucket = bot.inventory. findInventoryItem (mcData.itemsByName.
bucket.id);
await bot.equip(bucket , "hand");
33


// Activate
the bucket to collect
water
await bot.activateItem ();
bot.chat("Filled the bucket
with
water.");
}
Skill library example 5: catchFiveFishSafely.
async
function
catchFiveFishSafely (bot) {
// Check if the bot has a fishing
rod in its
inventory
let
fishingRod = bot.inventory. findInventoryItem (mcData.itemsByName.
fishing_rod.id);
if (! fishingRod) {
await
craftFishingRod (bot);
fishingRod = bot.inventory. findInventoryItem (mcData.itemsByName.
fishing_rod.id);
}
// Find a nearby
water
block
let
waterBlock;
while (! waterBlock) {
waterBlock = await
exploreUntil(bot , new Vec3(1, 0, 1), 60, () =>
{
const
foundWaterBlock = bot.findBlock ({
matching: mcData.blocksByName .water.id ,
maxDistance: 32
});
return
foundWaterBlock ;
});
if (! waterBlock) {
bot.chat("No path to the water
block. Trying to find
another
water
block ...");
}
}
// Move to a block
adjacent to the water
block
const
adjacentBlock = waterBlock.position.offset (0, 1, 0);
await bot.pathfinder.goto(new
GoalBlock( adjacentBlock .x,
adjacentBlock.y, adjacentBlock.z));
// Look at the water
block
await bot.lookAt(waterBlock.position);
// Equip the
fishing
rod
await bot.equip(fishingRod , "hand");
// Fish in the water 5 times
for (let i = 0; i < 5; i++) {
try {
await bot.fish ();
bot.chat(‘Fish ${i + 1} caught .‘);
} catch (error) {
if (error.message
=== "Fishing
cancelled") {
bot.chat("Fishing
was
cancelled. Trying
again ...");
i--; // Retry the same
iteration
} else {
throw
error;
}
}
}
}
34


A.5
Self-Verification
A.5.1
Components in the Prompt
The input prompt to GPT-4 consists of the following components:
(1) The agent’s state: We exclude other blocks that are recently seen and nearby entities from the
agent’s state since they are not useful for assessing the task’s completeness. See Sec. A.3.1
for each element of the agent’s state;
(2) Task proposed by the automatic curriculum;
(3) Task context: We prompt GPT-3.5 to ask for general suggestions about how to solve the
task. In practice, this part is handled by the automatic curriculum since it has a systematic
mechanism for question-answering (Sec. A.3.2);
(4) Chain-of-thought prompting [46] in response: We request GPT-4 to initially reason about
the task’s success or failure, then output a boolean variable indicating the task’s outcome,
and finally provide a critique to the agent if the task fails.
(5) Few-shot examples for in-context learning [36–38].
A.5.2
Full Prompt
Prompt 6: Full system prompt for self-verification.
You are an assistant
that
assesses my progress of playing
Minecraft
and
provides
useful
guidance.
You are
required to evaluate if I have met the task
requirements .
Exceeding
the task
requirements is also
considered a success
while
failing to meet them
requires
you to provide
critique to help me
improve.
I will give you the
following
information:
Biome: The biome
after the task
execution.
Time: The
current
time.
Nearby
blocks: The
surrounding
blocks. These
blocks are not
collected
yet. However , this is useful for some
placing or planting
tasks.
Health: My current
health.
Hunger: My current
hunger
level. For eating task , if my hunger
level
is 20.0, then I successfully
ate the food.
Position: My current
position.
Equipment: My final
equipment. For
crafting tasks , I sometimes
equip
the
crafted
item.
Inventory (xx /36): My final
inventory. For mining and
smelting tasks ,
you only need to check
inventory.
Chests: If the task
requires me to place
items in a chest , you can
find
chest
information
here.
Task: The
objective I need to accomplish.
Context: The
context of the task.
You should
only
respond in JSON
format as described
below:
{
"reasoning ": "reasoning",
"success ": boolean ,
"critique ": "critique",
}
Ensure the
response
can be parsed by Python ‘json.loads ‘, e.g.: no
trailing
commas , no single quotes , etc.
Here are some
examples:
INPUT:
Inventory
(2/36): {’oak_log ’:2, ’spruce_log ’:2}
35


Task: Mine 3 wood logs
RESPONSE:
{
"reasoning ": "You need to mine 3 wood logs. You have 2 oak logs
and 2 spruce logs , which add up to 4 wood logs.",
"success ": true ,
"critique ": ""
}
INPUT:
Inventory
(3/36): {’crafting_table ’: 1, ’spruce_planks ’: 6, ’stick ’:
4}
Task: Craft a wooden
pickaxe
RESPONSE:
{
"reasoning ": "You have
enough
materials to craft a wooden
pickaxe ,
but you didn ’t craft it.",
"success ": false ,
"critique ": "Craft a wooden
pickaxe
with a crafting
table
using 3
spruce
planks and 2 sticks ."
}
INPUT:
Inventory
(2/36): {’raw_iron ’: 5, ’stone_pickaxe ’: 1}
Task: Mine 5 iron_ore
RESPONSE:
{
"reasoning ": "Mining
iron_ore in Minecraft
will get
raw_iron. You
have 5 raw_iron in your
inventory .",
"success ": true ,
"critique ": ""
}
INPUT:
Biome: plains
Nearby
blocks: stone , dirt , grass_block , grass , farmland , wheat
Inventory
(26/36): ...
Task:
Plant 1 wheat
seed.
RESPONSE:
{
"reasoning ": "For
planting tasks , inventory
information is useless
. In nearby blocks , there is farmland
and wheat , which
means you
succeed to plant the wheat
seed.",
"success ": true ,
"critique ": ""
}
INPUT:
Inventory
(11/36): {... ,’rotten_flesh ’: 1}
Task: Kill 1 zombie
Context: ...
RESPONSE
{
36


"reasoning ": "You have
rotten
flesh in your
inventory , which
means
you
successfully
killed one zombie .",
"success ": true ,
"critique ": ""
}
INPUT:
Hunger: 20.0/20.0
Inventory
(11/36): ...
Task: Eat 1 ...
Context: ...
RESPONSE
{
"reasoning ": "For all eating task , if the player ’s hunger is 20.0 ,
then the player
successfully
ate the food.",
"success ": true ,
"critique ": ""
}
INPUT:
Nearby
blocks: chest
Inventory
(28/36): {’rail ’: 1, ’coal ’: 2, ’oak_planks ’: 13, ’
copper_block ’: 1, ’diorite ’: 7, ’cooked_beef ’: 4, ’granite ’: 22, ’
cobbled_deepslate ’: 23, ’feather ’: 4, ’leather ’: 2, ’
cooked_chicken ’: 3, ’white_wool ’: 2, ’stick ’: 3, ’black_wool ’: 1,
’stone_sword ’: 2, ’stone_hoe ’: 1, ’stone_axe ’: 2, ’stone_shovel ’:
2, ’cooked_mutton ’: 4, ’cobblestone_wall ’: 18, ’crafting_table ’:
1, ’furnace ’: 1, ’iron_pickaxe ’: 1, ’stone_pickaxe ’: 1, ’
raw_copper ’: 12}
Chests:
(81, 131, 16): {’andesite ’: 2, ’dirt ’: 2, ’cobblestone ’: 75, ’
wooden_pickaxe ’: 1, ’wooden_sword ’: 1}
Task: Deposit
useless
items
into the chest at (81, 131, 16)
Context: ...
RESPONSE
{
"reasoning ": "You have 28 items in your
inventory
after
depositing
, which is more than 20. You need to deposit
more
items
from your
inventory to the chest.",
"success ": false ,
"critique ": "Deposit
more
useless
items
such as copper_block ,
diorite , granite , cobbled_deepslate , feather , and
leather to meet
the
requirement of having
only 20 occupied
slots in your
inventory
."
}
A.6
System-level Comparison between VOYAGER and Prior Works
We make a system-level comparison in Table. A.2. Voyager stands out as the only method featuring a
combination of automatic curriculum, iterative planning, and a skill library. Moreover, it learns to
play Minecraft without the need for any gradient update.
37


Table A.2: System-level comparison between VOYAGER and prior works.
VPT [8]
DreamerV3 [69] DECKARD [53] DEPS [55]
Plan4MC [71]
VOYAGER
Demos
Videos
None
Videos
None
None
None
Rewards
Sparse
Dense
Sparse
None
Dense
None
Observations
Pixels Only
Pixels &
Meta
Pixels &
Inventory
Feedback &
Inventory
Pixels &
Meta
Feedback &
Meta &
Inventory
Actions
Keyboard
&
Mouse
Discrete
Keyboard
&
Mouse
Keyboard
&
Mouse
Discrete
Code
Automatic
Curriculum
✓
✓
(in-context
GPT-4
pro-
posal)
Iterative Plan-
ning
✓
✓
(3
types
of
feedback)
Skill Library
✓
(pre-defined)
✓
(self-
generated)
Gradient-Free
✓
B
Experiments
B.1
Experimental Setup
Our simulation environment is built upon MineDojo [23] and utilizes Mineflayer [52] JavaScript APIs
for motor controls (Sec. A.4.2). Additionally, we incorporate many bot.chat() into Mineflayer
functions to provide abundant environment feedback and implement various condition checks along
with try-catch exceptions for continuous execution. If the bot dies, it is resurrected near the closest
ground, and its inventory is preserved for uninterrupted exploration. The bot recycles its crafting table
and furnace after program execution. For detailed implementations, please refer to our codebase.
B.2
Baselines
ReAct [29] uses chain-of-thought prompting [46] by generating both reasoning traces and action
plans with LLMs. We provide it with our environment feedback and the agent states as observations.
ReAct undergoes one round of code generation from scratch, followed by three rounds of code
refinement. This process is then repeated until the maximum prompting iteration is reached.
Reflexion [30] is built on top of ReAct [29] with self-reflection to infer more intuitive future actions.
We provide it with environment feedback, the agent states, execution errors, and our self-verification
module. Similar to ReAct, Reflexion undergoes one round of code generation from scratch, followed
by three rounds of code refinement. This process is then repeated until the maximum prompting
iteration is reached.
AutoGPT [28] is a popular software tool that automates NLP tasks by decomposing a high-level goal
into multiple subgoals and executing them in a ReAct-style loop. We re-implement AutoGPT by
using GPT-4 to do task decomposition and provide it with the agent states, environment feedback,
and execution errors as observations for subgoal execution. Compared with VOYAGER, AutoGPT
lacks the skill library for accumulating knowledge, self-verification for assessing task success, and
automatic curriculum for open-ended exploration. During each subgoal execution, if no execution
error occurs, we consider the subgoal completed and proceed to the next one. Otherwise, we refine
the program until three rounds of code refinement (equivalent to four rounds of code generation)
are completed, and then move on to the next subgoal. If three consecutive subgoals do not result in
acquiring a new item, we replan by rerunning the task decomposition.
The task is “explore the world and get as many items as possible” for all baselines.
38


Table A.3: Comparison between VOYAGER and baselines.
ReAct [29]
Reflexion [30]
AutoGPT [28]
VOYAGER
Chain-of-Thought [46]
✓
✓
✓
✓
Self Verification
✓
✓
Environment Feedback
✓
✓
✓
✓
Execution Errors
✓
✓
✓
Agent State
✓
✓
✓
✓
Skill Library
✓
Automatic Curriculum
✓
Figure A.1: Minecraft item icons with corresponding names.
B.3
Ablations
We ablate 6 design choices (automatic curriculum, skill library, environment feedback, execution
errors, self-verification, and GPT-4 for code generation) in VOYAGER and study their impact on
exploration performance.
• Manual Curriculum: We substitute the automatic curriculum with a manually designed
curriculum for mining a diamond: “Mine 3 wood log”, “Craft 1 crafting table”, “Craft
1 wooden pickaxe”, “Mine 11 cobblestone”, “Craft 1 stone pickaxe”, “Craft 1 furnace”,
“Mine 3 iron ore”, “Smelt 3 iron ore”, “Craft 1 iron pickaxe”, “Mine 1 diamond”. A manual
curriculum requires human effort to design and is not scalable for open-ended exploration.
• Random Curriculum: We curate 101 items obtained by VOYAGER and create a random
curriculum by randomly selecting one item as the next task.
• w/o Skill Library: We remove the skill library, eliminating skill retrieval for code generation.
• w/o Environment Feedback: We exclude environment feedback (chat log) from the prompt
for code generation.
• w/o Execution Errors: We exclude execution errors from the prompt for code generation.
• w/o Self-Verification: For each task, we generate code without self-verification and it-
eratively refine the program for 3 rounds (equivalent to 4 rounds of code generation in
total).
• GPT-3.5: We replace GPT-4 with GPT-3.5 for code generation. We retain GPT-4 for the
automatic curriculum and the self-verification module.
B.4
Evaluation Results
B.4.1
Significantly Better Exploration
The meaning of each icon in Fig. 1 is shown in Fig. A.1.
We run three trials for each method. The items collected by VOYAGER in each trial is
39


• Trial 1:
‘iron_ingot’, ‘stone_shovel’, ‘iron_leggings’, ‘fishing_rod’, ‘pufferfish’,
‘oak_log’, ‘cooked_mutton’, ‘green_dye’, ‘flint’, ‘chest’, ‘iron_sword’, ‘string’, ‘en-
der_pearl’, ‘raw_copper’, ‘crafting_table’, ‘cactus’, ‘lapis_lazuli’, ‘iron_pickaxe’, ‘cop-
per_ingot’, ‘stone_pickaxe’, ‘wooden_hoe’, ‘scaffolding’, ‘stick’, ‘porkchop’, ‘cop-
per_block’, ‘gravel’, ‘grass_block’, ‘white_bed’, ‘bone’, ‘dirt’, ‘mutton’, ‘white_wool’,
‘oak_sapling’, ‘coal’, ‘bamboo’, ‘wooden_pickaxe’, ‘rotten_flesh’, ‘cooked_porkchop’,
‘cod’, ‘iron_boots’, ‘lightning_rod’, ‘diorite’, ‘water_bucket’, ‘shears’, ‘furnace’, ‘andesite’,
‘granite’, ‘bucket’, ‘wooden_sword’, ‘sandstone’, ‘iron_helmet’, ‘raw_iron’, ‘sand’, ‘aca-
cia_log’, ‘cooked_cod’, ‘oak_planks’, ‘azure_bluet’, ‘iron_shovel’, ‘acacia_planks’, ‘shield’,
‘iron_axe’, ‘iron_chestplate’, ‘cobblestone’;
• Trial 2: ‘iron_ingot’, ‘tuff’, ‘stone_shovel’, ‘iron_leggings’, ‘fishing_rod’, ‘cooked_mutton’,
‘spruce_planks’, ‘gunpowder’, ‘amethyst_shard’, ‘chest’, ‘string’, ‘cooked_salmon’,
‘iron_sword’, ‘raw_copper’, ‘crafting_table’, ‘torch’, ‘lapis_lazuli’, ‘iron_pickaxe’, ‘cop-
per_ingot’, ‘stone_pickaxe’, ‘wooden_hoe’, ‘stick’, ‘amethyst_block’, ‘salmon’, ‘cal-
cite’, ‘gravel’, ‘white_bed’, ‘bone’, ‘dirt’, ‘mutton’, ‘white_wool’, ‘spyglass’, ‘coal’,
‘wooden_pickaxe’, ‘cod’, ‘iron_boots’, ‘lily_pad’, ‘cobbled_deepslate’, ‘lightning_rod’,
‘snowball’, ‘stone_axe’, ‘smooth_basalt’, ‘diorite’, ‘water_bucket’, ‘furnace’, ‘andesite’,
‘bucket’, ‘granite’, ‘shield’, ‘iron_helmet’, ‘raw_iron’, ‘cobblestone’, ‘spruce_log’,
‘cooked_cod’, ‘tripwire_hook’, ‘stone_hoe’, ‘iron_chestplate’, ‘stone_sword’;
• Trial 3:
‘spruce_planks’, ‘dirt’, ‘shield’, ‘redstone’, ‘clock’, ‘diamond_sword’,
‘iron_chestplate’, ‘stone_pickaxe’, ‘leather’, ‘string’, ‘chicken’, ‘chest’, ‘diorite’,
‘iron_leggings’, ‘black_wool’, ‘cobblestone_wall’, ‘cobblestone’, ‘cooked_chicken’,
‘feather’, ‘stone_sword’, ‘raw_gold’, ‘gravel’, ‘birch_planks’, ‘coal’, ‘cobbled_deepslate’,
‘oak_planks’, ‘iron_pickaxe’, ‘granite’, ‘tuff’, ‘crafting_table’, ‘iron_helmet’, ‘stone_hoe’,
‘iron_ingot’, ‘stone_axe’, ‘birch_boat’, ‘stick’, ‘sand’, ‘bone’, ‘raw_iron’, ‘beef’, ‘rail’,
‘oak_sapling’, ‘kelp’, ‘gold_ingot’, ‘birch_log’, ‘wheat_seeds’, ‘cooked_mutton’, ‘furnace’,
‘arrow’, ‘stone_shovel’, ‘white_wool’, ‘andesite’, ‘jungle_slab’, ‘mutton’, ‘iron_sword’,
‘copper_ingot’, ‘diamond’, ‘torch’, ‘oak_log’, ‘cooked_beef’, ‘copper_block’, ‘flint’,
‘bone_meal’, ‘raw_copper’, ‘wooden_pickaxe’, ‘iron_boots’, ‘wooden_sword’.
The items collected by ReAct [29] in each trial is
• Trial 1: ‘bamboo’, ‘dirt’, ‘sand’, ‘wheat_seeds’;
• Trial 2: ‘dirt’, ‘rabbit’, ‘spruce_log’, ‘spruce_sapling’;
• Trial 3: ‘dirt’, ‘pointed_dripstone’;
The items collected by Reflexion [30] in each trial is
• Trial 1: ‘crafting_table’, ‘orange_tulip’, ‘oak_planks’, ‘oak_log’, ‘dirt’;
• Trial 2: ‘spruce_log’, ‘dirt’, ‘clay_ball’, ‘sand’, ‘gravel’;
• Trial 3: ‘wheat_seeds’, ‘oak_log’, ‘dirt’, ‘birch_log’, ‘sand’.
The items collected by AutoGPT [28] in each trial is
• Trial 1: ‘feather’, ‘oak_log’, ‘leather’, ‘stick’, ‘porkchop’, ‘chicken’, ‘crafting_table’,
‘wheat_seeds’, ‘oak_planks’, ‘dirt’, ‘mutton’;
• Trial 2:
‘wooden_pickaxe’, ‘iron_ingot’, ‘stone’, ‘coal’, ‘spruce_planks’, ‘string’,
‘raw_copper’, ‘crafting_table’, ‘diorite’, ‘andesite’, ‘furnace’, ‘torch’, ‘spruce_sapling’,
‘granite’, ‘iron_pickaxe’, ‘stone_pickaxe’, ‘wooden_axe’, ‘raw_iron’, ‘stick’, ‘spruce_log’,
‘dirt’, ‘cobblestone’;
• Trial 3: ‘wooden_shovel’, ‘wooden_pickaxe’, ‘iron_ingot’, ‘stone’, ‘cod’, ‘coal’, ‘oak_log’,
‘flint’, ‘raw_copper’, ‘crafting_table’, ‘diorite’, ‘furnace’, ‘andesite’, ‘torch’, ‘granite’,
‘lapis_lazuli’, ‘iron_pickaxe’, ‘stone_pickaxe’, ‘raw_iron’, ‘stick’, ‘gravel’, ‘oak_planks’,
‘dirt’, ‘iron_axe’, ‘cobblestone’.
40


Figure A.2: Map coverage: Two bird’s eye views of Minecraft maps. VOYAGER is able to traverse
2.3× longer distances compared to baselines while crossing diverse terrains. Trajectories are plotted
based on the positions where each agent interacts with GPT-4.
B.4.2
Extensive Map Traversal
Agent trajectories for map coverage are displayed in Fig. A.2. Fig. 7 is plotted based on Fig. A.2 by
drawing the smallest circle enclosing each trajectory. The terrains traversed by VOYAGER in each
trial is
• Trial 1: ‘meadow’, ‘desert’, ‘river’, ‘savanna’, ‘forest’, ‘plains’, ‘bamboo_jungle’, ‘drip-
stone_caves’;
• Trial 2: ‘snowy_plains’, ‘frozen_river’, ‘dripstone_caves’, ‘snowy_taiga’, ‘beach’;
• Trial 3:
‘flower_forest’,
‘meadow’,
‘old_growth_birch_forest’,
‘snowy_slopes’,
‘frozen_peaks’, ‘forest’, ‘river’, ‘beach’, ‘ocean’, ‘sunflower_plains’, ‘plains’, ‘stony_shore’.
The terrains traversed by ReAct [29] in each trial is
• Trial 1: ‘plains’, ‘desert’, ‘jungle’;
• Trial 2: ‘snowy_plains’, ‘snowy_taiga’, ‘snowy_slopes’;
• Trial 3: ‘dark_forest’, ‘dripstone_caves’, ‘grove’, ‘jagged_peaks’.
The terrains traversed by Reflexion [30] in each trial is
• Trial 1: ‘plains’, ‘flower_forest’;
• Trial 2: ‘snowy_taiga’;
• Trial 3: ‘old_growth_birch_forest’, ‘river’, ‘ocean’, ‘beach’, ‘plains’.
The terrains traversed by AutoGPT [28] in each trial is
• Trial 1: ‘plains’, ‘dripstone_caves’, ‘savanna’, ‘meadow’;
• Trial 2: ‘snowy_taiga’;
• Trial 3: ‘plains’, ‘stony_shore’, ‘forest’, ‘ocean’.
B.4.3
Efficient Zero-Shot Generalization to Unseen Tasks
The results of zero-shot generalization to unseen tasks for the other two tasks are presented in Fig. A.3.
Similar to Fig. 8, VOYAGER consistently solves all tasks, while the baselines are unable to solve any
41


Figure A.3: Zero-shot generalization to unseen tasks. We visualize the intermediate progress of each
method on the other two tasks. We do not plot ReAct and Reflexion since they do not make any
meaningful progress.
task within 50 prompting iterations. Our skill library, constructed from lifelong learning, not only
enhances VOYAGER’s performance but also provides a boost to AutoGPT [28].
B.4.4
Accurate Skill Retrieval
We conduct an evaluation of our skill retrieval (309 samples in total) and the results are in Table. A.4.
The top-5 accuracy standing at 96.5% suggests our retrieval process is reliable (note that we include
the top-5 relevant skills in the prompt for synthesizing a new skill).
Table A.4: Skill retrieval accuracy.
Top-1 Acc
Top-2 Acc
Top-3 Acc
Top-4 Acc
Top-5 Acc
80.2 ± 3.0
89.3 ± 1.8
93.2 ± 0.7
95.2 ± 1.8
96.5 ± 0.3
B.4.5
Robust to Model Variations
In the main paper, all of Voyager’s experiments are conducted with gpt-4-0314. We additionally
run new experiments with gpt-4-0613 and find that the performance is roughly the same (Fig. A.4).
It demonstrates that Voyager is robust to model variations.
Figure A.4: VOYAGER’s performance with GPT-4-0314 and GPT-4-0613.
42

Video PreTraining (VPT): Learning to Act by
Watching Unlabeled Online Videos
Abstract
Pretraining on noisy, internet-scale datasets has been heavily studied as a technique
for training models with broad, general capabilities for text, images, and other
modalities.1–6 However, for many sequential decision domains such as robotics,
video games, and computer use, publicly available data does not contain the labels
required to train behavioral priors in the same way. We extend the internet-scale
pretraining paradigm to sequential decision domains through semi-supervised
imitation learning wherein agents learn to act by watching online unlabeled videos.
Specifically, we show that with a small amount of labeled data we can train an
inverse dynamics model accurate enough to label a huge unlabeled source of online
data – here, online videos of people playing Minecraft – from which we can then
train a general behavioral prior. Despite using the native human interface (mouse
and keyboard at 20Hz), we show that this behavioral prior has nontrivial zero-
shot capabilities and that it can be fine-tuned, with both imitation learning and
reinforcement learning, to hard-exploration tasks that are impossible to learn from
scratch via reinforcement learning. For many tasks our models exhibit human-
level performance, and we are the first to report computer agents that can craft
diamond tools, which can take proficient humans upwards of 20 minutes (24,000
environment actions) of gameplay to accomplish.
1
Introduction
Work in recent years has demonstrated the efficacy of pretraining large and general foundation
models7 on noisy internet-scale datasets for use in downstream tasks in natural language1–4 and
computer vision.5,6,8 For sequential decision domains (e.g. robotics, game playing, and computer
usage) where agents must repeatedly act within an environment, a wealth of data also exists on the
web; however, most of this data is in the form of unlabeled video (i.e. without the actions taken
at each frame), making it much less straightforward to train a behavioral prior in these domains
than it is in e.g. natural language. In a few rare settings, such as Chess, Go, and StarCraft, there
∗This was a large effort by a dedicated team. Each author made huge contributions on many fronts over long
time periods. All members were full time on the project for over six months. BB, IA, PZ, and JC were on the
original VPT project team and were thus involved for even longer (over a year). Aside from those original team
members, author order is random. It was also randomized between IA and PZ.
†OpenAI
‡University of British Columbia
arXiv:2206.11795v1  [cs.LG]  23 Jun 2022


already exist large datasets with action labels from various online platforms that researchers have
used for imitation learning.9,10 When large labeled datasets do not exist, the canonical strategy
for training capable agents is reinforcement learning (RL),11 which can be sample inefficient and
expensive for hard-exploration problems.12–18 Many virtual tasks, e.g. navigating websites, using
Photoshop, booking flights, etc., can be very hard to learn with RL and do not have large, commonly
available sources of labeled data.19,20 In this paper, we seek to extend the paradigm of training
large, general-purpose foundation models to sequential decision domains by utilizing freely available
internet-scale unlabeled video datasets with a simple semi-supervised imitation learning method. We
call this method Video PreTraining (VPT) and demonstrate its efficacy in the domain of Minecraft.
Existing semi-supervised imitation learning methods aim to learn with few or no explicit action labels;
however, they generally rely on the policy’s ability to explore the environment throughout training,
making them susceptible to exploration bottlenecks.21–25 Furthermore, most prior semi-supervised
imitation learning work was tested in the relatively low data regime; because we experiment with far
more data (∼70k hours of unlabeled video), we hypothesize that we can achieve good performance
with a much simpler method, a trend that has proven true for pretraining in other modalities such
as text.1 In particular, given a large but unlabeled dataset, we propose generating pseudo-labels by
gathering a small amount of labeled data to train an inverse dynamics model (IDM) that predicts
the action taken at each timestep in a video. Behavioral cloning (BC) can require a large amount
of data because the model must learn to infer intent and the distribution over future behaviors from
only past observations. In contrast, the inverse dynamics modeling task is simpler because it is
non-causal, meaning it can look at both past and future frames to infer actions. In most settings,
environment mechanics are far simpler than the breadth of human behavior that can take place within
the environment, suggesting that non-causal IDMs could require far less data to train than causal BC
models. Using pseudo-labels generated from the IDM, we then train a model to mimic the distribution
of behavior in the previously unlabeled dataset with standard behavioral cloning at scale, which does
not require any model rollouts and thus does not suffer from any potential exploration bottlenecks
in the environment. Finally, we show we can fine-tune this model to downstream tasks with either
behavioral cloning or reinforcement learning.
Figure 1:
Example Minecraft
crafting GUI. Agents use the
mouse and keyboard to navigate
menus and drag and drop items.
We chose to test our method in Minecraft because (a) it is one
of the most actively played games in the world26 and thus has
a wealth of commonly available video data online, (b) it is a
fairly open-ended sandbox game with an extremely wide variety
of potential things to do, build, and collect, making our results
more applicable to real-world applications such as computer us-
age, which also tends to be varied and open-ended, and (c) it
has already garnered interest by the RL community as a research
domain due to its complexity and correspondingly difficult ex-
ploration challenges.27–31 In this work we use the native human
interface for Minecraft so that we can (1) most accurately model
the human behavior distribution and reduce domain shift between
video data and the environment, (2) make data collection easier by allowing our human contractors to
play the game without modification, and (3) eliminate the need to hand-engineer a custom interface
for models to interact with the environment. This choice means that our models play at 20 frames
per second and must use a mouse and keyboard interface to interact with human GUIs for crafting,
smelting, trading, etc., including dragging items to specific slots or navigating the recipe book with
the mouse cursor (Fig. 1). Compared to prior work in Minecraft that uses a lower frame rate and
constructs crafting and attacking macros,30,32–34 using the native human interface drastically increases
the environment’s exploration difficulty, making most simple tasks near impossible with RL from
scratch. Even the simple task of gathering a single wooden log while already facing a tree takes 60
consecutive attack actions with the human interface, meaning the chance for a naive random policy to
succeed is 1
2
60. While this paper shows results in Minecraft only, the VPT method is general and
could be applied to any domain.
In Section 4 we show that the VPT foundation model has nontrivial zero-shot performance, accom-
plishing tasks impossible to learn with RL alone, such as crafting planks and crafting tables (tasks
requiring a human proficient in Minecraft a median of 50 seconds or ∼970 consecutive actions).
Through fine-tuning with behavioral cloning to smaller datasets that target more specific behavior
distributions, our agent is able to push even further into the technology tree, crafting stone tools
2


(taking a human a median of 2.3 minutes or ∼2790 actions). Finally, fine-tuning via RL produces
the most dramatic improvements: our agent is able to craft diamond tools, an unprecedented result
in Minecraft made even more challenging by using the native human interface. This task requires
a proficient human a median upwards of 20 minutes or ∼24000 actions. The main contributions
of this work are (1) we are the first to show promising results applying semi-supervised imitation
learning to extremely large, noisy, and freely available video datasets for sequential decision domains,
(2) we show that such pretraining plus fine-tuning enables agents to solve tasks that were otherwise
impossible to learn, (3) we show that labeled contractor data is far more efficiently used within
the VPT method than it would be by directly training a foundation model from it and (4) we open
source our contractor data, trained model weights, and Minecraft environment for future research
into learning to act via semi-supervised imitation learning at scale.
2
Preliminaries and Related Work
Imitation learning methods35–38 seek to construct a policy that accurately models the distribution of
behavior in some dataset D = {(oi, ai)}, i ∈{1...N} of action-observation pairs. In order to roll
out these policies in an environment, they must be causal, meaning they condition on observations
from the current timestep t and past timesteps only, i.e. π ∼p(at|o1...ot). Imitation learning is
simplest when demonstrations are labeled with corresponding actions. Imitating labeled trajectories
has seen success in aerial vehicles,39,40 self-driving cars,41,42 board games,9,43 and video games.10,44
When labeled demonstrations are not available, standard behavioral cloning will not work; however,
there is a large body of work in imitating behavior from unlabeled demonstrations.22 For instance,
GAIL23 constructs an adversarial objective incentivizing the trained policy to exhibit behaviors
indistinguishable from those in the target dataset. Edwards et al. 45 propose to first learn a latent
policy using unlabeled demonstrations and then map the learned latent actions to real actions with
a small amount of environment interaction. Peng et al. 46 first use motion-capture methods to track
agent positions in videos and then train RL agents to match these waypoints. Similarly, Behbahani
et al. 47 and Aytar et al. 48 task a RL agent to match waypoints; however, they construct waypoints that
are embeddings from unsupervised feature learning models. Pathak et al. 49 and Nair et al. 50 train
goal conditioned policies to take actions that advance the current state towards expert-provided goal
states expressed as high dimensional visual waypoints. Most similar to our own work, Torabi et al. 24
simultaneously train (1) an inverse dynamics model (IDM),51 which aims to uncover the underlying
action between timesteps given observations of past and future timesteps, e.g. pIDM(at|ot, ot+1), and
(2) a behavioral cloning (BC) model on trajectories of observations labeled with the IDM. Data to
train the IDM is collected by rolling out the BC model in the target environment such that both
models improve in tandem. However, at any point in training if there are sequences in the dataset that
the IDM performs poorly on, it requires that the BC model perform those or similar sequences in
order for the IDM to improve and correctly label them. Therefore, if the BC model does not explore
efficiently, it could severely slow down learning. In order to avoid this potential issue we opted for a
simpler two-stage approach: we first train an IDM on a small number of labeled trajectories collected
from human contractors (they play the game as would normally as we record their keypresses and
mouse movements). Because human contractors reach most relevant parts of the state space, we can
hold the IDM fixed throughout BC training.
Compared to most previous work in semi-supervised imitation learning, we experiment in the much
more complex and open-ended environment of Minecraft. Minecraft is a voxel-based 3D video
game that, due its popularity and wide variety of mechanics, has attracted a vast amount of RL
research.27,28,30–34,52–60 A large body of work focuses on small, custom-made Minecraft worlds
with tasks such as navigation,53,60 block placing,54,55 instruction following,58,59 combat,56 and
others.28,31,57 Work operating in the massive, randomly generated environments of Minecraft itself
has included hill climbing,52 automated curriculum learning30 and, most closely related to the RL
experiments presented in Sec. 4.4, diamond mining.27,32–34 However, to the best of our knowledge,
there is no published work that operates in the full, unmodified human action space, which includes
drag-and-drop inventory management and item crafting.
3


Collecting “Clean” Data
Training the VPT Foundation Model
via Behavioral Cloning
Training the Inverse Dynamics Model (IDM)
~270k hours
unlabeled
video
~70k hours
unlabeled
video
~2k hours
video
labeled with
actions
Filter for “clean”
video segments
Search for relevant
Minecraft videos
via keywords
Contractors
collect data 
Label videos
with IDM 
~70k hours
video
IDM-labeled
with actions
Train non-causal IDM
Train causal
VPT Foundation Model
a
d
space
w
a
d
space
w
Figure 2: Video Pretraining (VPT) Method Overview.
3
Methods
Inverse Dynamics Models (IDM)
VPT, illustrated in Figure 2, requires we first collect a small
amount of labeled contractor data with which to train an inverse dynamics model pIDM(at|o1...T ),
which seeks to minimize the negative log-likelihood of an action at timestep t given a trajectory of T
observations ot : t ∈[1...T]. In contrast to an imitation learning policy, the IDM can be non-causal,
meaning its prediction for at can be a function of both past and future events, i.e. ot′>t. Compared to
the behavioral cloning objective of modeling the distribution of human intent given past frames only,
we hypothesize that inverting environment dynamics is easier and more data efficient to learn. Indeed,
Sec. 4.1 will show that the IDM objective is much easier to learn, and furthermore Sec. 4.6 will show
that with very little labeled data (as few as 100 hours) we can train a fairly accurate IDM. This IDM
can be used to label online videos, providing the large amount of data required for the harder task of
behavioral cloning. See appendices D and B for IDM training and data collection details.
Data Filtering
We gather a large dataset of Minecraft videos by searching the web for related
keywords (Appendix A). Online videos often (1) include overlaid artifacts, such as a video feed
of the player’s face, channel logos, watermarks, etc., (2) are collected from platforms other than
a computer with different gameplay, or (3) are from different game modes, e.g. in Minecraft we
only want "survival mode" where players start from scratch and must gather or craft all their items.
We call data “clean” if it does not contain visual artifacts and is from survival mode, and call all
other data “unclean.” With enough data, a large enough model, and enough training compute, a BC
model trained on both unclean and clean videos would likely still perform well in a clean Minecraft
environment. However, for simplicity and training compute efficiency, we choose to filter out unclean
segments of video (note that a video may contain both clean and unclean segments). We do this by
training a model to filter out unclean segments using a small dataset (8800) of images sampled from
online videos labeled by contractors as clean or unclean (Appendix A.2).
VPT Foundation Model
We train a foundation model with standard behavioral cloning, i.e. mini-
mizing the negative log-likelihood of actions predicted by the IDM on clean data. For a particular
trajectory of length T we minimize
min
θ
X
t∈[1...T ]
−log πθ(at|o1, . . . , ot), where at ∼pIDM(at|o1, . . . , ot, . . . , oT )
(1)
As we will see in the following sections, this model exhibits nontrivial zero-shot behavior and can be
fine-tuned with both imitation learning and RL to perform even more complex skills.
4
Results
4.1
Performance of the Inverse Dynamics Model
The IDM architecture is comprised primarily of a temporal convolution layer, a ResNet62 image
processing stack, and residual unmasked attention layers, from which the IDM simultaneously
predicts keypresses and mouse movements (see Appendix D for IDM architecture and training
details). A key hypothesis behind our work is that IDMs can be trained with a relatively small amount
of labeled data. While more data improves both mouse movement and keypress predictions, our best
4


Figure 3: (Left) IDM keypress accuracy and mouse movement R2 (explained variance61) as a
function of dataset size. (Right) IDM vs. behavioral cloning data efficiency.
IDM trains on only 1962 hours of data (compared to the ∼70k hours of clean data we collected from
the internet) and achieves 90.6% keypress accuracy and a 0.97 R2 for mouse movements evaluated
on a held-out validation set of contractor-labeled data (Figure 3 left).
Figure 3 (right) validates our hypothesis that IDMs are far more data efficient than BC models, likely
because inverting environment mechanics is far easier than modeling the entire distribution of human
behavior. The IDM is two orders of magnitude more data efficient than a BC model trained on the
same data and improves more quickly with more data. This evidence supports the hypothesis that it is
more effective to use contractor data within the VPT pipeline by training an IDM than it is to train a
foundation model from contractor data directly (Sections 4.5 and 4.6 provide additional evidence).
4.2
VPT Foundation Model Training and Zero-Shot Performance
Figure 4: (Left) Training and validation loss on the web_clean internet dataset with IDM pseudo-
labels, and loss on the main IDM contractor dataset, which has ground-truth labels but is out-of-
distribution (see text). (Right) Amount a given item was collected per episode averaged over 2500
60-minute survival episodes as a function of training epoch, shaded with the standard error of the
mean. Basic mining refers to collection of dirt, gravel, or sand (all materials that can be gathered
without tools). Logs are obtained by repeatedly hitting trees for three seconds, a difficult feat for an
RL agent to achieve as we show in Sec. 4.4. Planks can be crafted from logs, and crafting tables
crafted from planks. Crafting requires using in-game crafting GUIs, and proficient humans take a
median of 50 seconds (970 consecutive actions) to make a crafting table.
We now explore the emergent behavior learned by a behavioral cloning policy trained on an extremely
large, but noisy, internet dataset labeled with our IDM. To collect the unlabeled internet dataset,
we searched for publicly available videos of Minecraft play with search terms such as “minecraft
survival for beginners.” These searches resulted in ∼270k hours of video, which we filtered down to
“clean” video segments yielding an unlabeled dataset of ∼70k hours, which we refer to as web_clean
(Appendix A has further details on data scraping and filtering). We then generated pseudo-labels
for web_clean with our best IDM (Section 3) and then trained the VPT foundation model with
behavioral cloning. Preliminary experiments suggested that our model could benefit from 30 epochs
of training and that a 0.5 billion parameter model was required to stay in the efficient learning
regime63 for that training duration (Appendix H), which took ∼9 days on 720 V100 GPUs.
We evaluate our models by measuring validation loss (Fig. 4, left) and rolling them out in the
Minecraft environment. Unless otherwise noted, in all environment evaluations we spawn agents in a
standard survival mode game where they play for 60 minutes, i.e. 72000 consecutive actions, and we
plot the mean and shade the standard error of the mean for various game statistics such as crafting
and collection rates (Fig. 4, right). The VPT foundation model quickly learns to chop down trees
to collect logs, a task we found near impossible for an RL agent to achieve with the native human
interface (Sec. 4.4). It also learns to craft those logs into wooden planks and then use those planks
5


to craft a crafting table, which are required to unlock most other technology in the game and take a
human proficient in Minecraft approximately 50 seconds (970 consecutive actions) to collect. While
these behaviors are fairly complex in the native human action space, the VPT foundation model crafts
these items at a rate far below that of our proficient contractors, e.g. on average our contractors craft
5.44 crafting tables in 60 minutes of play versus 0.19 for the foundation model. The model also crafts
a non-negligible amount of wooden sticks, which are required to make wooden tools; collects various
flowers and crafts dyes from them; kills zombies that appear during the night; hunts wild animals;
collects various berries and mushrooms and eats them; and finds game-generated villages from which
to collect various rare items from chests. The model also learned to navigate uneven terrain, swim,
and pillar jump, which involves the agent repeatedly jumping and quickly placing a block below itself
such that it climbs upward by making a pillar.(iv)
While training and validation loss decrease healthily over training (Fig. 4, left), loss on our contractor
dataset (which the VPT model does not train on) begins increasing after 7 epochs. Contractor data
could be out-of-distribution because our contractors may have a different distribution of play or
because there is some impactful visual domain shift compared to videos from the web. While one
could have expected this would be predictive of declining evaluation performance, we do not see
notable game statistics from the VPT foundation model rollouts (Figure 4, right) decrease over
training, and in the next section we show that BC fine-tuning performance continually improves as the
VPT foundation model trains. We provide more insight into this curious phenomenon in Appendix H.
4.3
Fine-Tuning with Behavioral Cloning
Foundation models are designed to have a broad behavior profile and be generally capable across a
wide variety of tasks. To incorporate new knowledge or allow them to specialize on a narrower task
distribution, it is common practice to fine-tune these models to smaller, more specific datasets.1 The
VPT foundation model trained on the broad web_clean dataset had nontrivial zero-shot performance;
it was able to craft a crafting table yet unable to go past this in the technology tree. As a case
study into BC fine-tuning, we attempt to improve the VPT foundation model’s ability to collect
and craft these “early game” items by fine-tuning to two narrower datasets targeted at Minecraft
behavior within the first few minutes of players starting in a fresh world. In the first dataset,
contractor_house, contractors have 10 minutes to build a basic house from scratch using primarily
wood, sand, and dirt. Collecting contractor data can be difficult and expensive, so we also construct a
dataset earlygame_keyword by searching for videos online with descriptions that match keywords
such as “new world”, “let’s play episode 1”, etc.; this is a subset of web_clean and is labeled with
the IDM. See Appendix B.4 and A.3 for full descriptions of both datasets.
Effect of Foundation Model Quality on BC Fine-Tuning
59x
213x
59x
Figure 5:
(Left) Collection and crafting rates for three policies:
the zero-shot VPT foun-
dation model, and the VPT foundation model BC fine-tuned to the earlygame_keyword or
contractor_house datasets. BC fine-tuning to either dataset improves performance, including (for
the contractor_house dataset) yielding wooden and stone tools. Proficient Minecraft players take
a median of 1.2 minutes (1390 actions) to construct wooden tools and 2.3 minutes (2790 actions)
to construct stone tools. (Right) Collection and crafting rates for VPT foundation model snapshots
throughout training after they are BC fine-tuned to the contractor_house dataset. In general,
crafting-related behaviors increase throughout foundation model training. Fig. 4 defines the other
task terms (logs, planks, crafting tables, and total crafting).
(iv)Sample videos: https://www.youtube.com/playlist?list=PLNAOIb_agjf3U3rSvG_BCWqJ869NdBhcP
6


Fine-tuning to earlygame_keyword results in a large boost compared to the zero-shot foundation
model: 2.5x more crafting tables, 6.1x more planks, 4.3x more logs, and 5.5x more crafting overall
(Fig. 5). However, when fine-tuning to this dataset we did not see any new behaviors emerge,
only a refinement of existing skills. We saw an even bigger improvement when fine-tuning to the
contractor_house dataset: 213x more crafting tables, 59x more wooden planks, 7x more logs,
and 59x more crafting over all. In addition, we saw the emergence of crafting wooden tools, which
requires placing a crafting table on the ground, opening it to reveal a new crafting interface, and then
using it to craft wooden tools. This entire sequence takes a proficient human player a median of 1.2
minutes (1390 consecutive actions) to accomplish. The model goes further and collects cobblestone,
which requires a wooden pickaxe to mine, and crafts stone tools, requiring it to again use a crafting
table; this takes a proficient human player a median of 2.3 minutes (2790 consecutive actions). We
also saw this model more frequently raiding villages that randomly spawn in the game, hunting
animals for food, in addition to many behaviors we saw performed by the foundation model.(v)
Despite the foundation model’s zero-shot rollout performance plateauing 1/3 into training (Fig. 4,
right), fine-tuning performance does continue to increase throughout foundation model training
(Fig. 5, right). Additionally, there is a stark difference in performance when training from scratch vs.
fine-tuning from the VPT foundation model (Fig. 5 right, comparing the left and rightmost points).
4.4
Fine-Tuning with Reinforcement Learning
Figure 6: Typical sequence of items for obtaining a diamond pickaxe. Below each item is the median
time and number of actions contractors required to obtain that item and the percentage of contractors
that got the item within 10 minutes. The median time to obtain a diamond pickaxe is unknown (except
that it is > 20m) because contractors obtained this item in less than 50% of 20-minute episodes.
To demonstrate the efficacy of RL fine-tuning, we chose the challenging goal of obtaining a diamond
pickaxe within 10 minutes starting from a fresh Minecraft survival world. Doing so involves acquiring
a sequence of difficult-to-obtain items that require complex skills like mining, inventory management,
crafting with and without a crafting table, tool use, operating a furnace, and mining at the lowest
depths, where many hazards like enemies and lava exist (Fig. 6). Adding to the difficulty, progress
can be easily lost by dropping items, destroying items, or dying. Obtaining a diamond pickaxe more
often than not takes a proficient human over 20 minutes (24,000 actions).
Agents are rewarded for each item obtained in the sequence, with lower rewards for items that have to
be collected in bulk and higher rewards for items near the end of the sequence. Agents are optimized
with the phasic policy gradient64 RL algorithm for ∼1.3 million episodes (roughly 1.4×1010 frames).
Episodes last for 10 minutes. See Appendix G.1 for reward function and RL training details. Due to
computational constraints, RL experiments use a ∼248 million parameter VPT model (Appendix H).
A major problem when fine-tuning with RL is catastrophic forgetting65,66 because previously learned
skills can be lost before their value is realized. For instance, while our VPT foundation model never
exhibits the entire sequence of behaviors required to smelt iron zero-shot, it did train on examples of
players smelting with furnaces. It therefore may have some latent ability to smelt iron once the many
prerequisites to do so have been performed. To combat the catastrophic forgetting of latent skills
such that they can continually improve exploration throughout RL fine-tuning, we add an auxiliary
Kullback-Leibler (KL) divergence loss between the RL model and the frozen pretrained policy.10
Training from a randomly initialized policy fails to achieve almost any reward, underscoring how
hard an exploration challenge the diamond pickaxe task is for RL in the native human action space
(Fig. 7a). The model never learns to reliably collect logs, typically the first of many steps to obtaining
a diamond pickaxe (Fig. 7b). RL fine-tuning from the VPT foundation model does substantially
better (Fig. 7a), learning everything up to mining iron ore and crafting furnaces. (Fig. 7c). However,
this agent fails at smelting an iron ingot, the next item required to get further into the tech tree, likely
(v)Sample Videos: https://www.youtube.com/playlist?list=PLNAOIb_agjf2yDSs4AqcoyPv4z_eWUiKm
7


0
5
10
15
20
25
Reward
Reward over episodes
RL from Rand. Init. model
RL from VPT Found. model
RL from Early-Game model
No KL-loss
(a)
0
20
40
60
80
100
% episodes
RL from Rand. Init. model
Logs
Planks
Sticks
Crafting T
ables
Wooden Pickaxe
Cobblestone
Stone Pickaxe
Coal
T
orch
Furnace
Iron Ore
Iron Ingot
Iron Pickaxe
Diamonds
Diamond Pickaxe
(b)
0.0
0.2
0.4
0.6
0.8
1.0
1.2
1.4
Episodes
1e6
0
20
40
60
80
100
% episodes
RL from VPT Found. model
(c)
0.0
0.2
0.4
0.6
0.8
1.0
1.2
1.4
Episodes
1e6
0
20
40
60
80
100
% episodes
2.5%
RL from Early-Game model
(d)
Figure 7: RL Fine-tuning results. (a) RL from a randomly initialized model fails to get almost
any reward, RL fine-tuning from the VPT foundation model performs substantially better with a
reward near 13, and RL fine-tuning from the early-game model performs best with a reward of 25.
When training the early-game model without a KL loss to the original policy (No KL-loss) progress
stalls after 100,000 episodes, suggesting that the skills necessary to make further progress have been
catastrophically forgotten. (b) RL from a randomly initialized model occasionally collects sticks by
breaking leaves (an easy but inefficient method of getting sticks that does not require logs or planks)
and never learns to reliably collect logs. (c) RL fine-tuning from the VPT Foundation model learns
everything in the curriculum up to iron ore and making furnaces, but fails to learn to use the furnace to
smelt iron ingots. (d) RL fine-tuning from the early-game model learns to obtain (at human-level) all
items in the sequence towards a diamond pickaxe and crafts a diamond pickaxe in 2.5% of episodes.
because the zero-shot probability that the VPT foundation model smelts an iron ingot is too low, even
when given the prerequisite materials.
Results further improve by first BC fine-tuning the VPT Foundation Model to the
earlygame_keyword dataset (the early-game model, Sec. 4.3) and then fine-tuning with RL
(Fig. 7a), which in preliminary experiments we found to perform better than first fine-tuning to
contractor_house followed by fine-tuning with RL (Appendix G.2). The three-phase training
(pretraining, BC fine-tuning, and then RL fine-tuning) succeeds in learning extremely difficult tasks:
it achieves over 80% reliability on iron pickaxes, almost 20% reliability on collecting diamonds, and
2.5% reliability on obtaining a diamond pickaxe (Fig. 7d). For comparison, human players given
the objective of obtaining a diamond pickaxe collect these items in 57%, 15%, and 12% of episodes,
respectively, meaning our model is human-level for crafting iron pickaxes and mining diamonds.
Others have managed to obtain diamonds with ∼0.1% reliability in 15 minutes32,33 but always with a
simplified action space designed to ease exploration. To the best of our knowledge, we are the first to
report non-zero success rates on crafting a diamond pickaxe. Qualitatively, the model developed
useful skills for diamond mining, such as efficient mining patterns, cave exploration, returning to
previously placed objects like crafting tables, and advanced techniques like using wooden pickaxes
as fuel when moving on to iron tools.(vi)
Finally, we validated the importance of the KL loss to the pretrained model during RL fine-tuning.
The treatment without a KL loss obtains only items early in the sequence (logs, planks, sticks, and
crafting tables) limiting its reward (Fig. 7a). This failure to progress further into the sequence is
likely because, while the initial skills of chopping logs and crafting planks are being learned with RL,
subsequent skills like crafting a wooden pickaxe are lost due to catastrophic forgetting.
4.5
Data Scaling Properties of the Foundation Model
In this section we validate a core hypothesis behind this work: that it is far more effective to use
labeled contractor data to train an IDM within the VPT method than it is to directly train a BC
foundation model from that same small contractor dataset. If we could cheaply collect a labeled
contractor dataset of a similar order of magnitude as web_clean, then this would not be important;
however, collecting that scale of data would have cost millions of dollars. Figure 8 compares
foundation models trained on increasing orders of magnitude of data from 1 hour up to the full ∼70k
web_clean dataset. Foundation models trained up to and including 1k hours are trained on the IDM
(vi)Videos found at https://www.youtube.com/playlist?list=PLNAOIb_agjf3e_UKweM5pQUSfTw8r-Wfc
8


Trained on
Contractor Data
Trained on IDM
Labeled Web Data
Figure 8: (Left) Zero-shot rollout performance of foundation models trained on varying amounts
of data. Models to the left of the dashed black line (points ≤1k hours) were trained on contractor
data (ground-truth labels), and models to the right were trained on IDM pseudo-labeled subsets
of web_clean. Due to compute limitations, this analysis was performed with smaller (71 million
parameter) models except for the final point, which is the 0.5 billion parameter VPT foundation
model. (Right) The corresponding performance of each model after BC fine-tuning each model to
the contractor_house dataset.
contractor data, and those trained on 5k hours and above are trained on subsets of web_clean, which
does not contain any IDM contractor data. Scaling training data increases log collection, mining, and
crafting capabilities. The zero-shot model only begins to start crafting crafting tables at over 5000
hours of training data. When fine-tuning each foundation model to contractor_house, we see that
crafting rates for crafting tables and wooden tools increase by orders of magnitude when using the
entire ∼70k hour web_clean dataset. We furthermore only see the emergence of crafting stone tools
at the largest data scale.
4.6
Effect of Inverse Dynamics Model Quality on Behavioral Cloning
Figure 9: Zero-shot performance of BC models
trained from scratch on the earlygame_keyword
dataset labeled with IDMs that were trained on
increasing amounts of contractor data.
This section investigates how downstream
BC performance is affected by IDM qual-
ity.
We train IDMs on increasingly larger
datasets and use each to independently label
the earlygame_keyword dataset (this smaller
dataset was chosen due to a limited compute bud-
get). We then train a BC model from scratch on
each dataset and report game statistics for each
model as a function of IDM contractor dataset
size (Fig. 9).
IDMs trained on at least 10 hours of data are
required for any crafting, and the crafting rate
increases quickly up until 100 hours of data, after which there are few to no gains and differences are
likely due to noise. Similarly, crafting tables are only crafted after 50 or more hours of IDM data, and
again gains plateau after 100 hours. While in all previous experiments we use our best IDM trained
on 1962 hours of data, these results suggest we could reduce that number to as low as 100 hours.
5
Discussion and Conclusion
The results presented in this paper help pave the path to utilizing the wealth of unlabeled data on the
web for sequential decision domains. Compared to generative video modeling or contrastive methods
that would only yield representational priors, VPT offers the exciting possibility of directly learning
to act during pretraining and using these learned behavioral priors as extremely effective exploration
priors for RL. VPT could even be a better general representation learning method even when the
downstream task is not learning to act in that domain—for example, fine-tuning to explain what is
happening in a video—because arguably the most important information in any given scene would be
present in features trained to correctly predict the distribution over future human actions. We leave
this intriguing direction to future work.
Future work could improve results with more data (we estimate we could collect >1M hours) and
larger, better-tuned models. Furthermore, all the models in this work condition on past observations
only; we cannot ask the model to perform specific tasks. Appendix I presents preliminary experiments
on conditioning our models on closed captions (text transcripts of speech in videos), showing they
9


become weakly steerable; we believe this a rich direction for future research. Also, loss was not
consistently correlated with downstream evaluation metrics (Sec. 4.2), which often made progress
slow and hard-won. Another fruitful future direction would be to investigate the correlation between
various training metrics and downstream evaluations. Finally, while we do not anticipate any direct
negative societal impacts from the models trained in this work, as VPT improves and expands to other
domains it will be important to assess and mitigate harms that emerge with other forms of pretraining
on internet datasets, such as emulating inappropriate behavior.67
In conclusion, VPT extends the paradigm of training large and general purpose behavioral priors from
freely available internet-scale data to sequential decision domains. Our models exhibited impressive
zero-shot behavior and, when fine-tuned with RL, achieved an unprecedented result of crafting a
diamond pickaxe in Minecraft (all the more difficult given the human interface). We further showed
that contractor data is far better used within the VPT pipeline than to train a foundation model directly
and that only a small amount of contractor data (about $2000 USD) was required to unlock massive
amounts of unlabeled online data for use in BC. Finally, learning with the human keyboard and mouse
interface is highly general and allows losslessly modeling the entire distribution of human behavior.
While we only experiment in Minecraft, we believe that VPT provides a general recipe for training
behavioral priors in hard, yet generic, action spaces in any domain that has a large amount of freely
available unlabeled data, such as computer usage.
References
[1] Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal,
Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language models are
few-shot learners. Advances in neural information processing systems, 33:1877–1901, 2020.
[2] Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. Bert: Pre-training of
deep bidirectional transformers for language understanding. arXiv preprint arXiv:1810.04805,
2018.
[3] Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike
Lewis, Luke Zettlemoyer, and Veselin Stoyanov. Roberta: A robustly optimized bert pretraining
approach. arXiv preprint arXiv:1907.11692, 2019.
[4] Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena,
Yanqi Zhou, Wei Li, and Peter J Liu. Exploring the limits of transfer learning with a unified
text-to-text transformer. arXiv preprint arXiv:1910.10683, 2019.
[5] Dhruv Mahajan, Ross Girshick, Vignesh Ramanathan, Kaiming He, Manohar Paluri, Yixuan Li,
Ashwin Bharambe, and Laurens Van Der Maaten. Exploring the limits of weakly supervised
pretraining. In Proceedings of the European conference on computer vision (ECCV), pages
181–196, 2018.
[6] Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal,
Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual
models from natural language supervision. In International Conference on Machine Learning,
pages 8748–8763. PMLR, 2021.
[7] Rishi Bommasani, Drew A Hudson, Ehsan Adeli, Russ Altman, Simran Arora, Sydney von
Arx, Michael S Bernstein, Jeannette Bohg, Antoine Bosselut, Emma Brunskill, et al. On the
opportunities and risks of foundation models. arXiv preprint arXiv:2108.07258, 2021.
[8] Xiaohua Zhai, Alexander Kolesnikov, Neil Houlsby, and Lucas Beyer. Scaling vision transform-
ers. CoRR, abs/2106.04560, 2021. URL https://arxiv.org/abs/2106.04560.
[9] David Silver, Aja Huang, Chris J Maddison, Arthur Guez, Laurent Sifre, George Van Den Driess-
che, Julian Schrittwieser, Ioannis Antonoglou, Veda Panneershelvam, Marc Lanctot, et al. Mas-
tering the game of go with deep neural networks and tree search. Nature, 529(7587):484–489,
2016.
[10] Oriol Vinyals, Igor Babuschkin, Wojciech M Czarnecki, Michaël Mathieu, Andrew Dudzik, Jun-
young Chung, David H Choi, Richard Powell, Timo Ewalds, Petko Georgiev, et al. Grandmaster
level in starcraft ii using multi-agent reinforcement learning. Nature, 575(7782):350–354, 2019.
10


[11] Richard S Sutton and Andrew G Barto. Reinforcement learning: An introduction. MIT press,
2018.
[12] Christopher Berner, Greg Brockman, Brooke Chan, Vicki Cheung, Przemysław D˛
ebiak, Christy
Dennison, David Farhi, Quirin Fischer, Shariq Hashme, Chris Hesse, et al. Dota 2 with large
scale deep reinforcement learning. arXiv preprint arXiv:1912.06680, 2019.
[13] Bowen Baker, Ingmar Kanitscheider, Todor Markov, Yi Wu, Glenn Powell, Bob McGrew,
and Igor Mordatch.
Emergent tool use from multi-agent autocurricula.
arXiv preprint
arXiv:1909.07528, 2019.
[14] Max Jaderberg, Wojciech M Czarnecki, Iain Dunning, Luke Marris, Guy Lever, Antonio Garcia
Castaneda, Charles Beattie, Neil C Rabinowitz, Ari S Morcos, Avraham Ruderman, et al.
Human-level performance in 3d multiplayer games with population-based reinforcement learn-
ing. Science, 364(6443):859–865, 2019.
[15] Adrià Puigdomènech Badia, Bilal Piot, Steven Kapturowski, Pablo Sprechmann, Alex Vitvit-
skyi, Zhaohan Daniel Guo, and Charles Blundell. Agent57: Outperforming the atari human
benchmark. In International Conference on Machine Learning, pages 507–517. PMLR, 2020.
[16] Marc Bellemare, Sriram Srinivasan, Georg Ostrovski, Tom Schaul, David Saxton, and Remi
Munos.
Unifying count-based exploration and intrinsic motivation.
Advances in neural
information processing systems, 29, 2016.
[17] Yuri Burda, Harrison Edwards, Amos Storkey, and Oleg Klimov. Exploration by random
network distillation. arXiv preprint arXiv:1810.12894, 2018.
[18] Adrien Ecoffet, Joost Huizinga, Joel Lehman, Kenneth O Stanley, and Jeff Clune. First return,
then explore. Nature, 590(7847):580–586, 2021.
[19] Peter C Humphreys, David Raposo, Toby Pohlen, Gregory Thornton, Rachita Chhaparia, Alistair
Muldal, Josh Abramson, Petko Georgiev, Alex Goldin, Adam Santoro, et al. A data-driven
approach for learning to control computers. arXiv preprint arXiv:2202.08137, 2022.
[20] Tianlin Shi, Andrej Karpathy, Linxi Fan, Jonathan Hernandez, and Percy Liang. World of
bits: An open-domain platform for web-based agents. In Doina Precup and Yee Whye Teh,
editors, Proceedings of the 34th International Conference on Machine Learning, volume 70 of
Proceedings of Machine Learning Research, pages 3135–3144. PMLR, 06–11 Aug 2017. URL
https://proceedings.mlr.press/v70/shi17a.html.
[21] Andrew Y Ng, Stuart J Russell, et al. Algorithms for inverse reinforcement learning. In Icml,
volume 1, page 2, 2000.
[22] Faraz Torabi, Garrett Warnell, and Peter Stone. Recent advances in imitation learning from
observation. arXiv preprint arXiv:1905.13566, 2019.
[23] Jonathan Ho and Stefano Ermon. Generative adversarial imitation learning. Advances in neural
information processing systems, 29, 2016.
[24] Faraz Torabi, Garrett Warnell, and Peter Stone. Behavioral cloning from observation. arXiv
preprint arXiv:1805.01954, 2018.
[25] YuXuan Liu, Abhishek Gupta, Pieter Abbeel, and Sergey Levine. Imitation from observation:
Learning to imitate behaviors from raw video via context translation. In 2018 IEEE International
Conference on Robotics and Automation (ICRA), pages 1118–1125. IEEE, 2018.
[26] Twinfinite
Staff.
Most
played
games
in
2021,
ranked
by
peak
concur-
rent
players.
Twinfinite.
URL
https://twinfinite.net/2021/12/
most-played-games-in-2020-ranked-by-peak-concurrent-players/.
[27] William H Guss, Brandon Houghton, Nicholay Topin, Phillip Wang, Cayden Codel, Manuela
Veloso, and Ruslan Salakhutdinov. Minerl: A large-scale dataset of minecraft demonstrations.
arXiv preprint arXiv:1907.13440, 2019.
11


[28] Chen Tessler, Shahar Givony, Tom Zahavy, Daniel Mankowitz, and Shie Mannor. A deep
hierarchical approach to lifelong learning in minecraft. In Proceedings of the AAAI Conference
on Artificial Intelligence, volume 31, 2017.
[29] Christian Scheller, Yanick Schraner, and Manfred Vogel. Sample efficient reinforcement
learning through learning from demonstrations in minecraft. In NeurIPS 2019 Competition and
Demonstration Track, pages 67–76. PMLR, 2020.
[30] Ingmar Kanitscheider, Joost Huizinga, David Farhi, William Hebgen Guss, Brandon Houghton,
Raul Sampedro, Peter Zhokhov, Bowen Baker, Adrien Ecoffet, Jie Tang, et al. Multi-task
curriculum learning in a complex, visual, hard-exploration domain: Minecraft. arXiv preprint
arXiv:2106.14876, 2021.
[31] Junhyuk Oh, Valliappa Chockalingam, Honglak Lee, et al. Control of memory, active perception,
and action in minecraft. In International Conference on Machine Learning, pages 2790–2799.
PMLR, 2016.
[32] Vihang P Patil, Markus Hofmarcher, Marius-Constantin Dinu, Matthias Dorfer, Patrick M Blies,
Johannes Brandstetter, Jose A Arjona-Medina, and Sepp Hochreiter. Align-rudder: Learning
from few demonstrations by reward redistribution. arXiv preprint arXiv:2009.14108, 2020.
[33] Alexey Skrynnik, Aleksey Staroverov, Ermek Aitygulov, Kirill Aksenov, Vasilii Davydov, and
Aleksandr I Panov. Forgetful experience replay in hierarchical reinforcement learning from
demonstrations. arXiv preprint arXiv:2006.09939, 2020.
[34] Zichuan Lin, Junyou Li, Jianing Shi, Deheng Ye, Qiang Fu, and Wei Yang.
Juewu-mc:
Playing minecraft with sample-efficient hierarchical reinforcement learning. arXiv preprint
arXiv:2112.04907, 2021.
[35] Dean A Pomerleau. Alvinn: An autonomous land vehicle in a neural network. Advances in
neural information processing systems, 1, 1988.
[36] Stefan Schaal. Is imitation learning the route to humanoid robots? Trends in cognitive sciences,
3(6):233–242, 1999.
[37] Brenna D Argall, Sonia Chernova, Manuela Veloso, and Brett Browning. A survey of robot
learning from demonstration. Robotics and autonomous systems, 57(5):469–483, 2009.
[38] Ahmed Hussein, Mohamed Medhat Gaber, Eyad Elyan, and Chrisina Jayne. Imitation learning:
A survey of learning methods. ACM Computing Surveys (CSUR), 50(2):1–35, 2017.
[39] Claude Sammut, Scott Hurst, Dana Kedzier, and Donald Michie. Learning to fly. In Machine
Learning Proceedings 1992, pages 385–393. Elsevier, 1992.
[40] Alessandro Giusti, Jérôme Guzzi, Dan C Cire¸
san, Fang-Lin He, Juan P Rodríguez, Flavio
Fontana, Matthias Faessler, Christian Forster, Jürgen Schmidhuber, Gianni Di Caro, et al. A
machine learning approach to visual perception of forest trails for mobile robots. IEEE Robotics
and Automation Letters, 1(2):661–667, 2015.
[41] Mariusz Bojarski, Davide Del Testa, Daniel Dworakowski, Bernhard Firner, Beat Flepp, Prasoon
Goyal, Lawrence D Jackel, Mathew Monfort, Urs Muller, Jiakai Zhang, et al. End to end learning
for self-driving cars. arXiv preprint arXiv:1604.07316, 2016.
[42] Felipe Codevilla, Matthias Müller, Antonio López, Vladlen Koltun, and Alexey Dosovitskiy.
End-to-end driving via conditional imitation learning. In 2018 IEEE international conference
on robotics and automation (ICRA), pages 4693–4700. IEEE, 2018.
[43] Rémi Coulom. Computing “elo ratings” of move patterns in the game of go. ICGA journal, 30
(4):198–208, 2007.
[44] Todd Hester, Matej Vecerik, Olivier Pietquin, Marc Lanctot, Tom Schaul, Bilal Piot, Dan Horgan,
John Quan, Andrew Sendonaris, Ian Osband, et al. Deep q-learning from demonstrations. In
Proceedings of the AAAI Conference on Artificial Intelligence, volume 32, 2018.
12


[45] Ashley Edwards, Himanshu Sahni, Yannick Schroecker, and Charles Isbell. Imitating latent
policies from observation. In International conference on machine learning, pages 1755–1763.
PMLR, 2019.
[46] Xue Bin Peng, Angjoo Kanazawa, Jitendra Malik, Pieter Abbeel, and Sergey Levine. Sfv:
Reinforcement learning of physical skills from videos. ACM Transactions On Graphics (TOG),
37(6):1–14, 2018.
[47] Feryal Behbahani, Kyriacos Shiarlis, Xi Chen, Vitaly Kurin, Sudhanshu Kasewa, Ciprian Stirbu,
Joao Gomes, Supratik Paul, Frans A Oliehoek, Joao Messias, et al. Learning from demonstration
in the wild. In 2019 International Conference on Robotics and Automation (ICRA), pages
775–781. IEEE, 2019.
[48] Yusuf Aytar, Tobias Pfaff, David Budden, Thomas Paine, Ziyu Wang, and Nando De Fre-
itas. Playing hard exploration games by watching youtube. Advances in neural information
processing systems, 31, 2018.
[49] Deepak Pathak, Parsa Mahmoudieh, Guanghao Luo, Pulkit Agrawal, Dian Chen, Yide Shentu,
Evan Shelhamer, Jitendra Malik, Alexei A. Efros, and Trevor Darrell. Zero-shot visual imitation.
In ICLR, 2018.
[50] Ashvin Nair, Dian Chen, Pulkit Agrawal, Phillip Isola, Pieter Abbeel, Jitendra Malik, and Sergey
Levine. Combining self-supervised learning and imitation for vision-based rope manipulation.
pages 2146–2153, 05 2017. doi: 10.1109/ICRA.2017.7989247.
[51] Duy Nguyen-Tuong, Jan Peters, Matthias Seeger, and Bernhard Schölkopf. Learning inverse
dynamics: a comparison. In European symposium on artificial neural networks, number CONF,
2008.
[52] David Abel, Alekh Agarwal, Fernando Diaz, Akshay Krishnamurthy, and Robert E Schapire.
Exploratory gradient boosting for reinforcement learning in complex domains. arXiv preprint
arXiv:1603.04119, 2016.
[53] Dilip Arumugam, Jun Ki Lee, Sophie Saskin, and Michael L Littman. Deep reinforcement
learning from policy-dependent human feedback. arXiv preprint arXiv:1902.04257, 2019.
[54] Alexander Trott, Stephan Zheng, Caiming Xiong, and Richard Socher. Keeping your dis-
tance: Solving sparse reward tasks using self-balancing shaped rewards. Advances in Neural
Information Processing Systems, 32, 2019.
[55] Stephan Alaniz. Deep reinforcement learning with model learning and monte carlo tree search
in minecraft. arXiv preprint arXiv:1803.08456, 2018.
[56] Hiroto Udagawa, Tarun Narasimhan, and Shim-Young Lee. Fighting zombies in minecraft with
deep reinforcement learning. Technical report, Technical report, Technical report, Stanford
University, 2016.
[57] Tianmin Shu, Caiming Xiong, and Richard Socher. Hierarchical and interpretable skill acquisi-
tion in multi-task reinforcement learning. arXiv preprint arXiv:1712.07294, 2017.
[58] Junhyuk Oh, Satinder Singh, Honglak Lee, and Pushmeet Kohli. Zero-shot task generalization
with multi-task deep reinforcement learning. In International Conference on Machine Learning,
pages 2661–2670. PMLR, 2017.
[59] Zhengxiang Shi, Yue Feng, and Aldo Lipani. Learning to execute or ask clarification questions.
arXiv preprint arXiv:2204.08373, 2022.
[60] Tambet Matiisen, Avital Oliver, Taco Cohen, and John Schulman. Teacher–student curriculum
learning. IEEE transactions on neural networks and learning systems, 31(9):3732–3740, 2019.
[61] Robert George Douglas Steel, James Hiram Torrie, et al. Principles and procedures of statistics.
Principles and procedures of statistics., 1960.
13


[62] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image
recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition,
pages 770–778, 2016.
[63] Jared Kaplan, Sam McCandlish, Tom Henighan, Tom B Brown, Benjamin Chess, Rewon Child,
Scott Gray, Alec Radford, Jeffrey Wu, and Dario Amodei. Scaling laws for neural language
models. arXiv preprint arXiv:2001.08361, 2020.
[64] Karl W Cobbe, Jacob Hilton, Oleg Klimov, and John Schulman. Phasic policy gradient. In
Marina Meila and Tong Zhang, editors, Proceedings of the 38th International Conference on
Machine Learning, volume 139 of Proceedings of Machine Learning Research, pages 2020–
2027. PMLR, 18–24 Jul 2021. URL https://proceedings.mlr.press/v139/cobbe21a.
html.
[65] Dhireesha Kudithipudi, Mario Aguilar-Simon, Jonathan Babb, Maxim Bazhenov, Douglas
Blackiston, Josh Bongard, Andrew P Brna, Suraj Chakravarthi Raja, Nick Cheney, Jeff Clune,
et al. Biological underpinnings for lifelong learning machines. Nature Machine Intelligence, 4
(3):196–210, 2022.
[66] James Kirkpatrick, Razvan Pascanu, Neil Rabinowitz, Joel Veness, Guillaume Desjardins,
Andrei A Rusu, Kieran Milan, John Quan, Tiago Ramalho, Agnieszka Grabska-Barwinska, et al.
Overcoming catastrophic forgetting in neural networks. Proceedings of the national academy of
sciences, 114(13):3521–3526, 2017.
[67] Emily M Bender, Timnit Gebru, Angelina McMillan-Major, and Shmargaret Shmitchell. On
the dangers of stochastic parrots: Can language models be too big???. In Proceedings of the
2021 ACM Conference on Fairness, Accountability, and Transparency, pages 610–623, 2021.
[68] Fabian Pedregosa, Gaël Varoquaux, Alexandre Gramfort, Vincent Michel, Bertrand Thirion,
Olivier Grisel, Mathieu Blondel, Peter Prettenhofer, Ron Weiss, Vincent Dubourg, et al. Scikit-
learn: Machine learning in python. the Journal of machine Learning research, 12:2825–2830,
2011.
[69] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez,
Łukasz Kaiser, and Illia Polosukhin. Attention is all you need. Advances in neural information
processing systems, 30, 2017.
[70] Jimmy Lei Ba, Jamie Ryan Kiros, and Geoffrey E Hinton. Layer normalization. arXiv preprint
arXiv:1607.06450, 2016.
[71] Yann A LeCun, Léon Bottou, Genevieve B Orr, and Klaus-Robert Müller. Efficient backprop.
In Neural networks: Tricks of the trade, pages 9–48. Springer, 2012.
[72] Diederik P Kingma and Jimmy Ba. Adam: A method for stochastic optimization. arXiv preprint
arXiv:1412.6980, 2014.
[73] Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan,
Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, Alban Desmaison, Andreas
Kopf, Edward Yang, Zachary DeVito, Martin Raison, Alykhan Tejani, Sasank Chilamkurthy,
Benoit Steiner, Lu Fang, Junjie Bai, and Soumith Chintala. Pytorch: An imperative style, high-
performance deep learning library. In H. Wallach, H. Larochelle, A. Beygelzimer, F. d'Alché-
Buc, E. Fox, and R. Garnett, editors, Advances in Neural Information Processing Systems 32,
pages 8024–8035. Curran Associates, Inc., 2019. URL http://papers.neurips.cc/paper/
9015-pytorch-an-imperative-style-high-performance-deep-learning-library.
pdf.
[74] Yann N Dauphin, Razvan Pascanu, Caglar Gulcehre, Kyunghyun Cho, Surya Ganguli, and
Yoshua Bengio. Identifying and attacking the saddle point problem in high-dimensional non-
convex optimization. Advances in neural information processing systems, 27, 2014.
[75] Jason Yosinski, Jeff Clune, Yoshua Bengio, and Hod Lipson. How transferable are features in
deep neural networks? Advances in neural information processing systems, 27, 2014.
14


[76] Zihang Dai, Zhilin Yang, Yiming Yang, Jaime Carbonell, Quoc V Le, and Ruslan Salakhutdinov.
Transformer-xl: Attentive language models beyond a fixed-length context. arXiv preprint
arXiv:1901.02860, 2019.
[77] John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal
policy optimization algorithms. arXiv preprint arXiv:1707.06347, 2017.
[78] John Schulman, Philipp Moritz, Sergey Levine, Michael Jordan, and Pieter Abbeel. High-
dimensional continuous control using generalized advantage estimation.
arXiv preprint
arXiv:1506.02438, 2015.
[79] Ronald J Williams. Simple statistical gradient-following algorithms for connectionist reinforce-
ment learning. Machine learning, 8(3):229–256, 1992.
[80] Volodymyr Mnih, Adria Puigdomenech Badia, Mehdi Mirza, Alex Graves, Timothy Lillicrap,
Tim Harley, David Silver, and Koray Kavukcuoglu. Asynchronous methods for deep reinforce-
ment learning. In International conference on machine learning, pages 1928–1937. PMLR,
2016.
[81] Marcin Andrychowicz, Filip Wolski, Alex Ray, Jonas Schneider, Rachel Fong, Peter Welinder,
Bob McGrew, Josh Tobin, OpenAI Pieter Abbeel, and Wojciech Zaremba. Hindsight experience
replay. Advances in neural information processing systems, 30, 2017.
[82] Tom Schaul, Daniel Horgan, Karol Gregor, and David Silver. Universal value function ap-
proximators. In International conference on machine learning, pages 1312–1320. PMLR,
2015.
[83] Open Ended Learning Team, Adam Stooke, Anuj Mahajan, Catarina Barros, Charlie Deck,
Jakob Bauer, Jakub Sygnowski, Maja Trebacz, Max Jaderberg, Michael Mathieu, et al. Open-
ended learning leads to generally capable agents. arXiv preprint arXiv:2107.12808, 2021.
[84] Jelena Luketina, Nantas Nardelli, Gregory Farquhar, Jakob Foerster, Jacob Andreas, Edward
Grefenstette, Shimon Whiteson, and Tim Rocktäschel.
A survey of reinforcement learn-
ing informed by natural language. In Proceedings of the Twenty-Eighth International Joint
Conference on Artificial Intelligence, IJCAI-19, pages 6309–6317. International Joint Confer-
ences on Artificial Intelligence Organization, 7 2019. doi: 10.24963/ijcai.2019/880. URL
https://doi.org/10.24963/ijcai.2019/880.
[85] DeepMind Interactive Agents Team, Josh Abramson, Arun Ahuja, Arthur Brussee, Federico
Carnevale, Mary Cassin, Felix Fischer, Petko Georgiev, Alex Goldin, Tim Harley, et al. Creating
multimodal interactive agents with imitation and self-supervised learning. arXiv preprint
arXiv:2112.03763, 2021.
[86] Aditya Ramesh, Prafulla Dhariwal, Alex Nichol, Casey Chu, and Mark Chen. Hierarchical
text-conditional image generation with clip latents. arXiv preprint arXiv:2204.06125, 2022.
[87] Dong Yu and Li Deng. Automatic speech recognition, volume 1. Springer, 2016.
[88] Daulet Nurmanbetov. rpunct, May 25 2021. URL https://github.com/Felflare/rpunct.
accessed 2022-04-22.
[89] Arvind Neelakantan, Tao Xu, Raul Puri, Alec Radford, Jesse Michael Han, Jerry Tworek,
Qiming Yuan, Nikolas Tezak, Jong Wook Kim, Chris Hallacy, et al. Text and code embeddings
by contrastive pre-training. arXiv preprint arXiv:2201.10005, 2022.
Acknowledgements
We thank the following people for helpful discussions and support: Bob McGrew, Ken Stanley,
Joel Lehman, Ilya Sutskever, Wojciech Zaremba, Ingmar Kanitscheider, David Farhi, Glenn Powell,
Jonathan Gordon, and the OpenAI supercomputing team, especially Christian Gibson, Ben Chess,
and Christopher Berner.
15


Supplementary Information
A
Collecting Internet Data
A.1
Initial Unclean Dataset Curation
Our goal was to curate a video dataset of Minecraft gameplay from the survival game mode. Addition-
ally, we prefer the data come from game modes as close as possible to our evaluation environment,
meaning preferably coming from Minecraft version 1.16, being on a computer (which uses a mouse
and keyboard vs. video game controllers with keypads and other buttons), being single- (vs. multi-)
player, and having the default look of the game (vs. modifications that alter that style, such as to
make it look realistic). To try to accomplish these goals, we collect a dataset by performing keyword
searches of publicly available videos on the internet. A list of search queries we used are given in
Table 1.
minecraft survival longplay
minecraft gameplay no webcam
minecraft gameplay survival mode
minecraft survival tutorial
minecraft survival guide
minecraft survival let’s play
minecraft survival for beginners
minecraft beginners guide
ultimate minecraft starter guide
minecraft survival guide 1.16
minecraft how to start a new survival world
minecraft survival fresh start
minecraft survival let’s play episode 1
let’s play minecraft episode 1
minecraft survival 101
minecraft survival learning to play
how to play minecraft survival
how to play minecraft
minecraft survival basic
minecraft survival for noobs
minecraft survival for dummies
how to play minecraft for beginners
minecraft survival tutorial series
minecraft survival new world
minecraft survival a new beginning
minecraft survival episodio 1
minecraft survival epizod 1
minecraft survival 1. bölüm
i made a new minecraft survival world
Table 1: Search terms used for generating the initial web dataset.
For videos that have metadata available, we perform an additional step of metadata-based filtering
to eliminate videos that do not fit our target distribution. In this step, we look for a list of blacklist
keywords in the video title and description and reject videos that contain these terms. The blacklist
keywords we use are: {ps3, ps4, ps5, xbox 360, playstation, timelapse, multiplayer, minecraft pe,
pocket edition, skyblock, realistic minecraft, how to install, how to download, realmcraft, animation}.
This process yielded us ∼270k hours of unlabeled data, which we filter down to only a “clean” subset
as described in the next section.
A.2
Training a Model to Filter out Unclean Video Segments
We restrict the scope of this work to the Minecraft Survival game mode and therefore limit our
training dataset to clips that are obtained from this mode that are relatively free from visual artifacts.
16


To do so, we asked contractors to label a set of random video frames (images) from Minecraft videos
(N=8800). These images were from a random subset of the videos we collected toward the beginning
of the project (Section A.1).
A.2.1
Label Collection
We asked 5 workers on Amazon Mechanical Turk (mTurk) that we selected with a sample qualification
task to label random screen capture images to be used in training the classifier. A sample worker
interface that the workers saw on mTurk is given in Figure 10.
We asked workers to label videos as being in one of the following three categories (see Figure 11 for
visual examples of each class):
1. Minecraft Survival Mode - No Artifacts: Video frames (images) that correspond
to the Minecraft Survival game mode that do not contain any non-game visual artifacts (e.g.
subscribe buttons, channel logos, advertisements, picture-in-picture of the narrator, etc.).
2. Minecraft Survival Mode - with Artifacts:
Video frames (images) of the
Minecraft Survival game mode that include such visual artifacts.
3. None of the Above: Video frames (images) that are not from the Minecraft survival
game mode, including those from other Minecraft game modes such as creative mode or
even other games/topics entirely.
The full set of instructions workers received are as follows (note that we also included multiple image
examples from each category in the worker instructions, similar to the sample subset provided in
Figure 11):
Please help us identify screenshots that belong only to the survival mode in Minecraft. Everything
else (Minecraft creative mode, other games, music videos, etc.) should be marked as None of the
above. Survival mode is identified by the info at the bottom of the screen:
• a health bar (row of hearts)
• a hunger bar (row of chicken drumsticks)
• a bar showing items held
Survival Mode
Valid survival mode videos have health/hunger bars and an item hotbar at the bottom of the screen.
Creative Mode
Creative mode only has an item hotbar and should be classified as None of the Above.
Label Descriptions
• Minecraft Survival Mode - No Artifacts: These images will be clean screenshots
from the Minecraft survival mode gameplay without any noticeable artifacts.
• Minecraft Survival Mode - with Artifacts: These images will be valid survival
mode screenshots, but with some added artifacts. Typical artifacts may include image
overlays (a logo/brand), text annotations, a picture-in-picture of the player, etc.
• None of the Above: Use this category when the image is not a valid Minecraft survival
screenshot. It may be a non-Minecraft frame or from a different game mode. In non-survival
game modes such as the creative mode, the health/hunger bars will be missing from the
image, the item hotbar may or may not be still present.
In total, we spent $319.96 on human labeling experiments on mTurk, of which $159.98 was directly
paid to workers. The remaining amount was spent towards Amazon platform fees. The workers
received $0.01 per labeled image, at an hourly compensation of $7.20 (based on an estimated labeling
time of 5 seconds/image – in our internal sample run of the same task, we found the average labeling
time to be < 3 seconds).
Since we perform rigorous keyword and metadata based filtering of videos (as described in A.1) from
which we served sample images to be labeled, serving offensive content to workers was extremely
17


low risk and no such images were detected during our manual checks. We only collected labels
during our experiment, and the workers were fully anonymized via the mTurk platform, therefore no
personally identifiable information (PII) was collected.
Figure 10: Amazon Mechanical Turk worker interface showing an example labeling task
Figure 11: (Left) Sample image for Class 1: Minecraft Survival Mode - No Artifacts.
(Middle) Sample image for Class 2: Minecraft Survival Mode - with Artifacts – Image
contains annotations and picture-in-picture of the narrator. (Right) Sample image for Class 3: None
of the Above – Image is missing the hotbar as well as health and armor bars, indicating that it was
not captured during survival mode gameplay
A.2.2
SVM Training
With the image labels collected as described in the previous section, we trained a classifier to extract
video segments that consist of frames from the Minecraft Survival Mode - No Artifacts
category. Given a set of labeled images, we obtain embeddings for each image using the RN50x64
ResNet CLIP Model.6 This is a ResNet-based CLIP model that is scaled up to have approximately
64x the compute of a ResNet-50. We then train a Support Vector Machine (SVM) using the RBF
kernel to obtain a frame classifier. We use the Scikit-learn68 SVM implementation with the parameter
configuration given in Table 2.
Finally, we apply the classifier to frames of raw video sequences at a rate of 3 frames/second. We filter
for videos that consist of at least 80% "clean" frames at this stage (Classes Minecraft Survival
Mode - with Artifacts and None of the Above are both considered not clean). From this set,
we apply a median filter (with a kernel size of 7) to the labels and segment videos by splitting the
"clean" segments that are at least 5s in duration. The result of this is our final web_clean dataset.
A.3
early_game Dataset
The early_game dataset is a ∼3000 hour subset of web_clean targeted at “early game” Minecraft
behavior, i.e. instances where players start in a fresh world with no items. We obtain the metadata
text that accompanies the videos in web_clean and determine whether any of the following regular
expressions match:
18


CLIP Model Specification
RN50x64 (see text)
CLIP Input Image Resolution
448x448x3
CLIP Embedding Feature Length
1024
SVM Parameters
Kernel
rbf
C
20
Gamma
scale
Sample Size
Class 1
2200
Class 2
2200
Class 3
4400
Table 2: Feature Extraction Details and SVM Configuration. The parameters are for the SVM
implementation in Scikit-learn68.
• (ep|episode|eps|day|session|sesh|chapter|chap
.|series|part|parte|pt|round|day|tâ
.p|bölüm|episodio|epizod|pizod)(
)*(\.1|#1|1|\.01|#01|01|one[ˆ0-9]|$)
• start
• beginning
• (new|fresh|clean).*(world|game|play)
• from scratch
From this set of videos, we take only the first 5 minutes of each video.
B
Contractor Data
B.1
Recording Contractor Play
Our contractors use a custom Minecraft recorder that we built that records their actions and game video
feeds as they play. The recorder is implemented using the MCP-Reborn (github.com/Hexeption/MCP-
Reborn) modding package. To ensure that the recorder environment is as close as possible to
the Minecraft environment used for RL rollouts and evaluations (Appendix C), we use the same
underlying game engine for both. The recorder is a Java app that runs in a window mode, with
constant resolution of 1280x760. Brightness is set to 0 (the "gloomy" setting in Minecraft), which
is the default setting. Other graphics settings (field of view, GUI scale) are fixed to the values used
in the Minecraft environment (C.1); we explicitly prevented users from changing graphics settings.
Unlike the environment, the recorder allows all keyboard key presses and continuous (as opposed to
binned) mouse actions. On every game step (or “tick”) the frame buffer used to display the game
window is downsized to 640x360 and written into a video file. In-game actions are recorded in a
separate JSONL file (a text file where each line is a JSON-formatted string). All recordings are
chunked into 5 minute clips: after each 5 minute segment of contractor game play the recorder
automatically uploads the video file, the JSONL file with actions, as well as a Minecraft state file.
To ensure that contractors cannot corrupt each other’s data, we provided every contractor with an
individual cloud bucket, as well as with credentials giving write access only to that bucket. Credentials
also included adjective-adjective-noun names (e.g. grumpy-amethyst-chipmunk), generated with the
namegenerator python package to ensure contractor anonymity when we publish the data.
B.2
Contractor Contract
We recruited contractors by posting the following offer on the UpWork freelancing platform.
“We are collecting data for training AI models in Minecraft. You’ll need to install
java, download the modified version of Minecraft (that collects and uploads your
play data), and play Minecraft survival mode! Paid per hour of gameplay. Prior
experience in Minecraft not necessary. We do not collect any data that is unrelated
to Minecraft from your computer.”
19


We had the applications open for a day, and then randomly selected 10 applicants for the first round of
contractors. Later in the project, as we needed more data and as some contractors asked to terminate
their contracts, we added more applicants from the original pool as well as referrals from the currently
working contractors. The contractors were paid $20 per hour (minus Upwork platform fees and
applicable taxes). All of the results presented in this paper are based on about 4,500 hours of data
(including data recorded to gather statistics of human play that was not used for training), which cost
us around $90,000. Over the course of the project, we collected some data we did not use due to
bugs in the recorder and for some ideas we ultimately did not pursue. In total, we spent about $160k
for contractor compensation over the course of the project. However, as we discuss in Sec. 4.6, we
could likely obtain most of our results with an IDM trained using only $2000 worth of data, i.e. the
foundation VPT model, BC fine-tuning to the earlygame_keyword dataset, and the RL fine-tuning
results. Collecting the contractor_house dataset cost about $8000. Because we used the IDM
trained on about 2000 hours of contractor data, the actual cost of contractor data for those results was
around $40,000.
In early stages of the project, we were planning to use contractor data solely for the purpose of training
the IDM. As such, no specific tasks were given, other than “play the survival mode of Minecraft like
you normally would.” Later in the project, we requested that contractors perform specific tasks in
Minecraft, such as:
• Collect as many units of wood as possible, using only wooden or stone tools (treechop)
• Start a new world every 30 minutes of game play
• Build a basic house in 10 minutes using only dirt, wood, sand, and either wooden or stone
tools (contractor_house, more details below in Appendix B.4).
• Starting from a new world and an empty inventory, find resources and craft a diamond
pickaxe in 20 minutes (obtain_diamond_pickaxe). This dataset was used to obtain
statistics for how long it takes humans on average to complete this task (and the subtasks
required to complete it) when obtaining a diamond pickaxe is their goal.
Since we only recorded in-game events and videos, the data does not include personally identifiable
information. That being said, the contractors could theoretically use Minecraft’s open-world property
to generate personally identifiable information and/or offensive content (e.g. by using Minecraft
blocks to write their name or offensive messages, then finding a spot from which the message would
be visible). In practice, we have not seen any attempts to do so in the contractor videos that we
watched. Of course, we train our BC models on videos from the internet of people playing Minecraft,
and if such behavior is in those videos our model could also potentially learn it, although we expect
such behavior is rare enough that our model would not be likely to reproduce it.
B.3
Data for the Inverse Dynamics Model.
Since the IDM’s task is to infer actions given the video, any labelled data is appropriate for IDM
training. In practice, we included general gameplay as well as the treechop task data described
in the previous section, which amounted to a total of 1962 hours. Due to collecting datasets like
contractor_house only at late stages of the project, they were not included in IDM training.
B.4
contractor_house.
The contractor_house contains about 420 hours of data. We asked contractors to build a basic
house in 10 minutes, using only basic dirt, wood, and sand, blocks. Each trajectory starts in a newly
generated world and a timer forcibly ends a trajectory after a 20 minute time limit. For this task, many
contractors chose to begin their trajectories by crafting basic tools and building blocks, specifically
it was common for the first 2 minutes to be spent crafting a wooden pickaxe and then mining stone
for an assortment of stone tools before gathering more building blocks and beginning to create their
structure.
C
Minecraft environment details
Our Minecraft training environment is a hybrid between MineRL27 and the MCP-Reborn
(github.com/Hexeption/MCP-Reborn) Minecraft modding package. Unlike the regular Minecraft
20


Figure 12: (Left) Sample of a Minecraft frame in the original resolution (640x360) with an in-game
GUI open. The mouse cursor can be seen in the center of the image. This particular GUI shows the
player’s inventory and can be used to craft very basic items. (Middle) We downsample images to
128x128 for computational reasons. Shown is a downsampled observation with an in-game GUI for
crafting. This is the resolution used by our models. (Right) A 128x128 observation as seen by our
models without in-game GUI. The health, hunger, hotbar overlays, and agent hand can be seen in the
lower part of the image.
game, in which the server (or the "world") always runs at 20Hz and the client runs as fast as rendering
can complete (typically at 60-100Hz), in our version the client and server run in the same thread
at the same frequency. This allows us to run the environment slower or faster than real time, while
avoiding artifacts like missing chunks of the world. The action and observation spaces are similar
to those of MineRL environments and are described in more detail in the following subsections.
The environment also returns diagnostic information, such as in-game stats, contents of the agent’s
inventory, whether any in-game GUI is open, etc., which we use for tracking and recording but not as
inputs to the models. The episode length is 10 minutes for RL experiments and 60 minutes for BC
model evaluations. The agent can "die" in a number of ways, such as staying under water for too long
and drowning, being killed by hostile mobs, or falling from a tall structure. We do not terminate the
episode on agent "death". Instead, just as for humans in the regular Minecraft game, the agent drops
all its items when it dies and respawns at a random spot close to the initial spawning spot in the same
Minecraft world. The policy state is not masked on death, so the model can remember the fact that it
has died and act accordingly.
C.1
Observation space
The environment observations are simply the raw pixels from the Minecraft game that a human
would see. Unlike MineRL, we do not remove overlays like the hotbar, health indicators, and the
animation of a moving hand shown in response to the attack or “use” actions. The field of view is
70 degrees, which corresponds to the Minecraft default. GUI scale (a parameter controlling the size
of the in-game GUI) is set to 2, and brightness is set to 2 (which is not a Minecraft default, but is
very frequently used in online videos). The rendering resolution is 640x360, which is downsampled
to 128x128 before being input to the models. We empirically found 128x128 to be the smallest
resolution for which in-game GUI elements are still discernible, and then chose that to minimize
compute costs. Whenever an in-game GUI is open, we additionally render an image of a mouse
cursor at the appropriate mouse position to match what a human player’s operating system does (Fig.
12).
C.2
Action space
Our action space includes almost all actions directly available to human players, such as keypresses,
mouse movements, and clicks. The specific binary actions we include are shown in Table 3.
One difference between the human action space and our agent’s is that we disallow typing arbitrary
letters, which is only useful for entering text into the search bar of the crafting recipe book. Humans
can either do that or browse the recipe book with the mouse, the latter of which our agent can still
do. However, because we do allow the agent to press letters that are also shortcuts for actions (e.g.
outside of the GUI, the "W" key triggers the forward action) agents are able to press a few keys
within the GUI (W, A, S, D, E, Q) that produce letters if the recipe book search bar is selected. We
have not seen agents attempt to search the recipe book with these letters. Instead, our agents navigate
the recipe book with the mouse or craft by dragging items around the crafting window.
21


Action
Human action
Description
forward
W key
Move forward.
back
S key
Move backward.
left
A key
Strafe left.
right
D key
Strafe right.
jump
space key
Jump.
inventory
E key
Open or close inventory and the 2x2 crafting grid.
sneak
shift key
Move carefully in current direction of motion. In the
GUI it acts as a modifier key: when used with attack
it moves item from/to the inventory to/from the hot-
bar, and when used with craft it crafts the maximum
number of items possible instead of just 1.
sprint
ctrl key
Move fast in the current direction of motion.
attack
left mouse button
Attack; In GUI, pick up the stack of items or place the
stack of items in a GUI cell; when used as a double
click (attack - no attack - attack sequence), collect all
items of the same kind present in inventory as a single
stack.
use
right mouse button
Place the item currently held or use the block the player
is looking at. In GUI, pick up the stack of items or place
a single item from a stack held by mouse.
drop
Q key
Drop a single item from the stack of items the player
is currently holding. If the player presses ctrl-Q then
it drops the entire stack. In the GUI, the same thing
happens except to the item the mouse is hovering over.
hotbar.[1-9]
keys 1 – 9
Switch active item to the one in a given hotbar cell.
Table 3: Binary actions included in the action space. https://minecraft.fandom.com/wiki/
Controls has more detailed descriptions of each action.
In addition to the binary (on/off) keypress actions, our action space also includes mouse movements.
As with human gameplay, when in-game GUIs are not open, mouse X and Y actions change the
agent’s yaw and pitch, respectively. When a GUI is open, camera actions move the mouse cursor.
Mouse movements are relative (i.e. they move the mouse or camera relative to the current position,
and thus their effect depends on the current position).
Inventory interaction in Minecraft requires fine-grained mouse movements to achieve tasks such as
crafting and smelting, while mining and navigating the world can be achieved with coarser mouse
action. To be able to achieve both with the same action space, we implemented mouse movements
as a set of discrete actions with foveated binning along each axis (Fig. 13), which in preliminary
experiments we found to improve crafting performance.
D
Inverse Dynamics Model Training Details
D.1
IDM Architecture
The IDM model has approximately 0.5 billion trainable weights. The input to the IDM is 128
consecutive image frames (128 frames of video), each of which has dimensions 128 × 128 × 3. The
IDM is tasked with predicting the action at each frame. All image pixel values are first divided
by 255.0 such that they lie within the range [0, 1]. The first layer of the IDM is a 3-D convolution
with 128 learnable filters with a temporal kernel width of 5 and spatial kernel widths of 1. This
convolution is non-causal, meaning that embeddings at time index t are functions of pixel values at
times t −2, t −1, t, t + 1, and t + 2. We found this layer to be extremely important in IDM training
22


−80 −60 −40 −20
0
20
40
60
80
Mouse movement (pixels)
−10
−5
0
5
10
Camera angle (deg)
0
2
4
6
8
10
Camera bin
Figure 13: Relative camera angle or mouse movement in pixels vs. action bin. The same binning is
used for both X and Y coordinates. The binning is foveated, meaning that binning is more fine-grained
for smaller movements and more coarse-grained for larger movements. There are 11 bins for each
axis (X and Y). The center of each bin (indicated with green circles) is used when un-discretizing
movements (that is, when converting from an action expressed as a bin to a camera angle or mouse
movement).
as it incorporates neighboring temporal information immediately, and we show results comparing
IDM performance with and without this layer in Figure 14. This comparison was made on the default
(1962-hour) IDM dataset.
0
5
10
15
20
25
30
35
Training Progress (Epoch)
0.5
1.0
1.5
2.0
2.5
3.0
Loss
With 3D Conv
No 3D Conv
0
5
10
15
20
25
30
35
Training Progress (Epoch)
0.0
0.2
0.4
0.6
0.8
Keypress Accuracy
With 3D Conv
No 3D Conv
0
5
10
15
20
25
30
35
Training Progress (Epoch)
6
5
4
3
2
1
0
1
Mouse R2
With 3D Conv
No 3D Conv
Figure 14: Effect of 3-D Convolution in the IDM Architecture.
This initial temporal convolutional layer is followed by a ResNet62 image processing network. In this
part of the model, no extra temporal information is shared between neighboring frames; however,
since each frame was first processed with the temporal convolution, some temporal information
is present at this stage. The ResNet image processing network is comprised of three subsequent
stacks with widths W = {64, 128, 128}. Each stack is comprised of, in order, (1) an initial 3x3
convolutional layer with 1-pixel zero padding at the embedding boundary (such that the outgoing
embedding dimensions are the same as the incoming embedding dimension) with W output channels,
(2) a 3x3 max pooling with stride 2 and padding 1 such that the embedding width and height are
halved, and (3) two classic ResNet blocks as defined in He et al. 62 with each layer also having W
output channels.
The output of the ResNet stack is flattened into a 1-dimensional vector of size 217 = 131072 (one
vector for each frame in the video) such that at this stage there are 128 vectors of size 131072. Each
vector is independently processed with two frame-wise dense layers with 256 output activations and
then 4096 output activations, respectively. The result is then fed through 4 subsequent non-causal
(umasked) residual transformer69 blocks. Each block first has an unmasked attention layer, i.e. frames
may attend to future frames, with 32 attention heads of dimension 128 each and a surrounding residual
connection that skips this layer. The embedding is then passed through a frame-wise dense layer
with output dimension 16384 and another with output dimension returning to 4096; a single residual
connection skips past this pair of frame-wise dense layers (not skipping past each layer separately,
but skipping the pair). All dense layers have their weights tied through time, so each frame in the
video is processed with the same weights.
Finally, independent dense layer heads for each action are pulled from the final embedding – a 2
class on/off categorical parameterized with a softmax for each available key as well as a 11-way
23


categorical for both the discretized horizontal and vertical mouse movements (See Appendix C.2 for
details on the action space).
Each dense layer or convolutional layer in the network is preceded by a layernorm70 and followed by
a ReLU non-linearity. Weights are initialized with Fan-In initialization71 and biases are initialized to
zero.
D.2
IDM Training
The total loss for the network is the sum of each independent action prediction loss (one for each
key and one for both mouse directions). Each independent loss is the negative log-likelihood of the
correct action. We use the ADAM72 optimizer with a linear learning rate decay. We use an initial
learning rate of 0.003, a batch size of 128 (where each item in the batch is a video sequence of 128
frames), and a weight decay of 0.01. Hyperparameters were tuned in preliminary experiments. The
IDM is trained on our contractor collected dataset for 20 epochs. This took 4 days on 32 A100 GPUs.
We add data augmentation to each video segment; augmentations are randomly sampled once per
segment such they are temporally consistent. Using the Pytorch73 transforms library, we adjust the
hue by a random factor between -0.2 and 0.2, saturation between 0.8 and 1.2, brightness between
0.8 and 1.2, and contrast between 0.8 and 1.2. We also randomly rotate the image between -2 and 2
degrees, scale it by a random factor between 0.98 and 1.02, shear it between -2 and 2 degrees, and
translate it between -2 and 2 pixels in both the x and y dimensions.
Due the large computational cost of running all of the experiments in this paper, training results
are from one run of training (for IDM, BC, and RL training): this non-ideal situation is mitigated
because deep learning training tends to be low variance74,75 and because we often have data points
from sweeps (e.g. on dataset size) that suggest overall trends.
D.3
Generating Pseudo Labels with the IDM
Section 4.1 shows that inverse dynamics modeling is a much easier task than behavioral cloning
because IDMs can be non-causal. The IDM is trained to simultaneously predict all 128 actions for
each video sequence, so the IDM will effectively be causal for frames at the end of the video clip
because future frames are not included in the sequence. For this reason, we apply the IDM over a
video using a sliding window with stride 64 frames and only use the pseudo-label prediction for
frames 32 to 96 (the center 64 frames). By doing this, the IDM prediction at the boundary of the
video clip is never used except for the first and last frames of a full video.
E
Foundation Model Behavioral Cloning
E.1
Foundation Model Architecture
The behavioral cloning model architecture is the same as the IDM architecture described in Appendix
D.1 except that we modify the architecture so that it is causal (i.e. cannot see the future when making
predictions). This means the BC architecture does not have the initial non-causal convolution the
IDM has (this layer is omitted completely). Furthermore, the residual transformer layers are now
causally masked (as is standard in language modeling) and we do Transformer-XL-style76 training
where frames can attend to keys and values from past batches within the same video. We also use a
Transformer-XL-style relative attention position embedding.
E.2
Null Action Filtering
The most common action humans take is the null action (no keypresses or mouse movements), which
accounts for 35% of all actions they take. Among other reasons, a player may take the null action to
wait for something in the game to finish, to pause between actions, or to take a break to grab a glass
of water. Early on in the project we found that the BC model would take a much larger fraction than
35% of null actions, often upwards of 95%. In order to prevent this behavior we removed frames with
null actions from the dataset. We compare a few different treatments: we filter nulls if there have
been 1, 3, or 21 frames of consecutive null actions, and include a treatment that does not perform
any null filtering. Null action filtering generally helps, increasing all crafting rates (Figure 15 left).
24


basic mining
logs
planks
crafting tables
total crafting
null actions
10 2
100
102
104
Effect of Null Action Filtering
None
All
Groups of 3
Groups of 21
basic mining
logs
planks
crafting tables
total crafting
null actions
10 3
10 2
10 1
100
101
102
103
Joint Hierarchical vs Factored Action Spaces
Factored
Hierarchical Joint
Figure 15: (Left) Effect of Null Action Filtering during training. We compare environment metrics
and number of sampled null action during rollouts (rightmost group of columns) for the following
treatments: no null action filtering (blue), filtering all null actions (green), filtering only groups of
3 or more null actions (red), and filtering only groups of 21 or more null actions (purple). (Right)
Hierarchical versus Factored Action Spaces.
Filtering only groups of 3 performed slightly better than filtering all null action or groups of 21. Initial
experiments indicated that filtering all null actions was better; however, after further model tuning
and after we had already trained our largest models, we found that filtering only groups of 3 or more
null actions performed best. Due to compute constraints we were not able to redo all experiments
with this setting, but doing so would be a reasonable choice for any future work.
E.3
Joint Hierarchical Action Space
We originally worked with a factored action space, where each keypress could be independently on
or off, and this choice was independent of whether the mouse was being moved. This could cause
issues for modeling the human behavior distribution exactly. Say for a given state, humans either with
50% probability (a) move forward and attack or with 50% probability (b) move left and drop their
item. The best a factored distribution can do is to assign 50% probability to each of the 4 constituent
actions because it chooses to press each button simultaneously and independently. See Appendix C.2
for details on the entire action space.
For this reason, we implemented a joint distribution over actions; however, the full joint distribution
over 20 binary buttons and two mouse movement dimensions discretized into 11 bins each would
result in in 220 × 112 ≈1.2 × 108 possible combinations. This is far too large for many reasons, e.g.
the final layer from the transformer stack with a dimension of 4096 would need to be mapped to each
combination resulting in 4096 × 1.2 × 108 ≈5.2 × 1011 parameters for this final layer alone. In
order to reduce this we noted that many buttons in Minecraft have no effect when simultaneously
pressed; for example, if a player tries to move forward and backward at the same time, they remain in
place. Below we list the the sets of mutually exclusive actions. Furthermore, the inventory button is
exclusive with all other buttons and mouse movement.
Mutually Exclusive Actions
forward, back
left, right
sprint, sneak
hotbar.[1-9]
Even reducing the joint action space to reflect these mutually exclusive combinations still results in a
huge action space when combined with the discretized mouse movements, i.e. 33×10×24×112+1 ≈
5.2 × 105. This calculation results from 33 for the 3 sets of 2 mutually exclusive keys above where
taking neither in the set is an option, ×10 for the 9 hotbar keys or no hotbar keypress, ×24 for the
remaining binary 4 keys: use, drop, attack, and jump, ×112 for mouse movements, and finally
+1 for the inventory button which is mutually exclusive with all other actions. ∼5.2 × 105 is still
quite large so we chose to implement a hierarchical binary action for camera being moved or not. If
this action is on, then there is a secondary discrete action head with 121 classes (the joint distribution
of mouse movements because each discretized mouse direction has 11 bins) that determines where
25


to move the mouse. If the hierarchical action is off, then there is no mouse movement, loss for
the secondary mouse movement action is masked during training, and the secondary action head
need not be sampled during evaluations. While this no longer models the full joint distribution, it is
quite a bit better than the factored action space since dependencies between keypresses as well as
whether or not to move the mouse (although not which mouse movement) are modeled jointly. The
resulting action space has dimension 33 × 10 × 24 × 2 + 1 = 8461 (the 112 dimensional multiplier
for camera movement has been replaced by a multiplier of 2 here, corresponding to a binary action
for whether or not to move the mouse) with an additional 121-dimension head for the joint camera
movements. In the future it would be interesting to implement sequential conditional action spaces to
more completely model the joint distribution.
In Figure 15 (right) we compare environment rollout performance between BC models with the
hierarchical joint action space and with the factored action space. Environment statistics are fairly
comparable; however, we see that the factored action space model samples far more null actions.
This is an important example of the factored action space failing to correctly model the distribution in
the dataset because, due to null action filtering, there are 0 null actions in the dataset these models
train on. Despite this, the factored model samples many null actions because the prediction for each
key is not conditioned on other keypresses.
E.4
Foundation Model Training
The foundation model training is similar to the IDM training, with the exception of labels being
IDM-generated pseudo labels. The hyperparameters used for foundation model training are listed in
Table 4.
Hyperparameter
Value
Learning rate
0.002147
Weight decay
0.0625
Epochs
30
Batch size
880
Table 4: Hyperparameters for foundation model training
F
Behavioral Cloning Fine-Tuning
Behavior cloning fine-tuning is similar to the foundation model training, except we either use a focused
subset of all the videos (early_game dataset, described in A.3) with pseudo labels, or contractor data
(contractor_house dataset, described in B.4) with ground-truth labels. The hyperparameters used
for behavior cloning fine-tuning are listed in Table 5. We used 16 A100 GPUs for about 6 hours when
fine-tuning on contractor_house dataset, and 16 A100 GPUs for about 2 days when fine-tuning
on early_game dataset.
Hyperparameter
Value
Learning rate
0.000181
Weight decay
0.039428
Epochs
2
Batch size
16
Table 5: Hyperparameters for behavior cloning fine-tuning
G
Reinforcement Learning Fine-Tuning
G.1
Reinforcement Learning Fine-Tuning Training Details
RL experiments were performed with the phasic policy gradient (PPG) algorithm,64 an RL algorithm
based on the proximal policy optimization (PPO) algorithm77 that increases sample efficiency by
performing additional passes over the collected data to optimize the value function as well as an
26


auxiliary value function. These algorithms have been described extensively in previous work,64,77
so here we describe them only briefly. A major inefficiency when training on-policy algorithms is
that, to remain on-policy, one can only take a single gradient step before new rollout data needs
to be gathered to continue optimization. To alleviate the potentially destructive effects of taking
multiple optimization steps in a single iteration, PPO prevents the policy from changing too much
in a single step by clipping the loss when the difference between the current policy and the policy
before the update becomes too large.77 We also use generalized advantage estimation (GAE), which
can speed-up credit assignment by looking more than 1 step into the future when determining the
advantage of an action, with the look-ahead being determined by hyperparameter λ.78
PPG improves the sample efficiency of PPO when the policy and value function share the same
network by following different optimization processes for the policy, the value function, and their
shared representation. PPG splits optimization in two phases: a wake phase and a sleep phase. In the
wake phase, the policy and value function are optimized as in normal PPO training, with the only
exception being that every sample is used at most once, which prevents the policy from overfitting on
these samples. In the sleep phase PPG optimizes the value function and an auxiliary value function
(which is optimized with the exact same loss as the regular value function, but its output is never used
during training), while keeping a Kullback-Leibler (KL) divergence loss to the policy before the start
of the sleep phase to ensure that the policy does not change. Because the policy is not optimized
in this step, PPG does allow samples to be reused multiple times in this phase. The assumption
behind optimizing the value function during the sleep phase is that value function optimization is
less sensitive to being trained multiple times on the same sample. Optimizing the auxiliary value
function does not directly affect either the value function or the policy, but it can improve the shared
representation of both functions (the assumption being that predicting the value-function requires
encoding all features that are important for distinguishing states). The coefficients for the three
losses (value function loss, auxiliary value function loss, and KL loss) are listed in Table 6. In our
experiments a single iteration consists of two sleep cycles and one wake cycle.
Because the value and auxiliary value functions are not optimized during BC pre-training, they
are initialized at the start of RL fine-tuning. Each value function is implemented as a single, fully
connected layer on top of the last residual transformer block of the pretrained model (Appendix D.1).
The weights of the auxiliary value function are randomly initialized while the weights of the regular
value function are initialized with zero weights, which appeared to prevent destructive updates early
in training that could happen with a randomly initialized value function. To prevent the value-function
loss from having gradients that depend greatly on the magnitude of the reward, we normalize the
value-function target by subtracting the mean and dividing by the standard deviation, which are
estimated through an exponentially weighted moving average.
To prevent catastrophically forgetting the skills of the pretrained network when RL fine-tuning, we
apply an auxiliary KL divergence loss between the RL model and the frozen pretrained policy.10 This
loss is defined as:
Lklpt = ρKL(πpt, πθ)
(2)
Where πθ is the the policy being trained, πpt is the frozen pretrained policy, KL(πpt, πθ) is the
Kullback-Leibler divergence between the policy being trained and the pretrained policy, and ρ is a
coefficient to weight this loss relative to other losses.
In the fine-tuning experiments, this KL divergence loss replaces the common entropy maximization
loss, which is often added to RL experiments to encourage exploration.79,80 The idea behind entropy
maximization is that, when all actions appear to have equal value, such as when the agent has not
learned about the next reward, it should maximize its entropy to increase the chance that it discovers
the next reward. Blindly exploring by maximizing entropy is effective when the state and action
spaces are sufficiently small or the reward is sufficiently dense, but becomes infeasible when the
state and action spaces are large and rewards are sparse, which is the case in the diamond-pickaxe
task. Instead of blindly exploring through uniform-random actions, we assume that the pretrained
policy has an action distribution that is much more likely to take sequences of actions that lead
to interestingly new states, and thus, in states where the agent assigns equal value to each of its
actions, it should mimic the action-distribution of the pretrained policy instead of a uniform-random
action distribution. In experiments with a randomly initialized policy we do include the entropy
maximization loss with a coefficient of 0.01, which has been an effective setting in other Minecraft
work.30 Empirically, we found that a high coefficient ρ for this KL divergence loss would prevent
the agent from properly optimizing the reward function while a low coefficient ρ was ineffective at
27


Hyperparameter
Value
Learning rate:
2 × 10−5
Weight decay:
0.04
Batch size:
40
Batches per iteration:
48
Context length:
128
Discount factor (γ):
0.999
GAE λ:
0.95
PPO clip:
0.2
Max Grad norm:
5
Max Staleness:
2
PPG sleep cycles:
2
PPG sleep value-function coefficient:
0.5
PPG sleep auxiliary value-function coefficient:
0.5
PPG sleep KL coefficient:
1.0
PPG sleep max Sample Reuse:
6
KL divergence coefficient ρ:
0.2
Coefficient ρ decay:
0.9995
Table 6: Hyperparameters for RL experiments. These are the hyperparameters for all treatments with
two exceptions. First, when fine-tuning from the early-game model without a KL divergence loss,
in addition to the KL divergence loss being set to 0, the learning rate was set to 3 × 10−6 (the best
setting out of a sweep over 5 different learning rates), as we found that performance was substantially
lower with the standard learning rate of 2 × 10−5 and the agent did not even learn to collect logs.
We suspect that the reason that the learning rate needed to be lowered when fine-tuning without a
KL loss is that the KL loss prevents making optimization steps that change the policy too much in a
single step, especially in early iterations when the value function has not been optimized yet, and
the KL loss thus makes it possible to optimize with a higher learning rate. Second, when running
RL from a randomly initialized policy there is no KL divergence loss or KL divergence decay, but
instead we use an entropy bonus of 0.01, which reportedly worked well in previous work.30
protecting the learned skills of the pretrained policy and preventing catastrophic forgetting. As such,
we start with a relatively high coefficient ρ and decay it by a fixed factor after each iteration (Table 6).
This method protects policy skills in early iterations while guaranteeing that the policy can eventually
maximize the reward function, regardless of how different its behavior has to be to do so relative to
the pretrained policy.
For the reward function we estimated the rough quantities of each item that a human player might
gather when trying to craft a diamond pickaxe, and we reward the model for gathering up to that
quantity for each item. We started these estimates by iterating over the technology tree backward from
a diamond pickaxe and adding the requirements for each item to the reward function (e.g. first we
added a diamond pickaxe to the reward function, then we added the 3 diamonds and 2 sticks required
for crafting a diamond pickaxe, then we added the 1 iron pickaxe required for mining diamonds,
and so on). Then we added coal and torches to the reward function, with coal being useful as fuel
when smelting iron and for crafting torches while the torches themselves improve visibility and
prevent enemies from spawning. Finally, we reward the model for bringing additional logs (5 logs are
required to craft all items in the reward function, but we reward up to 8 logs), which can be used as
fuel or crafted into a crafting table or sticks if the agent runs out. In practice the agent rarely collects
the additional logs, places the torches, or uses coal as fuel when smelting, but the reward function
was based on human expectations on what would be useful to execute this task, rather than designed
around how an RL model behaves after training. Finally, to encourage the agent to keep mining
diamonds and crafting diamond pickaxes after it has crafted its first diamond pickaxe, we did not put
a limit on the number of diamonds or diamond pickaxes that would be rewarded.
The rewards for the different items are separated into 4 tiers, roughly depending on how late a player
would usually get the relevant item. The first tier consists of all wooden and stone items and has a
base reward of 1, the second tier consists of all items requiring coal with a base reward of 2, the third
tier consists of all items requiring iron with a base reward of 4, and the final tier is diamond with a
base reward of 8. Thus items later in the sequence of items towards a diamond pickaxe generally
28


Item
Quantity rewarded
Reward per item
Log
8
1/8
Planks
20
1/20
Stick
16
1/16
Crafting table
1
1
Wooden pickaxe
1
1
Cobblestone
11
1/11
Stone pickaxe
1
1
Furnace
1
1
Coal
5
2/5
Torch
16
1/8
Iron ore
3
4/3
Iron ingot
3
4/3
Iron pickaxe
1
4
Diamond
inf
8/3
Diamond pickaxe
inf
8
Table 7: Reward per item and total quantity rewarded.
give a higher reward. To make sure that the agent does not over-value items that are supposed to
be gathered in bulk (e.g. the agent is rewarded for up to 20 planks but only up to 1 crafting table,
which can cause the agent to focus on planks at the expense of creating a crafting table), we divide
the base reward of each item by the total quantity that the agent gets rewarded for (for the purpose
of determining the reward, the total quantity for diamonds is 3 and the total quantity for diamond
pickaxes is 1, even though we did not put a limit on the number of these items being rewarded). For
example, the agent is rewarded for 3 iron ore, which has a base reward of 4 for being in the iron tier
and up to 3 blocks of iron ore are rewarded, thus the reward per block of iron ore is 4/3. The quantity
and reward for each item are listed in Table 7.
While every item in the sequence towards a diamond pickaxe is rewarded, the reward function is still
sparse and, in some cases, even deceptive. The sparsity comes from the fact that it can take thousands
of actions to find the next reward, even after the agent has acquired all the necessary prerequisites
(e.g. human players often take more than 10,000 actions to find a diamond after crafting an iron
pickaxe). The reward function can be deceptive when the most efficient method for getting one item
can make it far more difficult to get the next item. For example, a good strategy for the agent to
craft a stone pickaxe quickly is to mine (i.e. spend a few seconds to pick up) its crafting table after
crafting a wooden pickaxe, such that the agent has immediate access to a crafting table as soon as it
has collected enough cobblestone. However, the fastest way to get a reward for gathering cobblestone
is to mine down immediately after crafting a wooden pickaxe, while leaving the crafting table behind.
Thus following the optimal strategy for gathering cobblestone makes it more difficult to learn to craft
a stone pickaxe.
Experiments ran for approximately 6 days (144 hours) on 80 GPUs (for policy optimization) and
56,719 CPUs (mostly for collecting rollouts from Minecraft). In this time the algorithm performed
roughly 4,000 optimization iterations and collected roughly 1.4 million Minecraft episodes consisting
of 12,000 frames each, for a total of 16.8 billion frames.
G.2
Reinforcement Learning Fine-Tuning Additional Data
Additional figures that are helpful for understanding the main results of the RL fine-tuning experiments
are presented in this section. First, we show the items-over-training figure when RL fine-tuning from
the early-game model without a KL loss (Fig. 16). When training without a KL loss, the model
only learns to obtain the four items that the early-game model is capable of getting zero-shot, which
are logs, planks, sticks, and crafting tables. Second, we present preliminary experiments in which
we directly compare RL fine-tuning from the house-building model and RL fine-tuning from the
early-game model (Fig. 17). These experiments differ from the main experiments in that, for both
treatments shown here, the KL loss coefficient was set to 0.4, the learning rate was set to 6 × 10−5,
and the reward for each item was 1/quantity for all items (i.e. items closer to the diamond pickaxe
did not have an increased reward). While RL fine-tuning from the house-building model initially
29


worked better than RL fine-tuning from the early-game model, fine-tuning from the early-game model
worked better after 800,000 episodes and showed signs of smelting iron ingots, which is why the
early-game model was chosen for the main experiments.
0.0
0.2
0.4
0.6
0.8
1.0
1.2
1.4
Episodes
1e6
0
20
40
60
80
100
% episodes
No KL-loss
Figure 16: Items obtained when RL fine-tuning from the early-game model without a KL loss. The
model learns to obtain all items that the early-game model can craft zero-shot, which are logs, planks,
sticks, and a crafting table. In contrast to the treatment with a KL-penalty, it does not learn any items
beyond these initial four, likely because skills that are not performed zero-shot, and for which the
model thus does not initially see any reward, are catastrophically forgotten while the first four items
are learned.
0
100000
200000
300000
400000
500000
600000
700000
800000
Episodes
0
2
4
6
8
Reward
Reward over episodes
RL from Early-Game model
RL from House-Builing model
0
100000
200000
300000
400000
500000
600000
700000
800000
Episodes
0.000
0.005
0.010
0.015
0.020
0.025
0.030
0.035
Iron Ingots Obtained
Iron Ingots Obtained Per Episodes
RL from Early-Game model
RL from House-Builing model
Figure 17: Preliminary experiments when RL fine-tuning from the early-game model compared to
RL fine-tuning from the house-building model. (Left) While reward initially increases faster when
fine-tuning from the house-building model, fine-tuning form the early-game model eventually obtains
a slightly higher reward. (Right) RL fine-tuning from the early-game model has a higher likelihood
of smelting an iron-ingot, which is why the early-game model was chosen for future RL fine-tuning
experiments.
H
Foundation Model Scaling
In early experiments we found that increasing model size led to models staying in the efficient
learning regime longer into training.63 Here we compare the 0.5B model described in Section 4.2 to
both a 248M and 71M parameter model. Both of these models are trained for 15 epochs as compared
to the 30 epochs the 0.5B model trained for. These models have the same architecture as the 0.5B
model but each layer in the 248M parameter model has 1/2 the width and each layer in the 71M
parameter model 1/3 the width. The 71M model was trained with an initial learning rate of 0.001586,
batch size of 480, and weight decay of 0.044506. The 248M model had an initial learning rate of
0.001831, batch size of 640, and weight decay of 0.051376.
In Figure 18 we show validation loss on web_clean with IDM pseudo-labels, loss on the contractor
dataset used to train the IDM with ground truth labels collected during contractor play, and zero-shot
environment performance for the 71M, 248M, and 0.5B models. While larger models have better
validation loss on web_clean, these results do not tell the clear story that the 0.5B model is better
than its smaller counterparts. The 71M model has the lowest contractor dataset loss while having the
highest web_clean loss, and it also has the best zero-shot environment performance. In fact, we see
that the 71M model even had non-zero wooden tool crafting (Fig. 18 bottom left). The 248M model
also appears to be better at crafting than the 0.5B, and also has lower contractor dataset loss.
While the zero-shot results suggest smaller models are better, fine-tuning tells another story. When
fine-tuning to contractor_house, model size rank ordering reverses and now the 0.5B model
performs best both in validation loss (Fig. 19 left) and in environment performance (Fig. 19 right)
30


10 3
10 2
10 1
100
101
~Compute
2.5
4
6
10
Web Clean Loss
Loss Web Clean Validation Dataset
71M
248M
0.5B
10 3
10 2
10 1
100
101
~Compute
2.5
4
6
10
IDM Contractor Dataset Loss
Loss on IDM Contractor Dataset
71M
248M
0.5B
basic
mining
logs
planks
crafting
tables
total
crafting
10 2
10 1
100
101
Collected or Crafted
Zero-Shot Performance vs Model Size
71M
248M
0.5B
0
2
4
6
8
10
12
14
16
Training Progress (Epoch)
0
10 3
10 2
10 1
100
101
Crafting or Collection
Zero-Shot Performance over Training
(71M Parameter Model)
basic mining
logs
planks
crafting tables
total crafting
wooden tools
0
2
4
6
8
10
12
14
16
Training Progress (Epoch)
0
10 3
10 2
10 1
100
101
Zero-Shot Performance over Training
(248M Parameter Model)
0
5
10
15
20
25
Training Progress (Epoch)
0
10 3
10 2
10 1
100
101
Zero-Shot Performance over Training
(0.5B Parameter Model)
Figure 18: Training and Zero-Shot Performance versus Model Scale. In the first two plots the x-axis
is compute normalized to that used by the 71M parameter model, such that after 15 epochs of training
the 71M model has used 1 "compute". The 248M parameter model and the 71M model are trained on
the same amount of data (15 epochs), and the 0.5B parameter model is trained on 30 epochs of data.
(Top Left) Loss on the web_clean validation dataset. (Top Middle) Loss on the IDM contractor
dataset; note that these models were trained only on web_clean and not on any contractor data.
(Top Right) Zero-shot environment rollout performance at the end of training. (Bottom) Zero-shot
environment rollout performance over training for the 71M model (bottom left), 248M model (bottom
middle), and 0.5B model (bottom right).
basic
mining
logs
planks crafting
tables
total
crafting
wooden
tools
stone
tools
10 3
10 2
10 1
100
101
102
103
104
Collected or Crafted
Fine-Tuning to Contractor House Dataset
71M
248M
0.5B
0.0
0.5
1.0
1.5
2.0
2.5
3.0
Fine-Tuning Epoch
2.2
2.4
2.6
2.8
3.0
contractor_house Validation Loss
Loss on contractor_house
71M
248M
0.5B
0.0
0.5
1.0
1.5
2.0
2.5
3.0
Fine-Tuning Epoch
2.4
2.5
2.6
2.7
2.8
2.9
3.0
Loss on IDM Contractor Dataset
Loss on full IDM Contractor Dataset
71M
248M
0.5B
Figure 19: contractor_house fine-tuning performance versus model size. (Left) Loss on the
contractor_house holdout validation set. (Middle) Loss on the full contractor dataset collected
to train the IDM; this dataset is disjoint from contractor_house. (Right) Environment rollout
performance at the end of fine-tuning.
followed by the 248M model and then the 71M model. Environment model rollouts are performed
using the same game engine that we use to collect contractor data, which could be visually distinct
from videos taken from the web. It is plausible that the larger models overfocus on the visual
peculiarities in web data during pretraining since they have worse contractor data loss (Fig.18 top
middle), and this causes them to perform more poorly in the environment zero-shot. However, we
hypothesize that because the contractor_house dataset we fine-tune to is collected from our game
engine, the larger models that are a better overall Minecraft prior (as indicated by lower web_clean
validation loss in Fig.18 top left) can quickly shift their low level features to perform better on data
coming from our game engine, resulting in better environment rollout performance. This hypothesis
is further supported by Fig. 19 (middle) showing loss on the contractor dataset collected for IDM
training, which has no overlap with contractor_house. After just a few steps of fine-tuning to
contractor_house, all models quickly improve in loss on the full IDM contractor dataset, with
larger models now performing best. While not conclusive, we believe this investigation provides
some intuition for future studies of model scaling for sequential decision making problems.
31


I
Text Conditioning
Goal-conditioned policies81,82 make it possible for a single agent to perform a wide variety of
goals in a single environment, which is particularly relevant in open-ended environments such as
Minecraft. In recent work, goal specification has increasingly taken the form of domain specific
languages83, or even natural language84,85. The benefits of language-conditioned agents can be
tremendous, especially natural-language-conditioned agents, as their goal space contains a wide
variety of potentially very complex tasks. Text conditional models have shown an amazing ability to
perform tasks zero-shot (or learn them few-shot) including generalizing in impressive ways via the
compositional and combinatorial possibilities allowed by natural language (e.g. GPT1 and DALL·E
286). We hypothesize that we should expect similar capabilities to emerge with natural-language-
conditioned virtual agents, if they are similarly trained on enormous amounts of data (that goes from
a natural language description to a sequence of actions that completes the specified goal). In this
section we take preliminary steps toward that future. Our preliminary experiments provide evidence
that it is possible to pretrain a natural-language-conditioned model for Minecraft using the general
approach presented in this paper (VPT) plus conditioning on the speech that often accompanies
videos.
In online videos, the human actor sometimes indicates their intent in their verbal commentary
(e.g. “Let’s go chop some trees to make a wooden axe” or “now let’s learn how to crop photos in
Photoshop”). Conditioning on this closed caption data could produce a steerable pre-trained model:
i.e., it may later be possible to condition the model with text such as “I am going to craft a wooden
pickaxe” or “I am going to build a house,” and have the agent perform those tasks specifically rather
than simply follow typical human behavior (as was investigated in the rest of this paper). An alternate
way to produce a steerable agent is via RL fine-tuning, which we could have done in Section 4.4
by adding a bit indicating the task to be completed, as has been done in prior work30. However,
conditioning on natural language offers many benefits over that approach. First, it is flexible and
powerful, being able to express any task. Second, one does not need to preconceive of the task to
be completed ahead of time. This would allow for general, capable, zero-shot agents like GPT, but
extending those capabilities to embodied tasks such as completing tasks on computers or in simulated
3D worlds. Third, text conditioning can be used even when tasks are difficult to specify via reward
functions (e.g. “Let’s build a house” or–if the agent is capable of doing it–more complex things like
“I will now build a castle surrounded by a moat”). In the limit, VPT+text could conceivably produce
powerful, capable, natural-language-conditional agents with the powers of GPT to meta-learn, follow
instructions, and complete tasks zero or few shot, but in the form of agents that can act in virtual
worlds, complete tasks on computers, and in other similar embodied sequential decision domains.
We do not reach those lofty goals in this work, but we began a first step towards exploring in that
direction.
Many Minecraft videos feature audio commentary from the player. This commentary is sometimes
present in the form of closed captions for the videos, or could be extracted post-hoc using automated
speech recognition (ASR).87 Our dataset features about 17k hours of content with associated closed
captions.
We fine-tuned the 220 million parameter VPT foundation model used in the RL-fine-tuning ex-
periments (chosen vs. 0.5B for the same reason: to reduce compute costs) with an additional
text-conditioning input on the subset of our data for which closed captions are available. To obtain the
conditioning input, we first split videos into 30 second chunks. The same text is associated with every
frame in a given chunk, and is made up of all the closed captions occurring within that chunk, as well
as the line of text preceding and following the chunk (if any). Because the vast majority (around
95%) of our closed caption data lacks capitalization and punctuation, it is punctuated using the rpunct
library88. We then obtain a text embedding vector of length 4,096 from the OpenAI embedding API89,
which is processed by a randomly initialized multi-layer perceptron (MLP) with two hidden layers of
size 2,048. The resulting activations are added for each frame to the pretrained model activations
before the transformer layers (pretransformerActivations += mlp(textEmbedding)). The
model is fine-tuned for four epochs.
Our model shows evidence of steerability. When conditioned on sentences that incite the agent to
explore (such as “I’m going to explore” and “I’m going to find water”) the agent travels significantly
farther from its spawn point (Figure 20a). Additionally, we can steer the agent to preferentially collect
32


dig
dirt
explore
house
seed
water
wood
Conditioning
120
130
140
150
Travel distance (blocks)
Travel Distance with Conditioning
(a)
dig
dirt
explore
house
seed
water
wood
Conditioning
0.4
0.5
0.6
0.7
0.8
0.9
Wheat seeds collected
Seed Collection with Conditioning
(b)
dig
dirt
explore
house
seed
water
wood
Conditioning
1.00
1.25
1.50
1.75
2.00
2.25
2.50
2.75
Oak logs collected
Log Collection with Conditioning
(c)
dig
dirt
explore
house
seed
water
wood
Conditioning
5
6
7
8
9
10
Dirt collected
Dirt Collection with Conditioning
(d)
Figure 20: Evidence for conditioning. In each plot, the variants expected to stand out are shown in
bold. The strings corresponding to each variant are shown in Table 8. Statistics are measured over 5
minute episodes. (a) Distance traveled by the agent . Both “explore” and “water” text strings should
encourage a steerable agent to move more than when doing other tasks, which is what occurs. Grass
(which is needed to get seeds) is not present in all biomes, which is likely why the “seed” condition
produces more travel (as the agent sometimes needs to move to a biome with grass). The travel
distance is the Euclidean distance from the spawn point to the farthest point the agent reached during
the episode on the horizontal (x-z) plane. (b) Collection of wheat seeds. The “seed” variant collects
substantially more than other variants, as expected of a steerable agent. (c) Collection of oak (the
most common type of wood) logs. The “wood” variant collects significantly more oak logs, as is to
be expected of a steerable agent (we speculate that the “water” variant collects less because there are
no trees in water). (d) Collection of dirt. The “dirt” and “dig” variants collect a large amount, and are
the variants that are (indirectly in the case of “dig”) conditioned to collect dirt. It is easy to mistakenly
aim at the ground rather than at grass or trees when collecting seeds or wood, which likely explains
the slightly higher amount of dirt collected by these variants. In all cases, the error bars are 95%
confidence intervals of the mean, over 1,000 episodes per conditioning variant. Treatments for which
the bars in each bar plot do not overlap are statistically significantly different at a p < 0.05 level.
Variant name
String
dig
I’m going to dig as far as possible
dirt
I’m going to collect dirt
explore
I’m going to explore
house
I’m going to make a house
seed
I’m going to collect seeds
water
I’m going to find water
wood
I’m going to chop wood
Table 8: Strings corresponding to each conditioning variant.
33


early game items such as seeds, wood, and dirt by conditioning with text such as “I’m going to collect
seeds/chop wood/collect dirt” (Figure 20b,c,d).
While our results show some level of steerability, more work is required to increase it. For example,
we were not able to successfully steer agents to gather flowers or to hunt, both of which are possible
in the early game, but less common (and, in the case of hunting animals, much more difficult) than
gathering dirt, wood, or seeds. Likewise, an experiment in which the agent is presented with a
crafting window and various resources, and conditioned to craft a given item (e.g. “I’m going to
craft a wooden axe”) failed to show that the conditioning had a significant effect on which items got
crafted. Instead, it seemed the agent was more influenced by the prior, unconditional probability
of what human players would craft next given the resources available, which is not too surprising
since in Minecraft, especially in the early game, there is a relatively consistent path to gathering
resources in a specific order go produce more powerful tools (Fig. 6). For example, if the agent had
the resources to make a stone pickaxe and we asked it instead to make a (weaker) wooden pickaxe, it
often would make the stone pickaxe anyway. Finally, looking at videos of agent behaviors failed to
convince us that the “house” conditioning causes the agents to take more steps towards building a
house than other variants.
Thus, our results show that it is possible to train a somewhat steerable natural-language-conditioned
agent. However, its steerability is still too weak to be practically useful, and it is far from what we
believe could be accomplished with more research, data, and training compute. Another exciting
research direction is to have the model predict future text as well as just the next action.
34


VOYAGER: An Open-Ended Embodied Agent
with Large Language Models
Guanzhi Wang1 2 #, Yuqi Xie3, Yunfan Jiang4∗, Ajay Mandlekar1∗,
Chaowei Xiao1 5, Yuke Zhu1 3, Linxi “Jim” Fan1† #, Anima Anandkumar1 2†
1NVIDIA, 2Caltech, 3UT Austin, 4Stanford, 5UW Madison
∗Equal contribution
†Equal advising
# Corresponding authors
https://voyager.minedojo.org
Abstract
We introduce VOYAGER, the first LLM-powered embodied lifelong learning agent
in Minecraft that continuously explores the world, acquires diverse skills, and
makes novel discoveries without human intervention. VOYAGER consists of three
key components: 1) an automatic curriculum that maximizes exploration, 2) an
ever-growing skill library of executable code for storing and retrieving complex
behaviors, and 3) a new iterative prompting mechanism that incorporates environ-
ment feedback, execution errors, and self-verification for program improvement.
VOYAGER interacts with GPT-4 via blackbox queries, which bypasses the need for
model parameter fine-tuning. The skills developed by VOYAGER are temporally
extended, interpretable, and compositional, which compounds the agent’s abilities
rapidly and alleviates catastrophic forgetting.
Empirically, VOYAGER shows
strong in-context lifelong learning capability and exhibits exceptional proficiency
in playing Minecraft. It obtains 3.3× more unique items, travels 2.3× longer
distances, and unlocks key tech tree milestones up to 15.3× faster than prior SOTA.
VOYAGER is able to utilize the learned skill library in a new Minecraft world to
solve novel tasks from scratch, while other techniques struggle to generalize.
Figure 1: VOYAGER discovers new Minecraft items and skills continually by self-driven exploration,
significantly outperforming the baselines. X-axis denotes the number of prompting iterations.
1
arXiv:2305.16291v2  [cs.AI]  19 Oct 2023


Mine Wood  Log
Make Crafting Table
Craft Stone Sword
Craft Shield
Make Furnace
Cook Steak
Combat Zombie
     Mine Wood Log
Make Crafting Table
Combat 
Zombie
Mine Diamond
New 
Task
Code as 
Actions
Refine Program
Env Feedback
Execution Errors
Update 
Exploration 
Progress
Skill 
Retrieval
Add New Skill
Automatic Curriculum
Iterative Prompting Mechanism
Skill Library
Environment
Self-Verification
Figure 2: VOYAGER consists of three key components: an automatic curriculum for open-ended
exploration, a skill library for increasingly complex behaviors, and an iterative prompting mechanism
that uses code as action space.
1
Introduction
Building generally capable embodied agents that continuously explore, plan, and develop new skills
in open-ended worlds is a grand challenge for the AI community [1–5]. Classical approaches
employ reinforcement learning (RL) [6, 7] and imitation learning [8–10] that operate on primitive
actions, which could be challenging for systematic exploration [11–15], interpretability [16–18], and
generalization [19–21]. Recent advances in large language model (LLM) based agents harness the
world knowledge encapsulated in pre-trained LLMs to generate consistent action plans or executable
policies [16, 22, 19]. They are applied to embodied tasks like games and robotics [23–27], as well as
NLP tasks without embodiment [28–30]. However, these agents are not lifelong learners that can
progressively acquire, update, accumulate, and transfer knowledge over extended time spans [31, 32].
Let us consider Minecraft as an example. Unlike most other games studied in AI [33, 34, 10],
Minecraft does not impose a predefined end goal or a fixed storyline but rather provides a unique
playground with endless possibilities [23]. Minecraft requires players to explore vast, procedurally
generated 3D terrains and unlock a tech tree using gathered resources. Human players typically start
by learning the basics, such as mining wood and cooking food, before advancing to more complex
tasks like combating monsters and crafting diamond tools. We argue that an effective lifelong learning
agent should have similar capabilities as human players: (1) propose suitable tasks based on its
current skill level and world state, e.g., learn to harvest sand and cactus before iron if it finds itself in
a desert rather than a forest; (2) refine skills based on environmental feedback and commit mastered
skills to memory for future reuse in similar situations (e.g. fighting zombies is similar to fighting
spiders); (3) continually explore the world and seek out new tasks in a self-driven manner.
Towards these goals, we introduce VOYAGER, the first LLM-powered embodied lifelong learning
agent to drive exploration, master a wide range of skills, and make new discoveries continually
without human intervention in Minecraft. VOYAGER is made possible through three key modules
(Fig. 2): 1) an automatic curriculum that maximizes exploration; 2) a skill library for storing
and retrieving complex behaviors; and 3) a new iterative prompting mechanism that generates
executable code for embodied control. We opt to use code as the action space instead of low-level
motor commands because programs can naturally represent temporally extended and compositional
actions [16, 22], which are essential for many long-horizon tasks in Minecraft. VOYAGER interacts
with a blackbox LLM (GPT-4 [35]) through prompting and in-context learning [36–38]. Our approach
bypasses the need for model parameter access and explicit gradient-based training or finetuning.
More specifically, VOYAGER attempts to solve progressively harder tasks proposed by the automatic
curriculum, which takes into account the exploration progress and the agent’s state. The curriculum
is generated by GPT-4 based on the overarching goal of “discovering as many diverse things as
possible”. This approach can be perceived as an in-context form of novelty search [39, 40]. VOYAGER
incrementally builds a skill library by storing the action programs that help solve a task successfully.
2


Inventory (5/36): {'oak_planks': 3, 'stick': 
4, 'crafting_table': 1, 'stone': 3, 
'wooden_pickaxe': 1}
Biome: river
Inventory (4/36): {'sugar_cane': 3, 'paper': 
3, 'fishing_rod': 1, 'wooden_pickaxe': 1}
Nearby entities: pig, cat, villager
Health: 12/20
Hunger: 0/20
Inventory (6/36): {'furnace': 1, 
'stone_pickaxe': 1, 'oak_planks': 7, 
'crafting_table': 1, 'raw_iron': 4, 'coal': 1} 
Time: night
Nearby entities: zombie
Equipment: 'stone_sword', 'shield'
Reasoning: Since you have a wooden pickaxe and some stones, it would be 
beneficial to upgrade your pickaxe to a stone pickaxe for better efficiency.
Task: Craft 1 stone pickaxe.
Reasoning: Since you have a fishing rod in your inventory and you are near a river 
biome, it would be a good opportunity to catch some fish for food and experience. 
Task: Catch 1 fish.
Reasoning: Your hunger is at 0, which means you need to find food to replenish 
your hunger. Since there are pigs nearby, you can kill one to obtain raw porkchops.
Task: Kill 1 pig.
Reasoning: You have raw iron and coal, and you have a furnace. It's time to smelt 
the iron to make iron ingots, which can be used to craft better tools and armor.
Task: Smelt 4 raw iron.
Reasoning: Since it's night and there's a zombie nearby, it's a good opportunity to 
try killing the zombie now that you have a stone sword and a shield equipped.
Task: Kill 1 zombie.
GPT-4
GPT-4
GPT-4
GPT-4
GPT-4
Figure 3: Tasks proposed by the automatic curriculum. We only display the partial prompt for brevity.
See Appendix, Sec. A.3 for the full prompt structure.
Each program is indexed by the embedding of its description, which can be retrieved in similar
situations in the future. Complex skills can be synthesized by composing simpler programs, which
compounds VOYAGER’s capabilities rapidly over time and alleviates catastrophic forgetting in other
continual learning methods [31, 32].
However, LLMs struggle to produce the correct action code consistently in one shot [41]. To address
this challenge, we propose an iterative prompting mechanism that: (1) executes the generated
program to obtain observations from the Minecraft simulation (such as inventory listing and nearby
creatures) and error trace from the code interpreter (if any); (2) incorporates the feedback into GPT-4’s
prompt for another round of code refinement; and (3) repeats the process until a self-verification
module confirms the task completion, at which point we commit the program to the skill library (e.g.,
craftStoneShovel() and combatZombieWithSword()) and query the automatic curriculum for
the next milestone (Fig. 2).
Empirically, VOYAGER demonstrates strong in-context lifelong learning capabilities. It can construct
an ever-growing skill library of action programs that are reusable, interpretable, and generalizable
to novel tasks. We evaluate VOYAGER systematically against other LLM-based agent techniques
(e.g., ReAct [29], Reflexion [30], AutoGPT [28]) in MineDojo [23], an open-source Minecraft AI
framework. VOYAGER outperforms prior SOTA by obtaining 3.3× more unique items, unlocking key
tech tree milestones up to 15.3× faster, and traversing 2.3× longer distances. We further demonstrate
that VOYAGER is able to utilize the learned skill library in a new Minecraft world to solve novel tasks
from scratch, while other methods struggle to generalize.
2
Method
VOYAGER consists of three novel components: (1) an automatic curriculum (Sec. 2.1) that suggests
objectives for open-ended exploration, (2) a skill library (Sec. 2.2) for developing increasingly
complex behaviors, and (3) an iterative prompting mechanism (Sec. 2.3) that generates executable
code for embodied control. Full prompts are presented in Appendix, Sec. A.
2.1
Automatic Curriculum
Embodied agents encounter a variety of objectives with different complexity levels in open-ended
environments. An automatic curriculum offers numerous benefits for open-ended exploration, ensur-
ing a challenging but manageable learning process, fostering curiosity-driven intrinsic motivation
for agents to learn and explore, and encouraging the development of general and flexible problem-
solving strategies [42–44]. Our automatic curriculum capitalizes on the internet-scale knowledge
contained within GPT-4 by prompting it to provide a steady stream of new tasks or challenges. The
curriculum unfolds in a bottom-up fashion, allowing for considerable adaptability and responsiveness
to the exploration progress and the agent’s current state (Fig. 3). As VOYAGER progresses to harder
self-driven goals, it naturally learns a variety of skills, such as “mining a diamond”.
3


Program Description
Skill Library
Top-5 Relevant Skills
Program Generated by GPT-4
Task: Craft Iron Pickaxe
Key
Add
Retrieve
Value
Skill Library
Query
How to craft an iron pickaxe in 
Minecraft?
To craft an iron pickaxe, you 
need to 3 iron ingots and 2 
sticks. Once you have gathered 
the materials, ....
----------------------------------
         Environment Feedback
Mine Wood  Log
Make Crafting Table
Craft Wooden Pickaxe
Craft Stone Sword
Make Furnace
...
Combat Cow
Cook Steak
Craft Iron Axe
Combat Zombie
Smelt Iron Ingot
Craft Stick
Make Crafting Table
Make Furnace
Craft Wooden Pickaxe
GPT-3.5
Embedding
Embedding
GPT-3.5
Figure 4: Skill library. Top: Adding a new skill. Each time GPT-4 generates and verifies a new
skill, we add it to the skill library, represented by a vector database. The key is the embedding vector
of the program description (generated by GPT-3.5), while the value is the program itself. Bottom:
Skill retrieval. When faced with a new task proposed by the automatic curriculum, we first leverage
GPT-3.5 to generate a general suggestion for solving the task, which is combined with environment
feedback as the query context. Subsequently, we perform querying to identify the top-5 relevant skills.
The input prompt to GPT-4 consists of several components:
(1) Directives encouraging diverse behaviors and imposing constraints,
such as
“My ultimate goal is to discover as many diverse things as possible
...
The next task should not be too hard since I may not have the
necessary resources or have learned enough skills to complete it
yet.”;
(2) The agent’s current state, including inventory, equipment, nearby blocks and entities,
biome, time, health and hunger bars, and position;
(3) Previously completed and failed tasks, reflecting the agent’s current exploration progress
and capabilities frontier;
(4) Additional context: We also leverage GPT-3.5 to self-ask questions based on the agent’s
current state and exploration progress and self-answer questions. We opt to use GPT-3.5
instead of GPT-4 for standard NLP tasks due to budgetary considerations.
2.2
Skill Library
With the automatic curriculum consistently proposing increasingly complex tasks, it is essential to
have a skill library that serves as a basis for learning and evolution. Inspired by the generality, inter-
pretability, and universality of programs [45], we represent each skill with executable code that scaf-
folds temporally extended actions for completing a specific task proposed by the automatic curriculum.
The input prompt to GPT-4 consists of the following components:
(1) Guidelines
for
code
generation,
such
as
“Your function will be reused
for building more complex functions.
Therefore, you should make
it generic and reusable.”;
(2) Control primitive APIs, and relevant skills retrieved from the skill library, which are
crucial for in-context learning [36–38] to work well;
(3) The generated code from the last round, environment feedback, execution errors, and
critique, based on which GPT-4 can self-improve (Sec. 2.3);
(4) The agent’s current state, including inventory, equipment, nearby blocks and entities,
biome, time, health and hunger bars, and position;
4


I cannot make stick because I need:  2 more planks
I cannot make stone_shovel because I need:  2 more stick
throw new Error(`No item named ${name}`);
No item named acacia_axe
at line 18:await craftItem(bot, "acacia_axe", 1);
Environment Feedback
Execution Error
GPT-4
GPT-4
Figure 5: Left: Environment feedback. GPT-4 realizes it needs 2 more planks before crafting sticks.
Right: Execution error. GPT-4 realizes it should craft a wooden axe instead of an acacia axe since
there is no acacia axe in Minecraft. We only display the partial prompt for brevity. The full prompt
structure for code generation is in Appendix, Sec. A.4.
(5) Chain-of-thought prompting [46] to do reasoning before code generation.
We iteratively refine the program through a novel iterative prompting mechanism (Sec. 2.3), in-
corporate it into the skill library as a new skill, and index it by the embedding of its description
(Fig. 4, top). For skill retrieval, we query the skill library with the embedding of self-generated task
plans and environment feedback (Fig. 4, bottom). By continuously expanding and refining the skill
library, VOYAGER can learn, adapt, and excel in a wide spectrum of tasks, consistently pushing the
boundaries of its capabilities in the open world.
2.3
Iterative Prompting Mechanism
We introduce an iterative prompting mechanism for self-improvement through three types of feedback:
(1) Environment feedback, which illustrates the intermediate progress of program execution
(Fig. 5, left). For example, “I cannot make an iron chestplate because I need:
7 more iron ingots” highlights the cause of failure in crafting an iron chestplate. We use
bot.chat() inside control primitive APIs to generate environment feedback and prompt
GPT-4 to use this function as well during code generation;
(2) Execution errors from the program interpreter that reveal any invalid operations or syntax
errors in programs, which are valuable for bug fixing (Fig. 5, right);
(3) Self-verification for checking task success. Instead of manually coding success checkers
for each new task proposed by the automatic curriculum, we instantiate another GPT-4
agent for self-verification. By providing VOYAGER’s current state and the task to GPT-4,
we ask it to act as a critic [47–49] and inform us whether the program achieves the task.
In addition, if the task fails, it provides a critique by suggesting how to complete the task
(Fig. 6). Hence, our self-verification is more comprehensive than self-reflection [30] by both
checking success and reflecting on mistakes.
During each round of code generation, we execute the generated program to obtain environment
feedback and execution errors from the code interpreter, which are incorporated into GPT-4’s prompt
for the next round of code refinement. This iterative process repeats until self-verification validates
5


Inventory (8/36): {'oak_planks': 5, 'cobblestone': 2, 
'porkchop': 2, 'wooden_sword': 1, 'coal': 5, 'wooden_pickaxe': 
1, 'oak_log': 3, 'dirt': 9}
Task: Mine 5 coal ores
Inventory (10/36): {'raw_copper': 9, 'copper_ingot': 3, 
'acacia_planks': 1, 'raw_iron': 1, 'stick': 1, 'iron_sword': 1, 
'iron_pickaxe': 1, 'iron_ingot': 3, 'crafting_table': 1, 'furnace': 1}
Task: Craft a spyglass
Inventory (7/36): {'oak_log': 5, 'oak_planks': 1, 
'wooden_pickaxe': 1, 'wooden_sword': 1, 'porkchop': 2, 
'white_wool': 2, 'mutton': 6}
Task: Kill 3 sheep
Inventory (9/36): {'string': 4, 'coal': 1, 'rotten_flesh': 1, 
'iron_sword': 1, 'furnace': 1, 'dirt': 6, 'stone_shovel': 1, 
'wooden_pickaxe': 1, 'granite': 5}
Task: Kill 1 zombie
Reasoning: Mining coal_ore in Minecraft will get coal. You have 5 coal in your 
inventory.
Success: True
Reasoning: To craft a spyglass, you need 2 copper ingots and 1 amethyst shard. 
You have 3 copper ingots, but you don't have any amethyst shards.
Success: False
Critique: Find and mine an amethyst shard underground.
Reasoning: You have 2 white_wool and 6 mutton in your inventory, which indicates 
that you killed 2 sheep. You needed to kill 3 sheep.
Success: False
Critique: Find and kill one more sheep to complete the task.
Reasoning: You have 1 rotten_flesh in your inventory, which means you have killed 
at least 1 zombie.
Success: True
GPT-4
GPT-4
GPT-4
GPT-4
Figure 6: Self-verification examples. We only display the partial prompt for brevity. See Appendix,
Sec. A.5 for the full prompt structure.
the task’s completion, at which point we add this new skill to the skill library and ask the automatic
curriculum for a new objective (Fig. 2). If the agent gets stuck after 4 rounds of code generation, then
we query the curriculum for another task. This iterative prompting approach significantly improves
program synthesis for embodied control, enabling VOYAGER to continuously acquire diverse skills
without human intervention.
3
Experiments
3.1
Experimental Setup
We leverage OpenAI’s gpt-4-0314 [35] and gpt-3.5-turbo-0301 [50] APIs for text completion,
along with text-embedding-ada-002 [51] API for text embedding. We set all temperatures to
0 except for the automatic curriculum, which uses temperature = 0.1 to encourage task diversity. Our
simulation environment is built on top of MineDojo [23] and leverages Mineflayer [52] JavaScript
APIs for motor controls. See Appendix, Sec. B.1 for more details.
3.2
Baselines
Because there is no LLM-based agents that work out of the box for Minecraft, we make our best
effort to select a number of representative algorithms as baselines. These methods are originally
designed only for NLP tasks without embodiment, therefore we have to re-interpret them to be
executable in MineDojo and compatible with our experimental setting:
ReAct [29] uses chain-of-thought prompting [46] by generating both reasoning traces and action
plans with LLMs. We provide it with our environment feedback and the agent states as observations.
Reflexion [30] is built on top of ReAct [29] with self-reflection to infer more intuitive future actions.
We provide it with execution errors and our self-verification module.
AutoGPT [28] is a popular software tool that automates NLP tasks by decomposing a high-level
goal into multiple subgoals and executing them in a ReAct-style loop. We re-implement AutoGPT
by using GPT-4 to do task decomposition and provide it with the agent states, environment feedback,
and execution errors as observations for subgoal execution. Compared with VOYAGER, AutoGPT
lacks the skill library for accumulating knowledge, self-verification for assessing task success, and
automatic curriculum for open-ended exploration.
Note that we do not directly compare with prior methods that take Minecraft screen pixels as input
and output low-level controls [53–55]. It would not be an apple-to-apple comparison, because we rely
on the high-level Mineflayer [52] API to control the agent. Our work’s focus is on pushing the limits
of GPT-4 for lifelong embodied agent learning, rather than solving the 3D perception or sensorimotor
control problems. VOYAGER is orthogonal and can be combined with gradient-based approaches like
6


Table 1: Tech tree mastery. Fractions indicate the number of successful trials out of three total runs.
0/3 means the method fails to unlock a level of the tech tree within the maximal prompting iterations
(160). Numbers are prompting iterations averaged over three trials. The fewer the iterations, the
more efficient the method.
Method
Wooden Tool
Stone Tool
Iron Tool
Diamond Tool
ReAct [29]
N/A (0/
3)
N/A (0/
3)
N/A (0/
3)
N/A (0/
3)
Reflexion [30]
N/A (0/
3)
N/A (0/
3)
N/A (0/
3)
N/A (0/
3)
AutoGPT [28]
92 ± 72 (3/
3)
94 ± 72 (3/
3)
135 ± 103 (3/
3)
N/A (0/
3)
VOYAGER w/o Skill Library
7 ± 2 (3/
3)
9 ± 4 (3/
3)
29 ± 11 (3/
3)
N/A (0/
3)
VOYAGER (Ours)
6 ± 2 (3/
3)
11 ± 2 (3/
3)
21 ± 7 (3/
3)
102 (1/
3)
Figure 7: Map coverage: bird’s eye views of Minecraft maps. VOYAGER is able to traverse 2.3×
longer distances compared to baselines while crossing diverse terrains.
VPT [8] as long as the controller provides a code API. We make a system-level comparison between
VOYAGER and prior Minecraft agents in Table. A.2.
3.3
Evaluation Results
We systematically evaluate VOYAGER and baselines on their exploration performance, tech tree
mastery, map coverage, and zero-shot generalization capability to novel tasks in a new world.
Significantly better exploration.
Results of exploration performance are shown in Fig. 1.
VOYAGER’s superiority is evident in its ability to consistently make new strides, discovering 63
unique items within 160 prompting iterations, 3.3× many novel items compared to its counterparts.
On the other hand, AutoGPT lags considerably in discovering new items, while ReAct and Reflexion
struggle to make significant progress, given the abstract nature of the open-ended exploration goal
that is challenging to execute without an appropriate curriculum.
Consistent tech tree mastery. The Minecraft tech tree tests the agent’s ability to craft and use a
hierarchy of tools. Progressing through this tree (wooden tool →stone tool →iron tool →diamond
tool) requires the agent to master systematic and compositional skills. Compared with baselines,
VOYAGER unlocks the wooden level 15.3× faster (in terms of the prompting iterations), the stone
level 8.5× faster, the iron level 6.4× faster, and VOYAGER is the only one to unlock the diamond level
of the tech tree (Fig. 2 and Table. 1). This underscores the effectiveness of the automatic curriculum,
which consistently presents challenges of suitable complexity to facilitate the agent’s progress.
Extensive map traversal. VOYAGER is able to navigate distances 2.3× longer compared to baselines
by traversing a variety of terrains, while the baseline agents often find themselves confined to local
areas, which significantly hampers their capacity to discover new knowledge (Fig. 7).
7


Table 2: Zero-shot generalization to unseen tasks. Fractions indicate the number of successful
trials out of three total attempts. 0/3 means the method fails to solve the task within the maximal
prompting iterations (50). Numbers are prompting iterations averaged over three trials. The fewer
the iterations, the more efficient the method.
Method
Diamond Pickaxe
Golden Sword
Lava Bucket
Compass
ReAct [29]
N/A (0/
3)
N/A (0/
3)
N/A (0/
3)
N/A (0/
3)
Reflexion [30]
N/A (0/
3)
N/A (0/
3)
N/A (0/
3)
N/A (0/
3)
AutoGPT [28]
N/A (0/
3)
N/A (0/
3)
N/A (0/
3)
N/A (0/
3)
AutoGPT [28] w/ Our Skill Library
39 (1/
3)
30 (1/
3)
N/A (0/
3)
30 (2/
3)
VOYAGER w/o Skill Library
36 (2/
3)
30 ± 9 (3/
3)
27 ± 9 (3/
3)
26 ± 3 (3/
3)
VOYAGER (Ours)
19 ± 3 (3/
3)
18 ± 7 (3/
3)
21 ± 5 (3/
3)
18 ± 2 (3/
3)
Figure 8: Zero-shot generalization to unseen tasks. We visualize the intermediate progress of each
method on two tasks. See Appendix, Sec. B.4.3 for the other two tasks. We do not plot ReAct and
Reflexion since they do not make any meaningful progress.
Efficient zero-shot generalization to unseen tasks. To evaluate zero-shot generalization, we clear
the agent’s inventory, reset it to a newly instantiated world, and test it with unseen tasks. For both
VOYAGER and AutoGPT, we utilize GPT-4 to break down the task into a series of subgoals. Table. 2
and Fig. 8 show VOYAGER can consistently solve all the tasks, while baselines cannot solve any task
within 50 prompting iterations. What’s interesting to note is that our skill library constructed from
lifelong learning not only enhances VOYAGER’s performance but also gives a boost to AutoGPT.
This demonstrates that the skill library serves as a versatile tool that can be readily employed by other
methods, effectively acting as a plug-and-play asset to enhance performance.
3.4
Ablation Studies
We ablate 6 design choices (automatic curriculum, skill library, environment feedback, execution
errors, self-verification, and GPT-4 for code generation) in VOYAGER and study their impact on
exploration performance (see Appendix, Sec. B.3 for details of each ablated variant). Results are
shown in Fig. 9. We highlight the key findings below:
• Automatic curriculum is crucial for the agent’s consistent progress. The discovered item
count drops by 93% if the curriculum is replaced with a random one, because certain tasks
may be too challenging if attempted out of order. On the other hand, a manually designed
curriculum requires significant Minecraft-specific expertise, and does not take into account
the agent’s live situation. It falls short in the experimental results compared to our automatic
curriculum.
• VOYAGER w/o skill library exhibits a tendency to plateau in the later stages. This
underscores the pivotal role that the skill library plays in VOYAGER. It helps create more
complex actions and steadily pushes the agent’s boundaries by encouraging new skills to be
built upon older ones.
8


Figure 9: Left: Ablation studies for the automatic curriculum, skill library, and GPT-4. GPT-3.5
means replacing GPT-4 with GPT-3.5 for code generation. VOYAGER outperforms all the alternatives,
demonstrating the critical role of each component. Right: Ablation studies for the iterative
prompting mechanism. VOYAGER surpasses all the other options, thereby highlighting the essential
significance of each type of feedback in the iterative prompting mechanism.
Figure 10: VOYAGER builds 3D structures with human feedback. The progress of building designs
that integrate human input is demonstrated from left to right.
• Self-verification is the most important among all the feedback types. Removing the
module leads to a significant drop (−73%) in the discovered item count. Self-verification
serves as a critical mechanism to decide when to move on to a new task or reattempt a
previously unsuccessful task.
• GPT-4 significantly outperforms GPT-3.5 in code generation and obtains 5.7× more
unique items, as GPT-4 exhibits a quantum leap in coding abilities. This finding corroborates
recent studies in the literature [56, 57].
3.5
Multimodal Feedback from Humans
VOYAGER does not currently support visual perception, because the available version of GPT-4 API
is text-only at the time of this writing. However, VOYAGER has the potential to be augmented by
multimodal perception models [58, 59] to achieve more impressive tasks. We demonstrate that given
human feedback, VOYAGER is able to construct complex 3D structures in Minecraft, such as a Nether
Portal and a house (Fig. 10). There are two ways to integrate human feedback:
(1) Human as a critic (equivalent to VOYAGER’s self-verification module): humans provide
visual critique to VOYAGER, allowing it to modify the code from the previous round. This
feedback is essential for correcting certain errors in the spatial details of a 3D structure that
VOYAGER cannot perceive directly.
(2) Human as a curriculum (equivalent to VOYAGER’s automatic curriculum module): humans
break down a complex building task into smaller steps, guiding VOYAGER to complete them
incrementally. This approach improves VOYAGER’s ability to handle more sophisticated 3D
construction tasks.
9


4
Limitations and Future Work
Cost. The GPT-4 API incurs significant costs. It is 15× more expensive than GPT-3.5. Nevertheless,
VOYAGER requires the quantum leap in code generation quality from GPT-4 (Fig. 9), which GPT-3.5
and open-source LLMs cannot provide [60].
Inaccuracies. Despite the iterative prompting mechanism, there are still cases where the agent gets
stuck and fails to generate the correct skill. The automatic curriculum has the flexibility to reattempt
this task at a later time. Occasionally, self-verification module may also fail, such as not recognizing
spider string as a success signal of beating a spider.
Hallucinations. The automatic curriculum occasionally proposes unachievable tasks. For example, it
may ask the agent to craft a “copper sword" or “copper chestplate", which are items that do not exist
within the game. Hallucinations also occur during the code generation process. For instance, GPT-4
tends to use cobblestone as a fuel input, despite being an invalid fuel source in the game. Additionally,
it may call functions absent in the provided control primitive APIs, leading to code execution errors.
We are confident that improvements in the GPT API models as well as novel techniques for finetuning
open-source LLMs will overcome these limitations in the future.
5
Related work
Decision-making Agents in Minecraft.
Minecraft is an open-ended 3D world with incredibly
flexible game mechanics supporting a broad spectrum of activities. Built upon notable Minecraft
benchmarks [23, 61–65], Minecraft learning algorithms can be divided into two categories: 1)
Low-level controller: Many prior efforts leverage hierarchical reinforcement learning to learn from
human demonstrations [66–68]. Kanitscheider et al. [14] design a curriculum based on success rates,
but its objectives are limited to curated items. MineDojo [23] and VPT [8] utilize YouTube videos
for large-scale pre-training. DreamerV3 [69], on the other hand, learns a world model to explore
the environment and collect diamonds. 2) High-level planner: Volum et al. [70] leverage few-shot
prompting with Codex [41] to generate executable policies, but they require additional human
interaction. Recent works leverage LLMs as a high-level planner in Minecraft by decomposing
a high-level task into several subgoals following Minecraft recipes [55, 53, 71], thus lacking full
exploration flexibility. Like these latter works, VOYAGER also uses LLMs as a high-level planner by
prompting GPT-4 and utilizes Mineflayer [52] as a low-level controller following Volum et al. [70].
Unlike prior works, VOYAGER employs an automatic curriculum that unfolds in a bottom-up manner,
driven by curiosity, and therefore enables open-ended exploration.
Large Language Models for Agent Planning.
Inspired by the strong emergent capabilities of
LLMs, such as zero-shot prompting and complex reasoning [72, 37, 38, 36, 73, 74], embodied agent
research [75–78] has witnessed a significant increase in the utilization of LLMs for planning purposes.
Recent efforts can be roughly classified into two groups. 1) Large language models for robot
learning: Many prior works apply LLMs to generate subgoals for robot planning [27, 27, 25, 79, 80].
Inner Monologue [26] incorporates environment feedback for robot planning with LLMs. Code as
Policies [16] and ProgPrompt [22] directly leverage LLMs to generate executable robot policies.
VIMA [19] and PaLM-E [59] fine-tune pre-trained LLMs to support multimodal prompts. 2)
Large language models for text agents: ReAct [29] leverages chain-of-thought prompting [46] and
generates both reasoning traces and task-specific actions with LLMs. Reflexion [30] is built upon
ReAct [29] with self-reflection to enhance reasoning. AutoGPT [28] is a popular tool that automates
NLP tasks by crafting a curriculum of multiple subgoals for completing a high-level goal while
incorporating ReAct [29]’s reasoning and acting loops. DERA [81] frames a task as a dialogue
between two GPT-4 [35] agents. Generative Agents [82] leverages ChatGPT [50] to simulate human
behaviors by storing agents’ experiences as memories and retrieving those for planning, but its agent
actions are not executable. SPRING [83] is a concurrent work that uses GPT-4 to extract game
mechanics from game manuals, based on which it answers questions arranged in a directed acyclic
graph and predicts the next action. All these works lack a skill library for developing more complex
behaviors, which are crucial components for the success of VOYAGER in lifelong learning.
Code Generation with Execution.
Code generation has been a longstanding challenge in
NLP [41, 84, 85, 73, 37], with various works leveraging execution results to improve program
10


synthesis. Execution-guided approaches leverage intermediate execution outcomes to guide program
search [86–88]. Another line of research utilizes majority voting to choose candidates based on their
execution performance [89, 90]. Additionally, LEVER [91] trains a verifier to distinguish and reject
incorrect programs based on execution results. CLAIRIFY [92], on the other hand, generates code
for planning chemistry experiments and makes use of a rule-based verifier to iteratively provide
error feedback to LLMs. VOYAGER distinguishes itself from these works by integrating environment
feedback, execution errors, and self-verification (to assess task success) into an iterative prompting
mechanism for embodied control.
6
Conclusion
In this work, we introduce VOYAGER, the first LLM-powered embodied lifelong learning agent,
which leverages GPT-4 to explore the world continuously, develop increasingly sophisticated skills,
and make new discoveries consistently without human intervention. VOYAGER exhibits superior
performance in discovering novel items, unlocking the Minecraft tech tree, traversing diverse terrains,
and applying its learned skill library to unseen tasks in a newly instantiated world. VOYAGER serves
as a starting point to develop powerful generalist agents without tuning the model parameters.
7
Broader Impacts
Our research is conducted within Minecraft, a safe and harmless 3D video game environment. While
VOYAGER is designed to be generally applicable to other domains, such as robotics, its application to
physical robots would require additional attention and the implementation of safety constraints by
humans to ensure responsible and secure deployment.
8
Acknowledgements
We are extremely grateful to Ziming Zhu, Kaiyu Yang, Rafał Kocielnik, Colin White, Or Sharir, Sahin
Lale, De-An Huang, Jean Kossaifi, Yuncong Yang, Charles Zhang, Minchao Huang, and many other
colleagues and friends for their helpful feedback and insightful discussions. This work is done during
Guanzhi Wang’s internship at NVIDIA. Guanzhi Wang is supported by the Kortschak fellowship in
Computing and Mathematical Sciences at Caltech.
References
[1] Eric Kolve, Roozbeh Mottaghi, Winson Han, Eli VanderBilt, Luca Weihs, Alvaro Herrasti,
Daniel Gordon, Yuke Zhu, Abhinav Gupta, and Ali Farhadi. Ai2-thor: An interactive 3d
environment for visual ai. arXiv preprint arXiv: Arxiv-1712.05474, 2017.
[2] Manolis Savva, Jitendra Malik, Devi Parikh, Dhruv Batra, Abhishek Kadian, Oleksandr
Maksymets, Yili Zhao, Erik Wijmans, Bhavana Jain, Julian Straub, Jia Liu, and Vladlen
Koltun. Habitat: A platform for embodied AI research. In 2019 IEEE/CVF International
Conference on Computer Vision, ICCV 2019, Seoul, Korea (South), October 27 - November 2,
2019, pages 9338–9346. IEEE, 2019.
[3] Yuke Zhu, Josiah Wong, Ajay Mandlekar, and Roberto Martín-Martín. robosuite: A mod-
ular simulation framework and benchmark for robot learning. arXiv preprint arXiv: Arxiv-
2009.12293, 2020.
[4] Fei Xia, William B. Shen, Chengshu Li, Priya Kasimbeg, Micael Tchapmi, Alexander Toshev,
Li Fei-Fei, Roberto Martín-Martín, and Silvio Savarese. Interactive gibson benchmark (igibson
0.5): A benchmark for interactive navigation in cluttered environments. arXiv preprint arXiv:
Arxiv-1910.14442, 2019.
[5] Bokui Shen, Fei Xia, Chengshu Li, Roberto Martín-Martín, Linxi Fan, Guanzhi Wang, Claudia
Pérez-D’Arpino, Shyamal Buch, Sanjana Srivastava, Lyne P. Tchapmi, Micael E. Tchapmi, Kent
Vainio, Josiah Wong, Li Fei-Fei, and Silvio Savarese. igibson 1.0: a simulation environment for
interactive tasks in large realistic scenes. arXiv preprint arXiv: Arxiv-2012.02924, 2020.
11


[6] Jens Kober, J Andrew Bagnell, and Jan Peters. Reinforcement learning in robotics: A survey.
The International Journal of Robotics Research, 32(11):1238–1274, 2013.
[7] Kai Arulkumaran, Marc Peter Deisenroth, Miles Brundage, and Anil Anthony Bharath. Deep
reinforcement learning: A brief survey. IEEE Signal Processing Magazine, 34(6):26–38, 2017.
[8] Bowen Baker, Ilge Akkaya, Peter Zhokhov, Joost Huizinga, Jie Tang, Adrien Ecoffet, Brandon
Houghton, Raul Sampedro, and Jeff Clune. Video pretraining (vpt): Learning to act by watching
unlabeled online videos. arXiv preprint arXiv: Arxiv-2206.11795, 2022.
[9] DeepMind Interactive Agents Team, Josh Abramson, Arun Ahuja, Arthur Brussee, Federico
Carnevale, Mary Cassin, Felix Fischer, Petko Georgiev, Alex Goldin, Mansi Gupta, Tim
Harley, Felix Hill, Peter C Humphreys, Alden Hung, Jessica Landon, Timothy Lillicrap, Hamza
Merzic, Alistair Muldal, Adam Santoro, Guy Scully, Tamara von Glehn, Greg Wayne, Nathaniel
Wong, Chen Yan, and Rui Zhu. Creating multimodal interactive agents with imitation and
self-supervised learning. arXiv preprint arXiv: Arxiv-2112.03763, 2021.
[10] Oriol Vinyals, Igor Babuschkin, Junyoung Chung, Michael Mathieu, Max Jaderberg, Wo-
jciech M Czarnecki, Andrew Dudzik, Aja Huang, Petko Georgiev, Richard Powell, et al.
Alphastar: Mastering the real-time strategy game starcraft ii. DeepMind blog, 2, 2019.
[11] Adrien Ecoffet, Joost Huizinga, Joel Lehman, Kenneth O. Stanley, and Jeff Clune. Go-explore:
a new approach for hard-exploration problems. arXiv preprint arXiv: Arxiv-1901.10995, 2019.
[12] Joost Huizinga and Jeff Clune. Evolving multimodal robot behavior via many stepping stones
with the combinatorial multiobjective evolutionary algorithm.
Evolutionary computation,
30(2):131–164, 2022.
[13] Rui Wang, Joel Lehman, Aditya Rawal, Jiale Zhi, Yulun Li, Jeffrey Clune, and Kenneth O.
Stanley. Enhanced POET: open-ended reinforcement learning through unbounded invention of
learning challenges and their solutions. In Proceedings of the 37th International Conference on
Machine Learning, ICML 2020, 13-18 July 2020, Virtual Event, volume 119 of Proceedings of
Machine Learning Research, pages 9940–9951. PMLR, 2020.
[14] Ingmar Kanitscheider, Joost Huizinga, David Farhi, William Hebgen Guss, Brandon Houghton,
Raul Sampedro, Peter Zhokhov, Bowen Baker, Adrien Ecoffet, Jie Tang, Oleg Klimov, and Jeff
Clune. Multi-task curriculum learning in a complex, visual, hard-exploration domain: Minecraft.
arXiv preprint arXiv: Arxiv-2106.14876, 2021.
[15] Michael Dennis, Natasha Jaques, Eugene Vinitsky, Alexandre M. Bayen, Stuart Russell, Andrew
Critch, and Sergey Levine. Emergent complexity and zero-shot transfer via unsupervised
environment design. In Hugo Larochelle, Marc’Aurelio Ranzato, Raia Hadsell, Maria-Florina
Balcan, and Hsuan-Tien Lin, editors, Advances in Neural Information Processing Systems 33:
Annual Conference on Neural Information Processing Systems 2020, NeurIPS 2020, December
6-12, 2020, virtual, 2020.
[16] Jacky Liang, Wenlong Huang, Fei Xia, Peng Xu, Karol Hausman, Brian Ichter, Pete Florence,
and Andy Zeng. Code as policies: Language model programs for embodied control. arXiv
preprint arXiv: Arxiv-2209.07753, 2022.
[17] Shao-Hua Sun, Te-Lin Wu, and Joseph J. Lim. Program guided agent. In 8th International
Conference on Learning Representations, ICLR 2020, Addis Ababa, Ethiopia, April 26-30, 2020.
OpenReview.net, 2020.
[18] Zelin Zhao, Karan Samel, Binghong Chen, and Le Song. Proto: Program-guided transformer for
program-guided tasks. In Marc’Aurelio Ranzato, Alina Beygelzimer, Yann N. Dauphin, Percy
Liang, and Jennifer Wortman Vaughan, editors, Advances in Neural Information Processing
Systems 34: Annual Conference on Neural Information Processing Systems 2021, NeurIPS
2021, December 6-14, 2021, virtual, pages 17021–17036, 2021.
[19] Yunfan Jiang, Agrim Gupta, Zichen Zhang, Guanzhi Wang, Yongqiang Dou, Yanjun Chen,
Li Fei-Fei, Anima Anandkumar, Yuke Zhu, and Linxi (Jim) Fan. Vima: General robot manipu-
lation with multimodal prompts. ARXIV.ORG, 2022.
12


[20] Mohit Shridhar, Lucas Manuelli, and Dieter Fox. Cliport: What and where pathways for robotic
manipulation. arXiv preprint arXiv: Arxiv-2109.12098, 2021.
[21] Linxi Fan, Guanzhi Wang, De-An Huang, Zhiding Yu, Li Fei-Fei, Yuke Zhu, and Animashree
Anandkumar. SECANT: self-expert cloning for zero-shot generalization of visual policies. In
Marina Meila and Tong Zhang, editors, Proceedings of the 38th International Conference on
Machine Learning, ICML 2021, 18-24 July 2021, Virtual Event, volume 139 of Proceedings of
Machine Learning Research, pages 3088–3099. PMLR, 2021.
[22] Ishika Singh, Valts Blukis, Arsalan Mousavian, Ankit Goyal, Danfei Xu, Jonathan Tremblay,
Dieter Fox, Jesse Thomason, and Animesh Garg. Progprompt: Generating situated robot task
plans using large language models. arXiv preprint arXiv: Arxiv-2209.11302, 2022.
[23] Linxi Fan, Guanzhi Wang, Yunfan Jiang, Ajay Mandlekar, Yuncong Yang, Haoyi Zhu, Andrew
Tang, De-An Huang, Yuke Zhu, and Anima Anandkumar. Minedojo: Building open-ended
embodied agents with internet-scale knowledge. arXiv preprint arXiv: Arxiv-2206.08853, 2022.
[24] Andy Zeng, Adrian Wong, Stefan Welker, Krzysztof Choromanski, Federico Tombari, Aveek
Purohit, Michael Ryoo, Vikas Sindhwani, Johnny Lee, Vincent Vanhoucke, and Pete Florence.
Socratic models: Composing zero-shot multimodal reasoning with language. arXiv preprint
arXiv: Arxiv-2204.00598, 2022.
[25] Michael Ahn, Anthony Brohan, Noah Brown, Yevgen Chebotar, Omar Cortes, Byron David,
Chelsea Finn, Keerthana Gopalakrishnan, Karol Hausman, Alex Herzog, Daniel Ho, Jasmine
Hsu, Julian Ibarz, Brian Ichter, Alex Irpan, Eric Jang, Rosario Jauregui Ruano, Kyle Jeffrey,
Sally Jesmonth, Nikhil J Joshi, Ryan Julian, Dmitry Kalashnikov, Yuheng Kuang, Kuang-Huei
Lee, Sergey Levine, Yao Lu, Linda Luu, Carolina Parada, Peter Pastor, Jornell Quiambao,
Kanishka Rao, Jarek Rettinghouse, Diego Reyes, Pierre Sermanet, Nicolas Sievers, Clayton Tan,
Alexander Toshev, Vincent Vanhoucke, Fei Xia, Ted Xiao, Peng Xu, Sichun Xu, and Mengyuan
Yan. Do as i can, not as i say: Grounding language in robotic affordances. arXiv preprint arXiv:
Arxiv-2204.01691, 2022.
[26] Wenlong Huang, Fei Xia, Ted Xiao, Harris Chan, Jacky Liang, Pete Florence, Andy Zeng,
Jonathan Tompson, Igor Mordatch, Yevgen Chebotar, Pierre Sermanet, Noah Brown, Tomas
Jackson, Linda Luu, Sergey Levine, Karol Hausman, and Brian Ichter. Inner monologue:
Embodied reasoning through planning with language models. arXiv preprint arXiv: Arxiv-
2207.05608, 2022.
[27] Wenlong Huang, Pieter Abbeel, Deepak Pathak, and Igor Mordatch. Language models as zero-
shot planners: Extracting actionable knowledge for embodied agents. In Kamalika Chaudhuri,
Stefanie Jegelka, Le Song, Csaba Szepesvári, Gang Niu, and Sivan Sabato, editors, International
Conference on Machine Learning, ICML 2022, 17-23 July 2022, Baltimore, Maryland, USA,
volume 162 of Proceedings of Machine Learning Research, pages 9118–9147. PMLR, 2022.
[28] Significant-gravitas/auto-gpt: An experimental open-source attempt to make gpt-4 fully au-
tonomous., 2023.
[29] Shunyu Yao, Jeffrey Zhao, Dian Yu, Nan Du, Izhak Shafran, Karthik Narasimhan, and Yuan
Cao. React: Synergizing reasoning and acting in language models. arXiv preprint arXiv:
Arxiv-2210.03629, 2022.
[30] Noah Shinn, Beck Labash, and Ashwin Gopinath. Reflexion: an autonomous agent with
dynamic memory and self-reflection. arXiv preprint arXiv: Arxiv-2303.11366, 2023.
[31] German Ignacio Parisi, Ronald Kemker, Jose L. Part, Christopher Kanan, and Stefan Wermter.
Continual lifelong learning with neural networks: A review. Neural Networks, 113:54–71, 2019.
[32] Liyuan Wang, Xingxing Zhang, Hang Su, and Jun Zhu. A comprehensive survey of continual
learning: Theory, method and application. arXiv preprint arXiv: Arxiv-2302.00487, 2023.
[33] Volodymyr Mnih, Koray Kavukcuoglu, David Silver, Alex Graves, Ioannis Antonoglou, Daan
Wierstra, and Martin Riedmiller. Playing atari with deep reinforcement learning. arXiv preprint
arXiv: Arxiv-1312.5602, 2013.
13


[34] OpenAI, :, Christopher Berner, Greg Brockman, Brooke Chan, Vicki Cheung, Przemysław
D˛
ebiak, Christy Dennison, David Farhi, Quirin Fischer, Shariq Hashme, Chris Hesse, Rafal Józe-
fowicz, Scott Gray, Catherine Olsson, Jakub Pachocki, Michael Petrov, Henrique P. d. O. Pinto,
Jonathan Raiman, Tim Salimans, Jeremy Schlatter, Jonas Schneider, Szymon Sidor, Ilya
Sutskever, Jie Tang, Filip Wolski, and Susan Zhang. Dota 2 with large scale deep reinforcement
learning. arXiv preprint arXiv: Arxiv-1912.06680, 2019.
[35] OpenAI. Gpt-4 technical report. arXiv preprint arXiv: Arxiv-2303.08774, 2023.
[36] Jason Wei, Yi Tay, Rishi Bommasani, Colin Raffel, Barret Zoph, Sebastian Borgeaud, Dani
Yogatama, Maarten Bosma, Denny Zhou, Donald Metzler, Ed H. Chi, Tatsunori Hashimoto,
Oriol Vinyals, Percy Liang, Jeff Dean, and William Fedus. Emergent abilities of large language
models. arXiv preprint arXiv: Arxiv-2206.07682, 2022.
[37] Tom B. Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal,
Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel
Herbert-Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel M.
Ziegler, Jeffrey Wu, Clemens Winter, Christopher Hesse, Mark Chen, Eric Sigler, Mateusz
Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam McCandlish, Alec
Radford, Ilya Sutskever, and Dario Amodei. Language models are few-shot learners. In Hugo
Larochelle, Marc’Aurelio Ranzato, Raia Hadsell, Maria-Florina Balcan, and Hsuan-Tien Lin,
editors, Advances in Neural Information Processing Systems 33: Annual Conference on Neural
Information Processing Systems 2020, NeurIPS 2020, December 6-12, 2020, virtual, 2020.
[38] Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena,
Yanqi Zhou, Wei Li, and Peter J. Liu. Exploring the limits of transfer learning with a unified
text-to-text transformer. J. Mach. Learn. Res., 21:140:1–140:67, 2020.
[39] Benjamin Eysenbach, Abhishek Gupta, Julian Ibarz, and Sergey Levine. Diversity is all you
need: Learning skills without a reward function. In 7th International Conference on Learning
Representations, ICLR 2019, New Orleans, LA, USA, May 6-9, 2019. OpenReview.net, 2019.
[40] Edoardo Conti, Vashisht Madhavan, Felipe Petroski Such, Joel Lehman, Kenneth O. Stanley,
and Jeff Clune. Improving exploration in evolution strategies for deep reinforcement learning via
a population of novelty-seeking agents. In Samy Bengio, Hanna M. Wallach, Hugo Larochelle,
Kristen Grauman, Nicolò Cesa-Bianchi, and Roman Garnett, editors, Advances in Neural
Information Processing Systems 31: Annual Conference on Neural Information Processing
Systems 2018, NeurIPS 2018, December 3-8, 2018, Montréal, Canada, pages 5032–5043, 2018.
[41] Mark Chen, Jerry Tworek, Heewoo Jun, Qiming Yuan, Henrique Ponde de Oliveira Pinto,
Jared Kaplan, Harri Edwards, Yuri Burda, Nicholas Joseph, Greg Brockman, Alex Ray, Raul
Puri, Gretchen Krueger, Michael Petrov, Heidy Khlaaf, Girish Sastry, Pamela Mishkin, Brooke
Chan, Scott Gray, Nick Ryder, Mikhail Pavlov, Alethea Power, Lukasz Kaiser, Mohammad
Bavarian, Clemens Winter, Philippe Tillet, Felipe Petroski Such, Dave Cummings, Matthias
Plappert, Fotios Chantzis, Elizabeth Barnes, Ariel Herbert-Voss, William Hebgen Guss, Alex
Nichol, Alex Paino, Nikolas Tezak, Jie Tang, Igor Babuschkin, Suchir Balaji, Shantanu Jain,
William Saunders, Christopher Hesse, Andrew N. Carr, Jan Leike, Josh Achiam, Vedant Misra,
Evan Morikawa, Alec Radford, Matthew Knight, Miles Brundage, Mira Murati, Katie Mayer,
Peter Welinder, Bob McGrew, Dario Amodei, Sam McCandlish, Ilya Sutskever, and Wojciech
Zaremba. Evaluating large language models trained on code. arXiv preprint arXiv: Arxiv-
2107.03374, 2021.
[42] Rui Wang, Joel Lehman, Jeff Clune, and Kenneth O. Stanley. Paired open-ended trailblazer
(poet): Endlessly generating increasingly complex and diverse learning environments and their
solutions. arXiv preprint arXiv: Arxiv-1901.01753, 2019.
[43] Rémy Portelas, Cédric Colas, Lilian Weng, Katja Hofmann, and Pierre-Yves Oudeyer. Auto-
matic curriculum learning for deep RL: A short survey. In Christian Bessiere, editor, Proceedings
of the Twenty-Ninth International Joint Conference on Artificial Intelligence, IJCAI 2020, pages
4819–4825. ijcai.org, 2020.
14


[44] Sébastien Forestier, Rémy Portelas, Yoan Mollard, and Pierre-Yves Oudeyer. Intrinsically
motivated goal exploration processes with automatic curriculum learning. The Journal of
Machine Learning Research, 23(1):6818–6858, 2022.
[45] Kevin Ellis, Catherine Wong, Maxwell Nye, Mathias Sable-Meyer, Luc Cary, Lucas Morales,
Luke Hewitt, Armando Solar-Lezama, and Joshua B. Tenenbaum. Dreamcoder: Growing
generalizable, interpretable knowledge with wake-sleep bayesian program learning. arXiv
preprint arXiv: Arxiv-2006.08381, 2020.
[46] Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Ed Chi, Quoc Le, and Denny
Zhou. Chain of thought prompting elicits reasoning in large language models. arXiv preprint
arXiv: Arxiv-2201.11903, 2022.
[47] Volodymyr Mnih, Adrià Puigdomènech Badia, Mehdi Mirza, Alex Graves, Timothy P. Lillicrap,
Tim Harley, David Silver, and Koray Kavukcuoglu. Asynchronous methods for deep rein-
forcement learning. In Maria-Florina Balcan and Kilian Q. Weinberger, editors, Proceedings
of the 33nd International Conference on Machine Learning, ICML 2016, New York City, NY,
USA, June 19-24, 2016, volume 48 of JMLR Workshop and Conference Proceedings, pages
1928–1937. JMLR.org, 2016.
[48] John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal
policy optimization algorithms. arXiv preprint arXiv: Arxiv-1707.06347, 2017.
[49] Timothy P. Lillicrap, Jonathan J. Hunt, Alexander Pritzel, Nicolas Heess, Tom Erez, Yuval
Tassa, David Silver, and Daan Wierstra. Continuous control with deep reinforcement learning.
In Yoshua Bengio and Yann LeCun, editors, 4th International Conference on Learning Repre-
sentations, ICLR 2016, San Juan, Puerto Rico, May 2-4, 2016, Conference Track Proceedings,
2016.
[50] Introducing chatgpt, 2022.
[51] New and improved embedding model, 2022.
[52] PrismarineJS. Prismarinejs/mineflayer: Create minecraft bots with a powerful, stable, and high
level javascript api., 2013.
[53] Kolby Nottingham, Prithviraj Ammanabrolu, Alane Suhr, Yejin Choi, Hanna Hajishirzi, Sameer
Singh, and Roy Fox. Do embodied agents dream of pixelated sheep?: Embodied decision
making using language guided world modelling. ARXIV.ORG, 2023.
[54] Shaofei Cai, Zihao Wang, Xiaojian Ma, Anji Liu, and Yitao Liang. Open-world multi-task
control through goal-aware representation learning and adaptive horizon prediction. arXiv
preprint arXiv: Arxiv-2301.10034, 2023.
[55] Zihao Wang, Shaofei Cai, Anji Liu, Xiaojian Ma, and Yitao Liang. Describe, explain, plan and
select: Interactive planning with large language models enables open-world multi-task agents.
arXiv preprint arXiv: Arxiv-2302.01560, 2023.
[56] Sébastien Bubeck, Varun Chandrasekaran, Ronen Eldan, Johannes Gehrke, Eric Horvitz, Ece
Kamar, Peter Lee, Yin Tat Lee, Yuanzhi Li, Scott Lundberg, Harsha Nori, Hamid Palangi,
Marco Tulio Ribeiro, and Yi Zhang. Sparks of artificial general intelligence: Early experiments
with gpt-4. arXiv preprint arXiv: Arxiv-2303.12712, 2023.
[57] Yiheng Liu, Tianle Han, Siyuan Ma, Jiayue Zhang, Yuanyuan Yang, Jiaming Tian, Hao He,
Antong Li, Mengshen He, Zhengliang Liu, Zihao Wu, Dajiang Zhu, Xiang Li, Ning Qiang,
Dingang Shen, Tianming Liu, and Bao Ge. Summary of chatgpt/gpt-4 research and perspective
towards the future of large language models. arXiv preprint arXiv: Arxiv-2304.01852, 2023.
[58] Shikun Liu, Linxi Fan, Edward Johns, Zhiding Yu, Chaowei Xiao, and Anima Anandkumar.
Prismer: A vision-language model with an ensemble of experts. arXiv preprint arXiv: Arxiv-
2303.02506, 2023.
15


[59] Danny Driess, Fei Xia, Mehdi S. M. Sajjadi, Corey Lynch, Aakanksha Chowdhery, Brian Ichter,
Ayzaan Wahid, Jonathan Tompson, Quan Vuong, Tianhe Yu, Wenlong Huang, Yevgen Chebotar,
Pierre Sermanet, Daniel Duckworth, Sergey Levine, Vincent Vanhoucke, Karol Hausman, Marc
Toussaint, Klaus Greff, Andy Zeng, Igor Mordatch, and Pete Florence. Palm-e: An embodied
multimodal language model. arXiv preprint arXiv: Arxiv-2303.03378, 2023.
[60] Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timo-
thée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, Aurelien Rodriguez,
Armand Joulin, Edouard Grave, and Guillaume Lample. Llama: Open and efficient foundation
language models. arXiv preprint arXiv: Arxiv-2302.13971, 2023.
[61] William H. Guss, Brandon Houghton, Nicholay Topin, Phillip Wang, Cayden Codel, Manuela
Veloso, and Ruslan Salakhutdinov. Minerl: A large-scale dataset of minecraft demonstrations.
In Sarit Kraus, editor, Proceedings of the Twenty-Eighth International Joint Conference on
Artificial Intelligence, IJCAI 2019, Macao, China, August 10-16, 2019, pages 2442–2448.
ijcai.org, 2019.
[62] William H. Guss, Cayden Codel, Katja Hofmann, Brandon Houghton, Noboru Kuno, Stephanie
Milani, Sharada Mohanty, Diego Perez Liebana, Ruslan Salakhutdinov, Nicholay Topin,
Manuela Veloso, and Phillip Wang. The minerl 2019 competition on sample efficient re-
inforcement learning using human priors. arXiv preprint arXiv: Arxiv-1904.10079, 2019.
[63] William H. Guss, Mario Ynocente Castro, Sam Devlin, Brandon Houghton, Noboru Sean Kuno,
Crissman Loomis, Stephanie Milani, Sharada Mohanty, Keisuke Nakata, Ruslan Salakhutdinov,
John Schulman, Shinya Shiroshita, Nicholay Topin, Avinash Ummadisingu, and Oriol Vinyals.
The minerl 2020 competition on sample efficient reinforcement learning using human priors.
arXiv preprint arXiv: Arxiv-2101.11071, 2021.
[64] Anssi Kanervisto, Stephanie Milani, Karolis Ramanauskas, Nicholay Topin, Zichuan Lin, Jun-
you Li, Jianing Shi, Deheng Ye, Qiang Fu, Wei Yang, Weijun Hong, Zhongyue Huang, Haicheng
Chen, Guangjun Zeng, Yue Lin, Vincent Micheli, Eloi Alonso, François Fleuret, Alexander
Nikulin, Yury Belousov, Oleg Svidchenko, and Aleksei Shpilman. Minerl diamond 2021
competition: Overview, results, and lessons learned. arXiv preprint arXiv: Arxiv-2202.10583,
2022.
[65] Matthew Johnson, Katja Hofmann, Tim Hutton, and David Bignell. The malmo platform for
artificial intelligence experimentation. In Subbarao Kambhampati, editor, Proceedings of the
Twenty-Fifth International Joint Conference on Artificial Intelligence, IJCAI 2016, New York,
NY, USA, 9-15 July 2016, pages 4246–4247. IJCAI/AAAI Press, 2016.
[66] Zichuan Lin, Junyou Li, Jianing Shi, Deheng Ye, Qiang Fu, and Wei Yang. Juewu-mc: Playing
minecraft with sample-efficient hierarchical reinforcement learning. arXiv preprint arXiv:
Arxiv-2112.04907, 2021.
[67] Hangyu Mao, Chao Wang, Xiaotian Hao, Yihuan Mao, Yiming Lu, Chengjie Wu, Jianye
Hao, Dong Li, and Pingzhong Tang. Seihai: A sample-efficient hierarchical ai for the minerl
competition. arXiv preprint arXiv: Arxiv-2111.08857, 2021.
[68] Alexey Skrynnik, Aleksey Staroverov, Ermek Aitygulov, Kirill Aksenov, Vasilii Davydov, and
Aleksandr I. Panov. Hierarchical deep q-network from imperfect demonstrations in minecraft.
Cogn. Syst. Res., 65:74–78, 2021.
[69] Danijar Hafner, Jurgis Pasukonis, Jimmy Ba, and Timothy Lillicrap. Mastering diverse domains
through world models. arXiv preprint arXiv: Arxiv-2301.04104, 2023.
[70] Ryan Volum, Sudha Rao, Michael Xu, Gabriel DesGarennes, Chris Brockett, Benjamin
Van Durme, Olivia Deng, Akanksha Malhotra, and Bill Dolan. Craft an iron sword: Dy-
namically generating interactive game characters by prompting large language models tuned on
code. In Proceedings of the 3rd Wordplay: When Language Meets Games Workshop (Wordplay
2022), pages 25–43, Seattle, United States, 2022. Association for Computational Linguistics.
[71] Haoqi Yuan, Chi Zhang, Hongcheng Wang, Feiyang Xie, Penglin Cai, Hao Dong, and Zongqing
Lu. Plan4mc: Skill reinforcement learning and planning for open-world minecraft tasks. arXiv
preprint arXiv: 2303.16563, 2023.
16


[72] Rishi Bommasani, Drew A. Hudson, Ehsan Adeli, Russ Altman, Simran Arora, Sydney von Arx,
Michael S. Bernstein, Jeannette Bohg, Antoine Bosselut, Emma Brunskill, Erik Brynjolfsson,
Shyamal Buch, Dallas Card, Rodrigo Castellon, Niladri Chatterji, Annie Chen, Kathleen Creel,
Jared Quincy Davis, Dora Demszky, Chris Donahue, Moussa Doumbouya, Esin Durmus, Stefano
Ermon, John Etchemendy, Kawin Ethayarajh, Li Fei-Fei, Chelsea Finn, Trevor Gale, Lauren
Gillespie, Karan Goel, Noah Goodman, Shelby Grossman, Neel Guha, Tatsunori Hashimoto,
Peter Henderson, John Hewitt, Daniel E. Ho, Jenny Hong, Kyle Hsu, Jing Huang, Thomas
Icard, Saahil Jain, Dan Jurafsky, Pratyusha Kalluri, Siddharth Karamcheti, Geoff Keeling,
Fereshte Khani, Omar Khattab, Pang Wei Koh, Mark Krass, Ranjay Krishna, Rohith Kuditipudi,
Ananya Kumar, Faisal Ladhak, Mina Lee, Tony Lee, Jure Leskovec, Isabelle Levent, Xiang Lisa
Li, Xuechen Li, Tengyu Ma, Ali Malik, Christopher D. Manning, Suvir Mirchandani, Eric
Mitchell, Zanele Munyikwa, Suraj Nair, Avanika Narayan, Deepak Narayanan, Ben Newman,
Allen Nie, Juan Carlos Niebles, Hamed Nilforoshan, Julian Nyarko, Giray Ogut, Laurel Orr,
Isabel Papadimitriou, Joon Sung Park, Chris Piech, Eva Portelance, Christopher Potts, Aditi
Raghunathan, Rob Reich, Hongyu Ren, Frieda Rong, Yusuf Roohani, Camilo Ruiz, Jack
Ryan, Christopher Ré, Dorsa Sadigh, Shiori Sagawa, Keshav Santhanam, Andy Shih, Krishnan
Srinivasan, Alex Tamkin, Rohan Taori, Armin W. Thomas, Florian Tramèr, Rose E. Wang,
William Wang, Bohan Wu, Jiajun Wu, Yuhuai Wu, Sang Michael Xie, Michihiro Yasunaga,
Jiaxuan You, Matei Zaharia, Michael Zhang, Tianyi Zhang, Xikun Zhang, Yuhui Zhang, Lucia
Zheng, Kaitlyn Zhou, and Percy Liang. On the opportunities and risks of foundation models.
arXiv preprint arXiv: Arxiv-2108.07258, 2021.
[73] Aakanksha Chowdhery, Sharan Narang, Jacob Devlin, Maarten Bosma, Gaurav Mishra, Adam
Roberts, Paul Barham, Hyung Won Chung, Charles Sutton, Sebastian Gehrmann, Parker
Schuh, Kensen Shi, Sasha Tsvyashchenko, Joshua Maynez, Abhishek Rao, Parker Barnes,
Yi Tay, Noam Shazeer, Vinodkumar Prabhakaran, Emily Reif, Nan Du, Ben Hutchinson,
Reiner Pope, James Bradbury, Jacob Austin, Michael Isard, Guy Gur-Ari, Pengcheng Yin,
Toju Duke, Anselm Levskaya, Sanjay Ghemawat, Sunipa Dev, Henryk Michalewski, Xavier
Garcia, Vedant Misra, Kevin Robinson, Liam Fedus, Denny Zhou, Daphne Ippolito, David
Luan, Hyeontaek Lim, Barret Zoph, Alexander Spiridonov, Ryan Sepassi, David Dohan, Shivani
Agrawal, Mark Omernick, Andrew M. Dai, Thanumalayan Sankaranarayana Pillai, Marie Pellat,
Aitor Lewkowycz, Erica Moreira, Rewon Child, Oleksandr Polozov, Katherine Lee, Zongwei
Zhou, Xuezhi Wang, Brennan Saeta, Mark Diaz, Orhan Firat, Michele Catasta, Jason Wei,
Kathy Meier-Hellstern, Douglas Eck, Jeff Dean, Slav Petrov, and Noah Fiedel. Palm: Scaling
language modeling with pathways. arXiv preprint arXiv: Arxiv-2204.02311, 2022.
[74] Hyung Won Chung, Le Hou, Shayne Longpre, Barret Zoph, Yi Tay, William Fedus, Eric Li,
Xuezhi Wang, Mostafa Dehghani, Siddhartha Brahma, Albert Webson, Shixiang Shane Gu,
Zhuyun Dai, Mirac Suzgun, Xinyun Chen, Aakanksha Chowdhery, Sharan Narang, Gaurav
Mishra, Adams Yu, Vincent Zhao, Yanping Huang, Andrew Dai, Hongkun Yu, Slav Petrov,
Ed H. Chi, Jeff Dean, Jacob Devlin, Adam Roberts, Denny Zhou, Quoc V. Le, and Jason Wei.
Scaling instruction-finetuned language models. arXiv preprint arXiv: Arxiv-2210.11416, 2022.
[75] Jiafei Duan, Samson Yu, Hui Li Tan, Hongyuan Zhu, and Cheston Tan. A survey of embodied
AI: from simulators to research tasks. IEEE Trans. Emerg. Top. Comput. Intell., 6(2):230–244,
2022.
[76] Dhruv Batra, Angel X. Chang, Sonia Chernova, Andrew J. Davison, Jia Deng, Vladlen Koltun,
Sergey Levine, Jitendra Malik, Igor Mordatch, Roozbeh Mottaghi, Manolis Savva, and Hao Su.
Rearrangement: A challenge for embodied ai. arXiv preprint arXiv: Arxiv-2011.01975, 2020.
[77] Harish Ravichandar, Athanasios S Polydoros, Sonia Chernova, and Aude Billard. Recent
advances in robot learning from demonstration.
Annual review of control, robotics, and
autonomous systems, 3:297–330, 2020.
[78] Jack Collins, Shelvin Chand, Anthony Vanderkop, and David Howard. A review of physics
simulators for robotic applications. IEEE Access, 9:51416–51431, 2021.
[79] So Yeon Min, Devendra Singh Chaplot, Pradeep Ravikumar, Yonatan Bisk, and R. Salakhutdi-
nov. Film: Following instructions in language with modular methods. International Conference
on Learning Representations, 2021.
17


[80] Valts Blukis, Chris Paxton, Dieter Fox, Animesh Garg, and Yoav Artzi. A persistent spatial
semantic representation for high-level natural language instruction execution. In 5th Annual
Conference on Robot Learning, 2021.
[81] Varun Nair, Elliot Schumacher, Geoffrey Tso, and Anitha Kannan. Dera: Enhancing large
language model completions with dialog-enabled resolving agents. arXiv preprint arXiv:
Arxiv-2303.17071, 2023.
[82] Joon Sung Park, Joseph C. O’Brien, Carrie J. Cai, Meredith Ringel Morris, Percy Liang, and
Michael S. Bernstein. Generative agents: Interactive simulacra of human behavior. arXiv
preprint arXiv: Arxiv-2304.03442, 2023.
[83] Yue Wu, Shrimai Prabhumoye, So Yeon Min, Yonatan Bisk, Ruslan Salakhutdinov, Amos
Azaria, Tom Mitchell, and Yuanzhi Li. Spring: Gpt-4 out-performs rl algorithms by studying
papers and reasoning. arXiv preprint arXiv: 2305.15486, 2023.
[84] Erik Nijkamp, Bo Pang, Hiroaki Hayashi, Lifu Tu, Huan Wang, Yingbo Zhou, Silvio Savarese,
and Caiming Xiong. A conversational paradigm for program synthesis. arXiv preprint arXiv:
Arxiv-2203.13474, 2022.
[85] Hung Le, Yue Wang, Akhilesh Deepak Gotmare, Silvio Savarese, and Steven C. H. Hoi. Coderl:
Mastering code generation through pretrained models and deep reinforcement learning. arXiv
preprint arXiv: Arxiv-2207.01780, 2022.
[86] Xinyun Chen, Chang Liu, and Dawn Song. Execution-guided neural program synthesis. In 7th
International Conference on Learning Representations, ICLR 2019, New Orleans, LA, USA,
May 6-9, 2019. OpenReview.net, 2019.
[87] Xinyun Chen, Dawn Song, and Yuandong Tian. Latent execution for neural program synthesis.
arXiv preprint arXiv: Arxiv-2107.00101, 2021.
[88] Kevin Ellis, Maxwell I. Nye, Yewen Pu, Felix Sosa, Josh Tenenbaum, and Armando Solar-
Lezama. Write, execute, assess: Program synthesis with a REPL. In Hanna M. Wallach, Hugo
Larochelle, Alina Beygelzimer, Florence d’Alché-Buc, Emily B. Fox, and Roman Garnett,
editors, Advances in Neural Information Processing Systems 32: Annual Conference on Neural
Information Processing Systems 2019, NeurIPS 2019, December 8-14, 2019, Vancouver, BC,
Canada, pages 9165–9174, 2019.
[89] Yujia Li, David Choi, Junyoung Chung, Nate Kushman, Julian Schrittwieser, Rémi Leblond,
Tom Eccles, James Keeling, Felix Gimeno, Agustin Dal Lago, Thomas Hubert, Peter Choy,
Cyprien de Masson d’Autume, Igor Babuschkin, Xinyun Chen, Po-Sen Huang, Johannes Welbl,
Sven Gowal, Alexey Cherepanov, James Molloy, Daniel J. Mankowitz, Esme Sutherland Robson,
Pushmeet Kohli, Nando de Freitas, Koray Kavukcuoglu, and Oriol Vinyals. Competition-level
code generation with alphacode. arXiv preprint arXiv: Arxiv-2203.07814, 2022.
[90] Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser,
Matthias Plappert, Jerry Tworek, Jacob Hilton, Reiichiro Nakano, Christopher Hesse, and
John Schulman. Training verifiers to solve math word problems. arXiv preprint arXiv: Arxiv-
2110.14168, 2021.
[91] Ansong Ni, Srini Iyer, Dragomir Radev, Ves Stoyanov, Wen tau Yih, Sida I. Wang, and
Xi Victoria Lin. Lever: Learning to verify language-to-code generation with execution. arXiv
preprint arXiv: Arxiv-2302.08468, 2023.
[92] Marta Skreta, Naruki Yoshikawa, Sebastian Arellano-Rubach, Zhi Ji, Lasse Bjørn Kristensen,
Kourosh Darvish, Alán Aspuru-Guzik, Florian Shkurti, and Animesh Garg. Errors are useful
prompts: Instruction guided task programming with verifier-assisted iterative prompting. arXiv
preprint arXiv: Arxiv-2303.14100, 2023.
18


A
Method
A.1
VOYAGER Algorithm
Pseudocode 1: VOYAGER algorithm.
def
voyager(
environment ,
# environment
that uses code as action
space
curriculum_agent ,
# curriculum
agent for
proposing
the next task
action_agent ,
# action
agent for code
generation
critic_agent ,
# critic
agent for self -verification
skill_manager ,
# skill
manager
for adding new skills and skill
retrieval
):
agent_state = environment.reset ()
while
True:
exploration_progress = (
curriculum_agent . get_exploration_progress (
curriculum_agent . get_completed_tasks (),
curriculum_agent . get_failed_tasks (),
)
)
task = curriculum_agent . propose_next_task (
agent_state , exploration_progress
)
code = None
environment_feedback = None
execution_errors = None
critique = None
success = False
# try at most 4 rounds
before
moving on to the next task
for i in range (4):
skills = skill_manager . retrieve_skills (
task , environment_feedback
)
code = action_agent. generate_code (
task ,
code ,
environment_feedback ,
execution_errors ,
critique ,
skills ,
)
(
agent_state ,
environment_feedback ,
execution_errors ,
) = environment.step(code)
success , critique = critic_agent. check_task_success (
task , agent_state
)
if success:
break
if success:
skill_manager.add_skill(code)
curriculum_agent . add_completed_task (task)
else:
curriculum_agent . add_failed_task (task)
A.2
Prompting
GPT-4 and GPT-3.5 offer users the ability to designate the role of each prompt message among three
options:
19


• System: A high-level instruction that guides the model behavior throughout the conversation.
It sets the overall tone and objective for the interaction.
• User: A detailed instruction that guides the assistant for the next immediate response.
• Assistant: A response message generated the model.
See https://platform.openai.com/docs/guides/chat/introduction for more details.
To save token usage, instead of engaging in multi-round conversations, we concatenate a system
prompt and a user prompt to obtain each assistant’s response.
A.3
Automatic Curriculum
A.3.1
Components in the Prompt
The input prompt to GPT-4 consists of several components:
(1) Directives encouraging diverse behaviors and imposing constraints (so that the proposed
task is achievable and verifiable): See Sec. A.3.4 for the full prompt;
(2) The agent’s current state:
• Inventory: A dictionary of items with counts, for example, {‘cobblestone’: 4, ‘furnace’:
1, ‘stone_pickaxe’: 1, ‘oak_planks’: 7, ‘dirt’: 6, ‘wooden_pickaxe’: 1, ‘crafting_table’:
1, ‘raw_iron’: 4, ‘coal’: 1};
• Equipment: Armors or weapons equipped by the agents;
• Nearby blocks: A set of block names within a 32-block distance to the agent, for
example, ‘dirt’, ‘water’, ‘spruce_planks’, ‘grass_block’, ‘dirt_path’, ‘sugar_cane’,
‘fern’;
• Other blocks that are recently seen: Blocks that are not nearby or in the inventory;
• Nearby entities: A set of entity names within a 32-block distance to the agent, for
example, ‘pig’, ‘cat’, ‘villager’, ‘zombie’;
• A list of chests that are seen by the agent: Chests are external containers where the
agent can deposit items. If a chest is not opened before, its content is “Unknown”.
Otherwise, the items inside each chest are shown to the agent.
• Biome: For example, ‘plains’, ‘flower_forest’, ‘meadow’, ‘river’, ‘beach’, ‘for-
est’, ‘snowy_slopes’, ‘frozen_peaks’, ‘old_growth_birch_forest’, ‘ocean’, ‘sun-
flower_plains’, ‘stony_shore’;
• Time: One of ‘sunrise’, ‘day’, ‘noon’, ‘sunset’, ‘night’, ‘midnight’;
• Health and hunger bars: The max value is 20;
• Position: 3D coordinate (x, y, z) of the agent’s position in the Minecraft world;
(3) Previously completed and failed tasks;
(4) Additional context: See Sec. A.3.2;
(5) Chain-of-thought prompting [46] in response: We request GPT-4 to first reason about the
current progress and then suggest the next task.
A.3.2
Additional Context
We leverage GPT-3.5 to self-ask questions to provide additional context. Each question is paired with
a concept that is used for retrieving the most relevant document from the wiki knowledge base [23].
We feed the document content to GPT-3.5 for self-answering questions. In practice, using a wiki
knowledge base is optional since GPT-3.5 already possesses a good understanding of Minecraft
game mechanics. However, the external knowledge base becomes advantageous if GPT-3.5 is not
pre-trained in that specific domain. See Sec. A.3.4 for the full prompt.
A.3.3
Warm-up Schedule
In practice, we adopt a warm-up schedule to gradually incorporate the agent’s state and the additional
context into the prompt based on how many tasks the agent has completed. This ensures that the
prompt is exposed to increasing amounts of information over the exploration progress and therefore
20


begins with basic skills and progressively advances towards more intricate and diverse ones. The
warm-up setting that we use across all the experiments is shown in Table. A.1.
Table A.1: Warm-up schedule for automatic curriculum.
Information in the prompt
After how many tasks are completed
core inventory (only including log, planks, stick,
crafting table, furnace, dirt, coal, pickaxe, sword,
and axe)
0
equipment
0
nearby blocks
0
position
0
nearby entities
5
full inventory
7
other blocks that are recently seen
10
biome
10
health bar
15
hunger bar
15
time
15
additional context
15
A.3.4
Full Prompt
Prompt 1: Full system prompt for automatic curriculum. The list of question-answer pairs represents
the additional context.
You are a helpful
assistant
that
tells me the next
immediate
task to
do in Minecraft. My ultimate
goal is to discover as many
diverse
things as possible , accomplish as many
diverse
tasks as possible
and become the best
Minecraft
player in the world.
I will give you the
following
information:
Question 1: ...
Answer: ...
Question 2: ...
Answer: ...
Question 3: ...
Answer: ...
...
Biome: ...
Time: ...
Nearby
blocks: ...
Other
blocks
that are
recently
seen: ...
Nearby
entities (nearest to farthest): ...
Health: Higher
than 15 means I’m healthy.
Hunger: Higher
than 15 means I’m not hungry.
Position: ...
Equipment: If I have
better
armor in my inventory , you should ask me
to equip it.
Inventory (xx /36): ...
Chests: You can ask me to deposit or take
items
from
these
chests.
There
also
might be some
unknown chest , you should ask me to open
and check
items
inside the
unknown
chest.
Completed
tasks so far: ...
Failed
tasks
that are too hard: ...
You must
follow the
following
criteria:
1) You should act as a mentor and guide me to the next task
based on
my current
learning
progress.
2) Please be very
specific
about
what
resources I need to collect ,
what I need to craft , or what mobs I need to kill.
21


3) The next task
should
follow a concise
format , such as "Mine [
quantity] [block ]", "Craft [quantity] [item]", "Smelt [quantity] [
item]", "Kill [quantity] [mob]", "Cook [quantity] [food]", "Equip
[item ]" etc. It should be a single
phrase. Do not
propose
multiple
tasks at the same time. Do not
mention
anything
else.
4) The next task
should not be too hard
since I may not have the
necessary
resources or have
learned
enough
skills to complete it
yet.
5) The next task
should be novel and
interesting. I should
look for
rare
resources , upgrade my equipment
and tools
using
better
materials , and
discover
new things. I should not be doing the same
thing
over and over
again.
6) I may
sometimes
need to repeat
some
tasks if I need to collect
more
resources to complete
more
difficult
tasks. Only
repeat
tasks if
necessary.
7) Do not ask me to build or dig
shelter
even if it ’s at night. I want
to explore
the world and
discover
new things. I don ’t want to
stay in one place.
8) Tasks
that
require
information
beyond the player ’s status to verify
should be avoided. For instance , "Placing 4 torches" and "Dig a 2
x1x2 hole" are not ideal
since
they
require
visual
confirmation
from the screen. All the placing , building , planting , and
trading
tasks
should be avoided. Do not
propose
task
starting
with
these
keywords.
You should
only
respond in the format as described
below:
RESPONSE
FORMAT:
Reasoning: Based on the
information I listed above , do reasoning
about
what the next task
should be.
Task: The next task.
Here ’s an example
response:
Reasoning: The
inventory is empty now , chop down a tree to get some
wood.
Task: Obtain a wood log.
Prompt 2: Full system prompt for asking questions. We provide both good and bad examples as
few-shot exemplars.
You are a helpful
assistant
that asks
questions to help me decide the
next
immediate
task to do in Minecraft. My ultimate
goal is to
discover as many
things as possible , accomplish as many
tasks as
possible
and become the best
Minecraft
player in the world.
I will give you the
following
information:
Biome: ...
Time: ...
Nearby
blocks: ...
Other
blocks
that are
recently
seen: ...
Nearby
entities (nearest to farthest): ...
Health: ...
Hunger: ...
Position: ...
Equipment: ...
Inventory (xx /36): ...
Chests: ...
Completed
tasks so far: ...
Failed
tasks
that are too hard: ...
You must
follow the
following
criteria:
1) You should ask at least 5 questions (but no more than 10 questions)
to help me decide the next
immediate
task to do. Each
question
should be followed by the
concept
that the
question is about.
2) Your
question
should be specific to a concept in Minecraft.
Bad
example (the
question is too
general):
22


Question: What is the best way to play
Minecraft?
Concept: unknown
Bad
example (axe is still general , you should
specify
the type of
axe such as wooden axe):
What are the
benefits of using an axe to gather
resources?
Concept: axe
Good
example:
Question: How to make a wooden
pickaxe?
Concept: wooden
pickaxe
3) Your
questions
should be self -contained
and not
require
any
context
.
Bad
example (the
question
requires
the
context of my current
biome):
Question: What are the blocks
that I can find in my current
biome?
Concept: unknown
Bad
example (the
question
requires
the
context of my current
inventory):
Question: What are the
resources
you need the most
currently?
Concept: unknown
Bad
example (the
question
requires
the
context of my current
inventory):
Question: Do you have any gold or emerald
resources?
Concept: gold
Bad
example (the
question
requires
the
context of my nearby
entities
):
Question: Can you see any
animals
nearby
that you can kill for
food?
Concept: food
Bad
example (the
question
requires
the
context of my nearby
blocks):
Question: Is there any water
source
nearby?
Concept: water
Good
example:
Question: What are the blocks
that I can find in the sparse
jungle
?
Concept: sparse
jungle
4) Do not ask
questions
about
building
tasks (such as building a
shelter) since
they are too hard for me to do.
Let ’s say your
current
biome is sparse
jungle. You can ask
questions
like:
Question: What are the items
that I can find in the sparse
jungle?
Concept: sparse
jungle
Question: What are the mobs that I can find in the sparse
jungle?
Concept: sparse
jungle
Let ’s say you see a creeper
nearby , and you have not
defeated a
creeper
before. You can ask a question
like:
Question: How to defeat the
creeper?
Concept: creeper
Let ’s say you last
completed
task is "Craft a wooden
pickaxe ". You can
ask a question
like:
Question: What are the
suggested
tasks
that I can do after
crafting a
wooden
pickaxe?
Concept: wooden
pickaxe
Here are some more
question
and
concept
examples:
Question: What are the ores that I can find in the sparse
jungle?
Concept: sparse
jungle
(the above
concept
should not be "ore" because I need to look up the
page of "sparse
jungle" to find out what ores I can find in the
sparse
jungle)
Question: How can you obtain
food in the sparse
jungle?
Concept: sparse
jungle
23


(the above
concept
should not be "food" because I need to look up the
page of "sparse
jungle" to find out what food I can obtain in the
sparse
jungle)
Question: How can you use the
furnace to upgrade
your
equipment
and
make
useful
items?
Concept: furnace
Question: How to obtain a diamond
ore?
Concept: diamond
ore
Question: What are the
benefits of using a stone
pickaxe
over a wooden
pickaxe?
Concept: stone
pickaxe
Question: What are the tools
that you can craft
using
wood
planks and
sticks?
Concept: wood
planks
You should
only
respond in the format as described
below:
RESPONSE
FORMAT:
Reasoning: ...
Question 1: ...
Concept 1: ...
Question 2: ...
Concept 2: ...
Question 3: ...
Concept 3: ...
Question 4: ...
Concept 4: ...
Question 5: ...
Concept 5: ...
...
Prompt 3: Full system prompt for answering questions. Context represents the optional content from
a wiki knowledge base.
You are a helpful
assistant
that
answer my question
about
Minecraft.
I will give you the
following
information:
Question: ...
You will
answer the
question
based on the
context (only if available
and
helpful) and your own
knowledge of Minecraft.
1) Start
your
answer
with "Answer: ".
2) Answer "Answer: Unknown" if you don ’t know the answer.
A.4
Skill Library
A.4.1
Components in the Prompt
The input prompt to GPT-4 consists of the following components:
(1) Guidelines for code generation: See Sec A.4.2 for the full prompt;
(2) Control primitive APIs implemented by us: These APIs serve a dual purpose: they demon-
strate the usage of Mineflayer APIs, and they can be directly called by GPT-4.
• exploreUntil(bot, direction, maxTime = 60, callback): Allow the agent
to explore in a fixed direction for maxTime. The callback is the stopping condition
implemented by the agent to determine when to stop exploring;
• mineBlock(bot, name, count = 1): Mine and collect the specified number of
blocks within a 32-block distance;
• craftItem(bot, name, count = 1): Craft the item with a crafting table nearby;
• placeItem(bot, name, position): Place the block at the specified position;
• smeltItem(bot, itemName, fuelName, count = 1): Smelt the item with the
specified fuel. There must be a furnace nearby;
24


• killMob(bot, mobName, timeout = 300):
Attack the mob and collect its
dropped item;
• getItemFromChest(bot, chestPosition, itemsToGet): Move to the chest at
the specified position and get items from the chest;
• depositItemIntoChest(bot, chestPosition, itemsToDeposit):
Move to
the chest at the specified position and deposit items into the chest;
(3) Control primitive APIs provided by Mineflayer:
• await bot.pathfinder.goto(goal): Go to a specific position. See below for how
to set the goal;
• new GoalNear(x, y, z, range): Move the bot to a block within the specified
range of the specified block;
• new GoalXZ(x, z): For long-range goals that don’t have a specific Y level;
• new GoalGetToBlock(x, y, z): Not get into the block, but get directly adjacent
to it. Useful for fishing, farming, filling a bucket, and using a bed.;
• new GoalFollow(entity, range): Follow the specified entity within the specified
range;
• new GoalPlaceBlock(position, bot.world, {}): Position the bot in order to
place a block;
• new GoalLookAtBlock(position, bot.world, {}): Path towards a position
where a face of the block at position is visible;
• bot.isABed(bedBlock): Return true if bedBlock is a bed;
• bot.blockAt(position): Return the block at position;
• await bot.equip(item, destination): Equip the item in the specified destina-
tion. destination must be one of “hand”, “head”, “torso”, “legs”, “feet”, “off-hand”;
• await bot.consume(): Consume the item in the bot’s hand. You must equip the
item to consume first. Useful for eating food, drinking potions, etc.;
• await bot.fish(): Let bot fish. Before calling this function, you must first get to a
water block and then equip a fishing rod. The bot will automatically stop fishing when
it catches a fish;
• await bot.sleep(bedBlock): Sleep until sunrise. You must get to a bed block
first;
• await bot.activateBlock(block): This is the same as right-clicking a block in
the game. Useful for buttons, doors, etc. You must get to the block first;
• await bot.lookAt(position): Look at the specified position. You must go near
the position before you look at it. To fill a bucket with water, you must look at it first;
• await bot.activateItem(): This is the same as right-clicking to use the item in
the bot’s hand. Useful for using a bucket, etc. You must equip the item to activate first;
• await bot.useOn(entity): This is the same as right-clicking an entity in the game.
Useful for shearing a sheep. You must get to the entity first;
(4) Retrieved skills from the skill library;
(5) Generated code from the last round;
(6) Environment feedback: The chat log in the prompt;
(7) Execution errors;
(8) Critique from the self-verification module;
(9) The agent’s current state: See Sec. A.3.1 for each element of the agent’s state;
(10) Task proposed by the automatic curriculum;
(11) Task context: We prompt GPT-3.5 to ask for general suggestions about how to solve the
task. In practice, this part is handled by the automatic curriculum since it has a systematic
mechanism for question-answering (Sec. A.3.2);
(12) Chain-of-thought prompting [46] in response: We ask GPT-4 to first explain the reason why
the code from the last round fails, then give step-by-step plans to finish the task, and finally
generate code. See Sec. A.4.2 for the full prompt.
25


A.4.2
Full Prompt
Prompt 4: Full system prompt for code generation.
You are a helpful
assistant
that
writes
Mineflayer
javascript
code to
complete
any
Minecraft
task
specified by me.
Here are some
useful
programs
written
with
Mineflayer
APIs.
/*
Explore
until
find an iron_ore , use Vec3(0, -1, 0) because
iron ores
are
usually
underground
await
exploreUntil(bot , new Vec3(0, -1, 0), 60, () => {
const
iron_ore = bot.findBlock ({
matching: mcData.blocksByName [" iron_ore "].id ,
maxDistance: 32,
});
return
iron_ore;
});
Explore
until
find a pig , use Vec3(1, 0, 1) because
pigs are
usually
on the
surface
let pig = await
exploreUntil(bot , new Vec3(1, 0, 1), 60, () => {
const pig = bot.nearestEntity (( entity) => {
return (
entity.name === "pig" &&
entity.position.distanceTo(bot.entity.position) < 32
);
});
return pig;
});
*/
async
function
exploreUntil (bot , direction , maxTime = 60, callback) {
/*
Implementation of this
function is omitted.
direction: Vec3 , can only
contain
value of
-1, 0 or 1
maxTime: number , the max time for
exploration
callback: function , early
stop
condition , will be called
each
second , exploration
will stop if return
value is not null
Return: null if explore
timeout , otherwise
return the return
value
of callback
*/
}
// Mine 3 cobblestone: mineBlock(bot , "stone", 3);
async
function
mineBlock(bot , name , count = 1) {
const
blocks = bot.findBlocks ({
matching: (block) => {
return
block.name === name;
},
maxDistance: 32,
count: count ,
});
const
targets = [];
for (let i = 0; i < Math.min(blocks.length , count); i++) {
targets.push(bot.blockAt(blocks[i]));
}
await bot.collectBlock.collect(targets , { ignoreNoPath: true });
}
// Craft 8 oak_planks
from 2 oak_log (do the recipe 2 times):
craftItem(bot , "oak_planks", 2);
26


// You must
place a crafting
table
before
calling
this
function
async
function
craftItem(bot , name , count = 1) {
const
item = mcData.itemsByName[name ];
const
craftingTable = bot.findBlock ({
matching: mcData.blocksByName . crafting_table .id ,
maxDistance: 32,
});
await bot.pathfinder.goto(
new
GoalLookAtBlock ( craftingTable .position , bot.world)
);
const
recipe = bot.recipesFor(item.id , null , 1, craftingTable)[0];
await bot.craft(recipe , count , craftingTable );
}
// Place a crafting_table
near the player , Vec3(1, 0, 0) is just an
example , you shouldn ’t always use that: placeItem(bot , "
crafting_table ", bot.entity.position.offset (1, 0, 0));
async
function
placeItem(bot , name , position) {
const
item = bot.inventory. findInventoryItem (mcData.itemsByName[
name ].id);
// find a reference
block
const
faceVectors = [
new Vec3(0, 1, 0),
new Vec3(0, -1, 0),
new Vec3(1, 0, 0),
new Vec3(-1, 0, 0),
new Vec3(0, 0, 1),
new Vec3(0, 0,
-1),
];
let
referenceBlock = null;
let
faceVector = null;
for (const
vector of faceVectors) {
const
block = bot.blockAt(position.minus(vector));
if (block ?. name !== "air") {
referenceBlock = block;
faceVector = vector;
break;
}
}
// You must
first go to the block
position
you want to place
await bot.pathfinder.goto(new
GoalPlaceBlock (position , bot.world ,
{}));
// You must
equip the item
right
before
calling
placeBlock
await bot.equip(item , "hand ");
await bot.placeBlock(referenceBlock , faceVector);
}
// Smelt 1 raw_iron
into 1 iron_ingot
using 1 oak_planks as fuel:
smeltItem(bot , "raw_iron", "oak_planks ");
// You must
place a furnace
before
calling
this
function
async
function
smeltItem(bot , itemName , fuelName , count = 1) {
const
item = mcData.itemsByName[itemName ];
const
fuel = mcData.itemsByName[fuelName ];
const
furnaceBlock = bot.findBlock ({
matching: mcData.blocksByName .furnace.id ,
maxDistance: 32,
});
await bot.pathfinder.goto(
new
GoalLookAtBlock (furnaceBlock.position , bot.world)
);
const
furnace = await bot.openFurnace(furnaceBlock);
for (let i = 0; i < count; i++) {
await
furnace.putFuel(fuel.id , null , 1);
27


await
furnace.putInput(item.id , null , 1);
// Wait 12 seconds
for the
furnace to smelt the item
await bot.waitForTicks (12 * 20);
await
furnace.takeOutput ();
}
await
furnace.close ();
}
// Kill a pig and
collect
the
dropped
item: killMob(bot , "pig", 300);
async
function
killMob(bot , mobName , timeout = 300) {
const
entity = bot.nearestEntity (
(entity) =>
entity.name ===
mobName &&
entity.position.distanceTo(bot.entity.position) < 32
);
await bot.pvp.attack(entity);
await bot.pathfinder.goto(
new
GoalBlock(entity.position.x, entity.position.y, entity.
position.z)
);
}
// Get a torch
from
chest at (30, 65, 100): getItemFromChest (bot , new
Vec3 (30, 65, 100) , {" torch ": 1});
// This
function
will work no matter how far the bot is from the chest
.
async
function
getItemFromChest (bot , chestPosition , itemsToGet) {
await
moveToChest(bot , chestPosition );
const
chestBlock = bot.blockAt( chestPosition );
const
chest = await bot.openContainer (chestBlock);
for (const
name in itemsToGet) {
const
itemByName = mcData.itemsByName[name ];
const
item = chest. findContainerItem (itemByName.id);
await
chest.withdraw(item.type , null , itemsToGet[name ]);
}
await
closeChest(bot , chestBlock);
}
// Deposit a torch
into
chest at (30, 65, 100): depositItemIntoChest (
bot , new Vec3 (30, 65, 100) , {" torch ": 1});
// This
function
will work no matter how far the bot is from the chest
.
async
function
depositItemIntoChest (bot , chestPosition , itemsToDeposit
) {
await
moveToChest(bot , chestPosition );
const
chestBlock = bot.blockAt( chestPosition );
const
chest = await bot.openContainer (chestBlock);
for (const
name in itemsToDeposit ) {
const
itemByName = mcData.itemsByName[name ];
const
item = bot.inventory. findInventoryItem (itemByName.id);
await
chest.deposit(item.type , null , itemsToDeposit [name ]);
}
await
closeChest(bot , chestBlock);
}
// Check the items
inside the chest at (30, 65, 100):
checkItemInsideChest (bot , new Vec3 (30, 65, 100));
// You only need to call this
function
once
without
any action to
finish
task of checking
items
inside the chest.
async
function
checkItemInsideChest (bot , chestPosition ) {
await
moveToChest(bot , chestPosition );
const
chestBlock = bot.blockAt( chestPosition );
await bot.openContainer (chestBlock);
// You must
close the chest
after
opening it if you are asked to
open a chest
28


await
closeChest(bot , chestBlock);
}
await bot.pathfinder.goto(goal); // A very
useful
function. This
function
may change
your main -hand
equipment.
// Following
are some
Goals you can use:
new
GoalNear(x, y, z, range); // Move the bot to a block
within the
specified
range of the
specified
block. ‘x‘, ‘y‘, ‘z‘, and ‘range ‘
are ‘number ‘
new GoalXZ(x, z); // Useful for long -range
goals
that don ’t have a
specific Y level. ‘x‘ and ‘z‘ are ‘number ‘
new
GoalGetToBlock (x, y, z); // Not get into the block , but get
directly
adjacent to it. Useful for fishing , farming , filling
bucket , and beds. ‘x‘, ‘y‘, and ‘z‘ are ‘number ‘
new
GoalFollow(entity , range); // Follow the
specified
entity
within
the
specified
range. ‘entity ‘ is ‘Entity ‘, ‘range ‘ is ‘number ‘
new
GoalPlaceBlock (position , bot.world , {}); // Position
the bot in
order to place a block. ‘position ‘ is ‘Vec3 ‘
new
GoalLookAtBlock (position , bot.world , {}); // Path into a position
where a blockface of the block at position is visible. ‘position ‘
is ‘Vec3 ‘
// These are other
Mineflayer
functions
you can use:
bot.isABed(bedBlock); // Return
true if ‘bedBlock ‘ is a bed
bot.blockAt(position); // Return the block at ‘position ‘. ‘position ‘
is ‘Vec3 ‘
// These are other
Mineflayer
async
functions
you can use:
await bot.equip(item , destination); // Equip the item in the
specified
destination. ‘item ‘ is ‘Item ‘, ‘destination ‘ can only be "hand",
"head", "torso", "legs", "feet", "off -hand"
await bot.consume (); // Consume
the item in the bot ’s hand. You must
equip the item to consume
first. Useful for eating food , drinking
potions , etc.
await bot.fish (); // Let bot fish. Before
calling
this function , you
must
first get to a water
block and then
equip a fishing
rod. The
bot will
automatically
stop
fishing
when it catches a fish
await bot.sleep(bedBlock); // Sleep
until
sunrise. You must get to a
bed block
first
await bot.activateBlock(block); // This is the same as right -clicking
a block in the game. Useful for buttons , doors , using hoes , etc.
You must get to the block
first
await bot.lookAt(position); // Look at the
specified
position. You
must go near the
position
before you look at it. To fill
bucket
with water , you must
lookAt
first. ‘position ‘ is ‘Vec3 ‘
await bot.activateItem (); // This is the same as right -clicking to use
the item in the bot ’s hand. Useful for using buckets , etc. You
must
equip the item to activate
first
await bot.useOn(entity); // This is the same as right -clicking an
entity in the game. Useful for
shearing sheep , equipping
harnesses
, etc. You must get to the entity
first
{ retrieved_skills }
At each
round of conversation , I will give you
Code from the last
round: ...
Execution
error: ...
Chat log: ...
Biome: ...
Time: ...
Nearby
blocks: ...
Nearby
entities (nearest to farthest):
Health: ...
29


Hunger: ...
Position: ...
Equipment: ...
Inventory (xx /36): ...
Chests: ...
Task: ...
Context: ...
Critique: ...
You should
then
respond to me with
Explain (if applicable): Are there any steps
missing in your plan? Why
does the code not
complete
the task? What does the chat log and
execution
error
imply?
Plan: How to complete
the task step by step. You should pay
attention
to Inventory
since it tells
what you have. The task
completeness
check is also
based on your
final
inventory.
Code:
1) Write an async
function
taking the bot as the only
argument.
2) Reuse the above
useful
programs as much as possible.
- Use ‘mineBlock(bot , name , count)‘ to collect
blocks. Do not
use ‘bot.dig ‘ directly.
- Use ‘craftItem(bot , name , count)‘ to craft
items. Do not use
‘bot.craft ‘ directly.
- Use ‘smeltItem(bot , name
count)‘ to smelt
items. Do not use
‘bot.openFurnace ‘ directly.
- Use ‘placeItem(bot , name , position)‘ to place
blocks. Do not
use ‘bot.placeBlock ‘ directly.
- Use ‘killMob(bot , name , timeout)‘ to kill mobs. Do not use ‘
bot.attack ‘ directly.
3) Your
function
will be reused for
building
more
complex
functions. Therefore , you should
make it generic
and
reusable. You
should not make
strong
assumption
about the
inventory (as it may
be changed at a later
time), and
therefore
you should
always
check
whether
you have the
required
items
before
using
them. If not ,
you should
first
collect
the
required
items and reuse the above
useful
programs.
4) Functions in the "Code from the last
round" section
will not be
saved or executed. Do not reuse
functions
listed
there.
5) Anything
defined
outside a function
will be ignored , define all
your
variables
inside
your
functions.
6) Call ‘bot.chat ‘ to show the
intermediate
progress.
7) Use ‘exploreUntil(bot , direction , maxDistance , callback)‘ when
you cannot
find
something. You should
frequently
call this
before
mining
blocks or killing
mobs. You should
select a direction at
random
every
time
instead of constantly
using (1, 0, 1).
8) ‘maxDistance ‘ should
always be 32 for ‘bot.findBlocks ‘ and ‘bot
.findBlock ‘. Do not cheat.
9) Do not write
infinite
loops or recursive
functions.
10) Do not use ‘bot.on ‘ or ‘bot.once ‘ to register
event
listeners.
You
definitely do not need them.
11) Name your
function in a meaningful
way (can infer the task
from the name).
You should
only
respond in the format as described
below:
RESPONSE
FORMAT:
Explain: ...
Plan:
1) ...
2) ...
3) ...
...
Code:
‘‘‘javascript
// helper
functions (only if needed , try to avoid
them)
...
30


// main
function
after the helper
functions
async
function
yourMainFunctionName (bot) {
// ...
}
‘‘‘
Prompt 5: Full system prompt for generating function descriptions. This is used when adding a new
skill to the skill library. We give a one-shot example in the prompt.
You are a helpful
assistant
that
writes a description of the given
function
written in Mineflayer
javascript
code.
1) Do not
mention
the
function
name.
2) Do not
mention
anything
about ‘bot.chat ‘ or helper
functions.
3) There
might be some
helper
functions
before the main function , but
you only need to describe
the main
function.
4) Try to summarize
the
function in no more than 6 sentences.
5) Your
response
should be a single
line of text.
For example , if the
function is:
async
function
mineCobblestone (bot) {
// Check if the wooden
pickaxe is in the inventory , if not , craft
one
let
woodenPickaxe = bot.inventory. findInventoryItem (mcData.
itemsByName [" wooden_pickaxe "].id);
if (! woodenPickaxe) {
bot.chat (" Crafting a wooden
pickaxe .");
await
craftWoodenPickaxe (bot);
woodenPickaxe = bot.inventory. findInventoryItem (mcData.itemsByName
[" wooden_pickaxe "].id);
}
// Equip the wooden
pickaxe if it exists
if (woodenPickaxe) {
await bot.equip(woodenPickaxe , "hand ");
// Explore
until we find a stone
block
await
exploreUntil(bot , new Vec3(1, -1, 1), 60, () => {
const
stone = bot.findBlock ({
matching: mcData.blocksByName [" stone "].id ,
maxDistance: 32
});
if (stone) {
return
true;
}
});
// Mine 8 cobblestone
blocks
using the wooden
pickaxe
bot.chat (" Found a stone
block. Mining 8 cobblestone
blocks .");
await
mineBlock(bot , "stone", 8);
bot.chat (" Successfully
mined 8 cobblestone
blocks .");
// Save the event of mining 8 cobblestone
bot.save (" cobblestone_mined ");
} else {
bot.chat (" Failed to craft a wooden
pickaxe. Cannot
mine
cobblestone .");
}
}
The main
function is ‘mineCobblestone ‘.
Then you would
write:
31


The
function is about
mining 8 cobblestones
using a wooden
pickaxe.
First
check if a wooden
pickaxe is in the
inventory. If not , craft
one. If the wooden
pickaxe is available , equip the wooden
pickaxe
in the hand. Next , explore
the
environment
until
finding a stone
block. Once a stone
block is found , mine a total of 8 cobblestone
blocks
using the wooden
pickaxe.
A.4.3
Examples
Skill library example 1: craftWoodenPlanks.
async
function
craftWoodenPlanks (bot) {
const
logNames = ["oak_log", "birch_log", "spruce_log", "jungle_log"
, "acacia_log", "dark_oak_log ", "mangrove_log"];
const
plankNames = ["oak_planks", "birch_planks", " spruce_planks ", "
jungle_planks", "acacia_planks", " dark_oak_planks ", "
mangrove_planks "];
const
logInInventory = logNames.find(logName => bot.inventory.count(
mcData.itemsByName[logName ].id) > 0);
if (! logInInventory) {
bot.chat("No wooden log in inventory. Mining a wooden log ...");
await
mineWoodLog(bot);
}
const
logIndex = logNames.indexOf( logInInventory );
const
plankName = plankNames[logIndex ];
bot.chat(‘Crafting 4 ${plankName }... ‘);
await
craftItem(bot , plankName , 1);
bot.chat(‘4 ${plankName} crafted .‘);
}
Skill library example 2: mineTenCobbledDeepslateBelowY0.
async
function
mineTenCobbledDeepslateBelowY0 (bot) {
// Equip the iron
pickaxe
const
ironPickaxe = bot.inventory. findInventoryItem (mcData.
itemsByName["iron_pickaxe"].id);
await bot.equip(ironPickaxe , "hand");
// Find
cobbled_deepslate
blocks
below Y=0
const
cobbledDeepslateBlocks = await
exploreUntil(bot , new Vec3(1,
-1, 1), 60, () => {
const
cobbledDeepslate = bot.findBlock ({
matching: mcData.blocksByName [" cobbled_deepslate "].id ,
maxDistance: 32,
position: pos => pos.y < 0
});
return
cobbledDeepslate ;
});
// Mine 10 cobbled_deepslate
blocks
await
mineBlock(bot , " cobbled_deepslate ", 10);
bot.chat("10 cobbled_deepslate
mined
below Y=0.");
}
Skill library example 3: smeltFiveRawIronV2.
async
function
findSuitablePosition (bot) {
const
offsets = [new Vec3(1, 0, 0), new Vec3(-1, 0, 0), new Vec3(0,
0, 1), new Vec3(0, 0,
-1)];
for (const
offset of offsets) {
const
position = bot.entity.position.offset(offset.x, offset.y,
offset.z);
const
block = bot.blockAt(position);
32


if (block.name === "air") {
return
position;
}
}
return
null;
}
async
function
smeltFiveRawIron (bot) {
// Check if there is a furnace in the
inventory
const
furnaceItem = bot.inventory. findInventoryItem (mcData.
itemsByName.furnace.id);
// If not , craft a furnace
using the
available
cobblestone
if (! furnaceItem) {
await
craftFurnace(bot);
}
// Find a suitable
position to place the
furnace
const
furnacePosition = await
findSuitablePosition (bot);
if (! furnacePosition) {
bot.chat("Could not find a suitable
position to place the
furnace.
");
return;
}
// Place the
furnace at the
suitable
position
await
placeItem(bot , "furnace", furnacePosition );
// Smelt 5 raw iron
using the
available
coal as fuel
await
smeltItem(bot , "raw_iron", "coal", 5);
bot.chat("5 raw iron
smelted.");
}
Skill library example 4: fillBucketWithWater.
async
function
fillBucketWithWater (bot) {
// Find a water
block
nearby
const
waterBlock = await
exploreUntil (bot , new Vec3(1, 0, 1), 60, ()
=> {
const
water = bot.findBlock ({
matching: mcData.blocksByName .water.id ,
maxDistance: 32
});
return
water;
});
if (! waterBlock) {
bot.chat("Could not find
water.");
return;
}
// Go to the water
block
const
adjacentBlock= waterBlock.position.offset (0, 1, 0);
await bot.pathfinder.goto(
new
GoalGetToBlock (adjacentBlock .x, adjacentBlock .y,
adjacentBlock.z)
);
// Look at the water
block
await bot.lookAt(waterBlock.position , true);
// Equip the bucket
const
bucket = bot.inventory. findInventoryItem (mcData.itemsByName.
bucket.id);
await bot.equip(bucket , "hand");
33


// Activate
the bucket to collect
water
await bot.activateItem ();
bot.chat("Filled the bucket
with
water.");
}
Skill library example 5: catchFiveFishSafely.
async
function
catchFiveFishSafely (bot) {
// Check if the bot has a fishing
rod in its
inventory
let
fishingRod = bot.inventory. findInventoryItem (mcData.itemsByName.
fishing_rod.id);
if (! fishingRod) {
await
craftFishingRod (bot);
fishingRod = bot.inventory. findInventoryItem (mcData.itemsByName.
fishing_rod.id);
}
// Find a nearby
water
block
let
waterBlock;
while (! waterBlock) {
waterBlock = await
exploreUntil(bot , new Vec3(1, 0, 1), 60, () =>
{
const
foundWaterBlock = bot.findBlock ({
matching: mcData.blocksByName .water.id ,
maxDistance: 32
});
return
foundWaterBlock ;
});
if (! waterBlock) {
bot.chat("No path to the water
block. Trying to find
another
water
block ...");
}
}
// Move to a block
adjacent to the water
block
const
adjacentBlock = waterBlock.position.offset (0, 1, 0);
await bot.pathfinder.goto(new
GoalBlock( adjacentBlock .x,
adjacentBlock.y, adjacentBlock.z));
// Look at the water
block
await bot.lookAt(waterBlock.position);
// Equip the
fishing
rod
await bot.equip(fishingRod , "hand");
// Fish in the water 5 times
for (let i = 0; i < 5; i++) {
try {
await bot.fish ();
bot.chat(‘Fish ${i + 1} caught .‘);
} catch (error) {
if (error.message
=== "Fishing
cancelled") {
bot.chat("Fishing
was
cancelled. Trying
again ...");
i--; // Retry the same
iteration
} else {
throw
error;
}
}
}
}
34


A.5
Self-Verification
A.5.1
Components in the Prompt
The input prompt to GPT-4 consists of the following components:
(1) The agent’s state: We exclude other blocks that are recently seen and nearby entities from the
agent’s state since they are not useful for assessing the task’s completeness. See Sec. A.3.1
for each element of the agent’s state;
(2) Task proposed by the automatic curriculum;
(3) Task context: We prompt GPT-3.5 to ask for general suggestions about how to solve the
task. In practice, this part is handled by the automatic curriculum since it has a systematic
mechanism for question-answering (Sec. A.3.2);
(4) Chain-of-thought prompting [46] in response: We request GPT-4 to initially reason about
the task’s success or failure, then output a boolean variable indicating the task’s outcome,
and finally provide a critique to the agent if the task fails.
(5) Few-shot examples for in-context learning [36–38].
A.5.2
Full Prompt
Prompt 6: Full system prompt for self-verification.
You are an assistant
that
assesses my progress of playing
Minecraft
and
provides
useful
guidance.
You are
required to evaluate if I have met the task
requirements .
Exceeding
the task
requirements is also
considered a success
while
failing to meet them
requires
you to provide
critique to help me
improve.
I will give you the
following
information:
Biome: The biome
after the task
execution.
Time: The
current
time.
Nearby
blocks: The
surrounding
blocks. These
blocks are not
collected
yet. However , this is useful for some
placing or planting
tasks.
Health: My current
health.
Hunger: My current
hunger
level. For eating task , if my hunger
level
is 20.0, then I successfully
ate the food.
Position: My current
position.
Equipment: My final
equipment. For
crafting tasks , I sometimes
equip
the
crafted
item.
Inventory (xx /36): My final
inventory. For mining and
smelting tasks ,
you only need to check
inventory.
Chests: If the task
requires me to place
items in a chest , you can
find
chest
information
here.
Task: The
objective I need to accomplish.
Context: The
context of the task.
You should
only
respond in JSON
format as described
below:
{
"reasoning ": "reasoning",
"success ": boolean ,
"critique ": "critique",
}
Ensure the
response
can be parsed by Python ‘json.loads ‘, e.g.: no
trailing
commas , no single quotes , etc.
Here are some
examples:
INPUT:
Inventory
(2/36): {’oak_log ’:2, ’spruce_log ’:2}
35


Task: Mine 3 wood logs
RESPONSE:
{
"reasoning ": "You need to mine 3 wood logs. You have 2 oak logs
and 2 spruce logs , which add up to 4 wood logs.",
"success ": true ,
"critique ": ""
}
INPUT:
Inventory
(3/36): {’crafting_table ’: 1, ’spruce_planks ’: 6, ’stick ’:
4}
Task: Craft a wooden
pickaxe
RESPONSE:
{
"reasoning ": "You have
enough
materials to craft a wooden
pickaxe ,
but you didn ’t craft it.",
"success ": false ,
"critique ": "Craft a wooden
pickaxe
with a crafting
table
using 3
spruce
planks and 2 sticks ."
}
INPUT:
Inventory
(2/36): {’raw_iron ’: 5, ’stone_pickaxe ’: 1}
Task: Mine 5 iron_ore
RESPONSE:
{
"reasoning ": "Mining
iron_ore in Minecraft
will get
raw_iron. You
have 5 raw_iron in your
inventory .",
"success ": true ,
"critique ": ""
}
INPUT:
Biome: plains
Nearby
blocks: stone , dirt , grass_block , grass , farmland , wheat
Inventory
(26/36): ...
Task:
Plant 1 wheat
seed.
RESPONSE:
{
"reasoning ": "For
planting tasks , inventory
information is useless
. In nearby blocks , there is farmland
and wheat , which
means you
succeed to plant the wheat
seed.",
"success ": true ,
"critique ": ""
}
INPUT:
Inventory
(11/36): {... ,’rotten_flesh ’: 1}
Task: Kill 1 zombie
Context: ...
RESPONSE
{
36


"reasoning ": "You have
rotten
flesh in your
inventory , which
means
you
successfully
killed one zombie .",
"success ": true ,
"critique ": ""
}
INPUT:
Hunger: 20.0/20.0
Inventory
(11/36): ...
Task: Eat 1 ...
Context: ...
RESPONSE
{
"reasoning ": "For all eating task , if the player ’s hunger is 20.0 ,
then the player
successfully
ate the food.",
"success ": true ,
"critique ": ""
}
INPUT:
Nearby
blocks: chest
Inventory
(28/36): {’rail ’: 1, ’coal ’: 2, ’oak_planks ’: 13, ’
copper_block ’: 1, ’diorite ’: 7, ’cooked_beef ’: 4, ’granite ’: 22, ’
cobbled_deepslate ’: 23, ’feather ’: 4, ’leather ’: 2, ’
cooked_chicken ’: 3, ’white_wool ’: 2, ’stick ’: 3, ’black_wool ’: 1,
’stone_sword ’: 2, ’stone_hoe ’: 1, ’stone_axe ’: 2, ’stone_shovel ’:
2, ’cooked_mutton ’: 4, ’cobblestone_wall ’: 18, ’crafting_table ’:
1, ’furnace ’: 1, ’iron_pickaxe ’: 1, ’stone_pickaxe ’: 1, ’
raw_copper ’: 12}
Chests:
(81, 131, 16): {’andesite ’: 2, ’dirt ’: 2, ’cobblestone ’: 75, ’
wooden_pickaxe ’: 1, ’wooden_sword ’: 1}
Task: Deposit
useless
items
into the chest at (81, 131, 16)
Context: ...
RESPONSE
{
"reasoning ": "You have 28 items in your
inventory
after
depositing
, which is more than 20. You need to deposit
more
items
from your
inventory to the chest.",
"success ": false ,
"critique ": "Deposit
more
useless
items
such as copper_block ,
diorite , granite , cobbled_deepslate , feather , and
leather to meet
the
requirement of having
only 20 occupied
slots in your
inventory
."
}
A.6
System-level Comparison between VOYAGER and Prior Works
We make a system-level comparison in Table. A.2. Voyager stands out as the only method featuring a
combination of automatic curriculum, iterative planning, and a skill library. Moreover, it learns to
play Minecraft without the need for any gradient update.
37


Table A.2: System-level comparison between VOYAGER and prior works.
VPT [8]
DreamerV3 [69] DECKARD [53] DEPS [55]
Plan4MC [71]
VOYAGER
Demos
Videos
None
Videos
None
None
None
Rewards
Sparse
Dense
Sparse
None
Dense
None
Observations
Pixels Only
Pixels &
Meta
Pixels &
Inventory
Feedback &
Inventory
Pixels &
Meta
Feedback &
Meta &
Inventory
Actions
Keyboard
&
Mouse
Discrete
Keyboard
&
Mouse
Keyboard
&
Mouse
Discrete
Code
Automatic
Curriculum
✓
✓
(in-context
GPT-4
pro-
posal)
Iterative Plan-
ning
✓
✓
(3
types
of
feedback)
Skill Library
✓
(pre-defined)
✓
(self-
generated)
Gradient-Free
✓
B
Experiments
B.1
Experimental Setup
Our simulation environment is built upon MineDojo [23] and utilizes Mineflayer [52] JavaScript APIs
for motor controls (Sec. A.4.2). Additionally, we incorporate many bot.chat() into Mineflayer
functions to provide abundant environment feedback and implement various condition checks along
with try-catch exceptions for continuous execution. If the bot dies, it is resurrected near the closest
ground, and its inventory is preserved for uninterrupted exploration. The bot recycles its crafting table
and furnace after program execution. For detailed implementations, please refer to our codebase.
B.2
Baselines
ReAct [29] uses chain-of-thought prompting [46] by generating both reasoning traces and action
plans with LLMs. We provide it with our environment feedback and the agent states as observations.
ReAct undergoes one round of code generation from scratch, followed by three rounds of code
refinement. This process is then repeated until the maximum prompting iteration is reached.
Reflexion [30] is built on top of ReAct [29] with self-reflection to infer more intuitive future actions.
We provide it with environment feedback, the agent states, execution errors, and our self-verification
module. Similar to ReAct, Reflexion undergoes one round of code generation from scratch, followed
by three rounds of code refinement. This process is then repeated until the maximum prompting
iteration is reached.
AutoGPT [28] is a popular software tool that automates NLP tasks by decomposing a high-level goal
into multiple subgoals and executing them in a ReAct-style loop. We re-implement AutoGPT by
using GPT-4 to do task decomposition and provide it with the agent states, environment feedback,
and execution errors as observations for subgoal execution. Compared with VOYAGER, AutoGPT
lacks the skill library for accumulating knowledge, self-verification for assessing task success, and
automatic curriculum for open-ended exploration. During each subgoal execution, if no execution
error occurs, we consider the subgoal completed and proceed to the next one. Otherwise, we refine
the program until three rounds of code refinement (equivalent to four rounds of code generation)
are completed, and then move on to the next subgoal. If three consecutive subgoals do not result in
acquiring a new item, we replan by rerunning the task decomposition.
The task is “explore the world and get as many items as possible” for all baselines.
38


Table A.3: Comparison between VOYAGER and baselines.
ReAct [29]
Reflexion [30]
AutoGPT [28]
VOYAGER
Chain-of-Thought [46]
✓
✓
✓
✓
Self Verification
✓
✓
Environment Feedback
✓
✓
✓
✓
Execution Errors
✓
✓
✓
Agent State
✓
✓
✓
✓
Skill Library
✓
Automatic Curriculum
✓
Figure A.1: Minecraft item icons with corresponding names.
B.3
Ablations
We ablate 6 design choices (automatic curriculum, skill library, environment feedback, execution
errors, self-verification, and GPT-4 for code generation) in VOYAGER and study their impact on
exploration performance.
• Manual Curriculum: We substitute the automatic curriculum with a manually designed
curriculum for mining a diamond: “Mine 3 wood log”, “Craft 1 crafting table”, “Craft
1 wooden pickaxe”, “Mine 11 cobblestone”, “Craft 1 stone pickaxe”, “Craft 1 furnace”,
“Mine 3 iron ore”, “Smelt 3 iron ore”, “Craft 1 iron pickaxe”, “Mine 1 diamond”. A manual
curriculum requires human effort to design and is not scalable for open-ended exploration.
• Random Curriculum: We curate 101 items obtained by VOYAGER and create a random
curriculum by randomly selecting one item as the next task.
• w/o Skill Library: We remove the skill library, eliminating skill retrieval for code generation.
• w/o Environment Feedback: We exclude environment feedback (chat log) from the prompt
for code generation.
• w/o Execution Errors: We exclude execution errors from the prompt for code generation.
• w/o Self-Verification: For each task, we generate code without self-verification and it-
eratively refine the program for 3 rounds (equivalent to 4 rounds of code generation in
total).
• GPT-3.5: We replace GPT-4 with GPT-3.5 for code generation. We retain GPT-4 for the
automatic curriculum and the self-verification module.
B.4
Evaluation Results
B.4.1
Significantly Better Exploration
The meaning of each icon in Fig. 1 is shown in Fig. A.1.
We run three trials for each method. The items collected by VOYAGER in each trial is
39


• Trial 1:
‘iron_ingot’, ‘stone_shovel’, ‘iron_leggings’, ‘fishing_rod’, ‘pufferfish’,
‘oak_log’, ‘cooked_mutton’, ‘green_dye’, ‘flint’, ‘chest’, ‘iron_sword’, ‘string’, ‘en-
der_pearl’, ‘raw_copper’, ‘crafting_table’, ‘cactus’, ‘lapis_lazuli’, ‘iron_pickaxe’, ‘cop-
per_ingot’, ‘stone_pickaxe’, ‘wooden_hoe’, ‘scaffolding’, ‘stick’, ‘porkchop’, ‘cop-
per_block’, ‘gravel’, ‘grass_block’, ‘white_bed’, ‘bone’, ‘dirt’, ‘mutton’, ‘white_wool’,
‘oak_sapling’, ‘coal’, ‘bamboo’, ‘wooden_pickaxe’, ‘rotten_flesh’, ‘cooked_porkchop’,
‘cod’, ‘iron_boots’, ‘lightning_rod’, ‘diorite’, ‘water_bucket’, ‘shears’, ‘furnace’, ‘andesite’,
‘granite’, ‘bucket’, ‘wooden_sword’, ‘sandstone’, ‘iron_helmet’, ‘raw_iron’, ‘sand’, ‘aca-
cia_log’, ‘cooked_cod’, ‘oak_planks’, ‘azure_bluet’, ‘iron_shovel’, ‘acacia_planks’, ‘shield’,
‘iron_axe’, ‘iron_chestplate’, ‘cobblestone’;
• Trial 2: ‘iron_ingot’, ‘tuff’, ‘stone_shovel’, ‘iron_leggings’, ‘fishing_rod’, ‘cooked_mutton’,
‘spruce_planks’, ‘gunpowder’, ‘amethyst_shard’, ‘chest’, ‘string’, ‘cooked_salmon’,
‘iron_sword’, ‘raw_copper’, ‘crafting_table’, ‘torch’, ‘lapis_lazuli’, ‘iron_pickaxe’, ‘cop-
per_ingot’, ‘stone_pickaxe’, ‘wooden_hoe’, ‘stick’, ‘amethyst_block’, ‘salmon’, ‘cal-
cite’, ‘gravel’, ‘white_bed’, ‘bone’, ‘dirt’, ‘mutton’, ‘white_wool’, ‘spyglass’, ‘coal’,
‘wooden_pickaxe’, ‘cod’, ‘iron_boots’, ‘lily_pad’, ‘cobbled_deepslate’, ‘lightning_rod’,
‘snowball’, ‘stone_axe’, ‘smooth_basalt’, ‘diorite’, ‘water_bucket’, ‘furnace’, ‘andesite’,
‘bucket’, ‘granite’, ‘shield’, ‘iron_helmet’, ‘raw_iron’, ‘cobblestone’, ‘spruce_log’,
‘cooked_cod’, ‘tripwire_hook’, ‘stone_hoe’, ‘iron_chestplate’, ‘stone_sword’;
• Trial 3:
‘spruce_planks’, ‘dirt’, ‘shield’, ‘redstone’, ‘clock’, ‘diamond_sword’,
‘iron_chestplate’, ‘stone_pickaxe’, ‘leather’, ‘string’, ‘chicken’, ‘chest’, ‘diorite’,
‘iron_leggings’, ‘black_wool’, ‘cobblestone_wall’, ‘cobblestone’, ‘cooked_chicken’,
‘feather’, ‘stone_sword’, ‘raw_gold’, ‘gravel’, ‘birch_planks’, ‘coal’, ‘cobbled_deepslate’,
‘oak_planks’, ‘iron_pickaxe’, ‘granite’, ‘tuff’, ‘crafting_table’, ‘iron_helmet’, ‘stone_hoe’,
‘iron_ingot’, ‘stone_axe’, ‘birch_boat’, ‘stick’, ‘sand’, ‘bone’, ‘raw_iron’, ‘beef’, ‘rail’,
‘oak_sapling’, ‘kelp’, ‘gold_ingot’, ‘birch_log’, ‘wheat_seeds’, ‘cooked_mutton’, ‘furnace’,
‘arrow’, ‘stone_shovel’, ‘white_wool’, ‘andesite’, ‘jungle_slab’, ‘mutton’, ‘iron_sword’,
‘copper_ingot’, ‘diamond’, ‘torch’, ‘oak_log’, ‘cooked_beef’, ‘copper_block’, ‘flint’,
‘bone_meal’, ‘raw_copper’, ‘wooden_pickaxe’, ‘iron_boots’, ‘wooden_sword’.
The items collected by ReAct [29] in each trial is
• Trial 1: ‘bamboo’, ‘dirt’, ‘sand’, ‘wheat_seeds’;
• Trial 2: ‘dirt’, ‘rabbit’, ‘spruce_log’, ‘spruce_sapling’;
• Trial 3: ‘dirt’, ‘pointed_dripstone’;
The items collected by Reflexion [30] in each trial is
• Trial 1: ‘crafting_table’, ‘orange_tulip’, ‘oak_planks’, ‘oak_log’, ‘dirt’;
• Trial 2: ‘spruce_log’, ‘dirt’, ‘clay_ball’, ‘sand’, ‘gravel’;
• Trial 3: ‘wheat_seeds’, ‘oak_log’, ‘dirt’, ‘birch_log’, ‘sand’.
The items collected by AutoGPT [28] in each trial is
• Trial 1: ‘feather’, ‘oak_log’, ‘leather’, ‘stick’, ‘porkchop’, ‘chicken’, ‘crafting_table’,
‘wheat_seeds’, ‘oak_planks’, ‘dirt’, ‘mutton’;
• Trial 2:
‘wooden_pickaxe’, ‘iron_ingot’, ‘stone’, ‘coal’, ‘spruce_planks’, ‘string’,
‘raw_copper’, ‘crafting_table’, ‘diorite’, ‘andesite’, ‘furnace’, ‘torch’, ‘spruce_sapling’,
‘granite’, ‘iron_pickaxe’, ‘stone_pickaxe’, ‘wooden_axe’, ‘raw_iron’, ‘stick’, ‘spruce_log’,
‘dirt’, ‘cobblestone’;
• Trial 3: ‘wooden_shovel’, ‘wooden_pickaxe’, ‘iron_ingot’, ‘stone’, ‘cod’, ‘coal’, ‘oak_log’,
‘flint’, ‘raw_copper’, ‘crafting_table’, ‘diorite’, ‘furnace’, ‘andesite’, ‘torch’, ‘granite’,
‘lapis_lazuli’, ‘iron_pickaxe’, ‘stone_pickaxe’, ‘raw_iron’, ‘stick’, ‘gravel’, ‘oak_planks’,
‘dirt’, ‘iron_axe’, ‘cobblestone’.
40


Figure A.2: Map coverage: Two bird’s eye views of Minecraft maps. VOYAGER is able to traverse
2.3× longer distances compared to baselines while crossing diverse terrains. Trajectories are plotted
based on the positions where each agent interacts with GPT-4.
B.4.2
Extensive Map Traversal
Agent trajectories for map coverage are displayed in Fig. A.2. Fig. 7 is plotted based on Fig. A.2 by
drawing the smallest circle enclosing each trajectory. The terrains traversed by VOYAGER in each
trial is
• Trial 1: ‘meadow’, ‘desert’, ‘river’, ‘savanna’, ‘forest’, ‘plains’, ‘bamboo_jungle’, ‘drip-
stone_caves’;
• Trial 2: ‘snowy_plains’, ‘frozen_river’, ‘dripstone_caves’, ‘snowy_taiga’, ‘beach’;
• Trial 3:
‘flower_forest’,
‘meadow’,
‘old_growth_birch_forest’,
‘snowy_slopes’,
‘frozen_peaks’, ‘forest’, ‘river’, ‘beach’, ‘ocean’, ‘sunflower_plains’, ‘plains’, ‘stony_shore’.
The terrains traversed by ReAct [29] in each trial is
• Trial 1: ‘plains’, ‘desert’, ‘jungle’;
• Trial 2: ‘snowy_plains’, ‘snowy_taiga’, ‘snowy_slopes’;
• Trial 3: ‘dark_forest’, ‘dripstone_caves’, ‘grove’, ‘jagged_peaks’.
The terrains traversed by Reflexion [30] in each trial is
• Trial 1: ‘plains’, ‘flower_forest’;
• Trial 2: ‘snowy_taiga’;
• Trial 3: ‘old_growth_birch_forest’, ‘river’, ‘ocean’, ‘beach’, ‘plains’.
The terrains traversed by AutoGPT [28] in each trial is
• Trial 1: ‘plains’, ‘dripstone_caves’, ‘savanna’, ‘meadow’;
• Trial 2: ‘snowy_taiga’;
• Trial 3: ‘plains’, ‘stony_shore’, ‘forest’, ‘ocean’.
B.4.3
Efficient Zero-Shot Generalization to Unseen Tasks
The results of zero-shot generalization to unseen tasks for the other two tasks are presented in Fig. A.3.
Similar to Fig. 8, VOYAGER consistently solves all tasks, while the baselines are unable to solve any
41


Figure A.3: Zero-shot generalization to unseen tasks. We visualize the intermediate progress of each
method on the other two tasks. We do not plot ReAct and Reflexion since they do not make any
meaningful progress.
task within 50 prompting iterations. Our skill library, constructed from lifelong learning, not only
enhances VOYAGER’s performance but also provides a boost to AutoGPT [28].
B.4.4
Accurate Skill Retrieval
We conduct an evaluation of our skill retrieval (309 samples in total) and the results are in Table. A.4.
The top-5 accuracy standing at 96.5% suggests our retrieval process is reliable (note that we include
the top-5 relevant skills in the prompt for synthesizing a new skill).
Table A.4: Skill retrieval accuracy.
Top-1 Acc
Top-2 Acc
Top-3 Acc
Top-4 Acc
Top-5 Acc
80.2 ± 3.0
89.3 ± 1.8
93.2 ± 0.7
95.2 ± 1.8
96.5 ± 0.3
B.4.5
Robust to Model Variations
In the main paper, all of Voyager’s experiments are conducted with gpt-4-0314. We additionally
run new experiments with gpt-4-0613 and find that the performance is roughly the same (Fig. A.4).
It demonstrates that Voyager is robust to model variations.
Figure A.4: VOYAGER’s performance with GPT-4-0314 and GPT-4-0613.
42