# LongBench v2 / 66ebc4af5a08c7b9b35dede0

task_id: 4b6915f4-4f34-5707-8051-baeea918473b
task_key: train--66ebc4af5a08c7b9b35dede0
task_revision_id: 2

{"choice_A":"In the Web Shopping subsets, DigiRL increased by 3.6% compared to Filtered BC, while in the General subsets it was about 10%.","choice_B":"The all possible actions for the agent in the DigiRL framework include tapping and swiping on the screen using normalized (x, y) coordinates and typing variable-length text inputs.","choice_C":"The automatic curriculum in DigiRL adjusts the instruction-level value function to filter out easy tasks, allowing the agent to focus solely on tasks it has not yet encountered during training.","choice_D":"The cross-entropy loss function is applied in DigiRL exclusively to the policy network, avoiding its use in the training of value functions to prevent overfitting in the model.","context":"DigiRL: Training In-The-Wild Device-Control\nAgents with Autonomous Reinforcement Learning\nAbstract\nTraining corpuses for vision language models (VLMs) typically lack sufficient\namounts of decision-centric data. This renders off-the-shelf VLMs sub-optimal\nfor decision-making tasks such as in-the-wild device control through graphical\nuser interfaces (GUIs). While training with static demonstrations has shown\nsome promise, we show that such methods fall short for controlling real GUIs\ndue to their failure to deal with real world stochasticity and non-stationarity not\ncaptured in static observational data. This paper introduces a novel autonomous\nRL approach, called DigiRL, for training in-the-wild device control agents through\nfine-tuning a pre-trained VLM in two stages: offline RL to initialize the model,\nfollowed by offline-to-online RL. To do this, we build a scalable and parallelizable\nAndroid learning environment equipped with a VLM-based evaluator and develop\na simple yet effective RL approach for learning in this domain. Our approach\nruns advantage-weighted RL with advantage estimators enhanced to account for\nstochasticity along with an automatic curriculum for deriving maximal learning\nsignal. We demonstrate the effectiveness of DigiRL using the Android-in-the-Wild\n(AitW) dataset, where our 1.3B VLM trained with RL achieves a 49.5% absolute\nimprovement – from 17.7 to 67.2% success rate – over supervised fine-tuning with\nstatic human demonstration data. These results significantly surpass not only the\nprior best agents, including AppAgent with GPT-4V (8.3% success rate) and the\n17B CogAgent trained with AitW data (38.5%), but also the prior best autonomous\nRL approach based on filtered behavior cloning (57.8%), thereby establishing a\nnew state-of-the-art for digital agents for in-the-wild device control.\n1\nIntroduction\nAdvances in vision-language models (VLMs), especially in regards to their remarkable common-\nsense, reasoning, and generalization abilities imply that realizing a fully autonomous digital AI\nassistant, that can simplify human life by automating day-to-day activities on computer devices via\nnatural language interfaces, is no longer a distant aspiration [16, 45, 56]. An effective device-control\nAI assistant should be able to complete tasks in-the-wild through Graphical User Interfaces (GUIs)\non digital devices: make travel plans; experiment with presentation designs; and operate a mobile\ndevice autonomously, all while running amidst stochasticity and distractors on the device, the Internet,\nand the tools it interacts with. However, enhanced reasoning or common-sense abilities do not\ndirectly transfer to intelligent assistant behavior: ultimately we want AI assistants to accomplish\n∗Equal contribution, listed in alphabetical order; work done at UC Berkeley. E-mails: haob2@illinois.edu,\nyifei_zhou@berkeley.edu, aviralkumar@google.com. Project page: https://digirl-agent.github.io/.\nCode available at https://github.com/DigiRL-agent/digirl.\nPreprint. Under review.\narXiv:2406.11896v1  [cs.LG]  14 Jun 2024\n\n\nAutoEval annotates \nreward for each \ntrajectory\nModel executes tasks \nin parallel and \nproduce trajectories\nTasks are sampled \nfrom task dataset\nAnnotated trajectories \nare used to update the \nmodel through online \nRL\nFine-tune on existing trajectories via offline RL\nStep I: Offline RL\nPretrained Model\nOffline Model\nVLM is generally pre-trained on Internet-scale \nvision-and-language data\nPretraining\nStep II: Online RL\nPretrained Model\nOnline \nModel\nAutoEval\nFigure 1: DigiRL overview. DigiRL is built upon a VLM that has been pre-trained on extensive web data\nto develop fundamental skills such as common knowledge, reasoning, and visual grounding. Initially, we\nemploy offline RL to fine-tune the VLM using stale task-specific data, which helps in eliciting goal-oriented\nbehaviors. Subsequently, our agent engages with real-world graphical user interfaces, continuously enhancing\nits performance through online RL and autonomous performance evaluations.\ntasks, exhibit rational behavior, and recover from their mistakes as opposed to simply producing a\nplausible completion to a given observation based on the data seen during pre-training. This implies\nthat a mechanism to channel abilities from pre-training into a deployable AI “agent” is lacking.\nEven the strongest proprietary VLMs, such as GPT-4V [24] and Gemini 1.5 Pro [7] 2, still struggle to\nproduce the right actions when completing tasks on devices. While general-purpose vision-language\nabilities help these models still make meaningful abstract deductions about novel scenes when\ndeployed, these deductions do not transfer to accurate reasoning for control [47, 45, 55, 44]. As a\nresult, most prior work for building device agents construct complex wrappers around proprietary\nVLMs by combining them with prompting, search, or tool use [47, 44, 52, 51, 45]. While building\nprompting or retrieval wrappers to improve decision-making performance of existing VLMs enhances\ntheir performance in the short run, without updating the weights, the effectiveness of the resulting\nagent is inherently limited by the capabilities of the base model [49, 3]. For example, we found that\noff-the-shelf VLMs make reasoning failures that derail the agent (e.g., Figure 2 and Figure 15), as\ndirect consequences of inability of the base model to reason with low-level device-control actions.\nA different solution is to fine-tune the model on demonstrations via imitation learning. However,\nthe dynamic nature of the web and device means that models trained to mimic actions in stale data\ncan result in sub-optimalilty as the eco-system changes [26]. Agents trained in this way struggle to\nrecover from the agents’ own mistakes [8, 12].\nIf we can instead build an interactive approach to train a VLM to directly adapt and learn from its\nown experience on the device and the Internet, that can be used to build a robust and reliable device-\ncontrol agent, without needing wrappers on top of proprietary models. However, this learning-based\napproach must satisfy some desiderata. First, it must make use of online interaction data since static\ndemonstration data would not be representative of the task when the model is deployed: for instance,\neven in the setting of web navigation alone, dynamic nature of in-the-wild websites means that the\nagent will frequently encounter website versions that differ significantly from the scenarios seen\nduring training and will need to behave reliably despite changes in visual appearance and distractions.\nSecond, learning on-the-fly means the approach must learn from multi-turn interaction data from\nthe model itself, a large of chunk of which would consist of failures. Proper mechanisms must be\ndesigned to automatically pick out the correct actions while filtering the wrong ones.\nTo this end, our main contribution is a novel autonomous RL approach, DigiRL (i.e., RL for\nDigital Agents), for training device control agents, as shown in Figure 1. The resulting agent attains\n2We use external versions of these models as of June 11, 2024. Experiments with GPT and Gemini models\nwere performed entirely by Hao Bai, Yifei Zhou, Mert Cemri, and Jiayi Pan.\n2\n\n\nDigiRL\nAutoUI\nGPT-4V\nGot \nstuck\n✘\nGot \nstuck\n✘\n✘\n✘\nGot \nstuck\n✘\nGeneral\n      How much \ndoes a 2 \nbedroom \napartment rent \nfor in Denver?\nWebShop\n      Go to \nbestbuy.com, \nsearch for \n“logitech \ng933”\nClick\nSkipped...\nClick\nClick\nType “razecg\nPress Back\nClick\nType “logi|g\nScroll Up\nPress Home\nClick\nType “2 bedr’g\nPress Enter\nWrong\n  page\nGot \nstuck\nGot \nstuck\n✘\nFigure 2: Qualitative comparison between DigiRL and other approaches. AutoUI trained from static\nhuman demonstrations can easily get stuck in out-of-distribution states while GPT-4V often get on a wrong goal\n(searched “logitech g933bestbuy.com logitech g933” in Google instead of bestbuy.com). In contrast, DigiRL can\nrecover from such states and complete complex instruction as requested.\nstate-of-the-art performance on a number of Android device-control tasks. To train this agent, our\napproach operates in two phases: an initial offline RL phase to initialize the agent using existing data,\nfollowed by an offline-to-online RL phase, that further fine-tunes the model obtained from offline\nRL on online rollout data. Online RL training requires access to an environment that the agent can\ninteract with and obtain reliable reward signals, all in a reasonable amount of wall-clock time. To\ndo so, we build a scalable and parallelizable Android learning environment equipped with a robust\nVLM-based general-purpose evaluator [26] (average error rate 2.8% against human judgement) that\nsupports running up to 64 real Android emulators at the same time to make online RL real-time.\nThen, to effectively learn autonomously, we develop an online RL approach that retains the simplicity\nof supervised learning, but incorporates several key deep RL insights to enable fast fine-tuning.\nConcretely, our approach is a variant of advantage-weighted regression (AWR) [28], equipped with:\n(i) an automatic curriculum that uses an instruction-level value function to order tasks so as to extract\nmaximal learning signal, which is inspired by prioritized replay methods [11, 32, 23], and (ii) another\nstep-level value function trained via effective cross-entropy loss [17, 5] to extract low-variance and\nless-biased learning signal amidst stochasticity and diverse tasks. This RL approach allows us to\nfine-tune VLMs on their own experience.\nWe evaluate our agent trained with DigiRL in carrying out diverse instructions from Android in the\nWild dataset [31] on real Android device emulators and find that our agent can achieve a 28.7%\nimprovement over the existing state-of-the-art agents (from 38.5% to 67.2% success rate) 18B\nCogAgent [9], and over 9% improvement over the prior best autonomous learning approach based\non Filtered Behavior Cloning [18, 26]. The performance of our agent also significantly surpasses\nwrappers on top of state-of-the-art proprietary VLMs such as GPT-4V [24] and Gemini 1.5 Pro [7]\n(17.7% success rate), despite using a significantly smaller model (with 1.3B parameters). To our\nknowledge, this is the first work to successfully build an autonomous offline-to-online RL approach\nto enable state-of-the-art performance on device-control problems.\n2\nRelated Work\nMulti-modal digital agents. In contrast to language-only agents that largely interact with both\ntext or code inputs and outputs [33, 49, 3, 30, 46, 20, 13], training multi-modal agents capable of\ncontrolling devices presents different challenges: first, device control is done directly at the pixel-\nlevel and in a coordinate-based action space, instead of natural language [31, 44] that LLM is most\nfamiliar with, and second, the ecosystem of a device and the Internet tends to be quite stochastic and\nunpredictable, which is absent with high-level planning in language only. To handle these challenges,\nprior work largely builds on strong proprietary VLMs [24, 7], and designs complex rule-based\nwrappers [47, 51, 45, 52] to enhance the visual grounding capabilities of VLMs in GUI interfaces\nand convert text output into pixel interactions. However, without any form of fine-tuning, this limits\nthe room for possible performance improvement [44, 47, 49, 3, 50], especially when pre-training\n3\n\n\ncorpora only present limited action-labeled data. A separate line of work fine-tunes VLMs with\ndemonstration data [19, 15, 9, 53] via imitation learning, but maximizing single-step accuracy from\nstale demonstrations without accounting for consequences of these actions in subsequent steps may\nlead to poor solutions amidst stochasticity [26], as agents trained in such ways will struggle to recover\nfrom out-of-distribution states not included in the demonstration data [8, 12]. The third category, and\nperhaps the closest to us, are works that run filtered imitation learning on autonomously-collected\ndata to directly maximize the episode success rate [26, 18]. In contrast, ours is the first work to scale\nautonomous, offline-to-online RL for device control, producing an agent that outperforms prior agents\nbuilt via imitation. Even when compared to prior work running on-policy RL in simplified web\nnavigation settings (MiniWob++ [37, 10]), our approach is 1000x more sample efficient (around 1e3\ntrajectories compared to around 1e6 trajectories), and operates in real-world GUI navigation tasks.\nEnvironments for device control agents. Recent works have introduced simulated environments\nfor building device control agents [48, 56, 16, 54, 4, 44]. However, these environments are primarily\ndesigned for evaluation, and present only a limited range of tasks within fully deterministic and\nstationary settings, infeasible for acquiring a diverse repertoire of skills needed for device control.\nAlternatively, other works use environments with a greater diversity of tasks [48, 37], but these\nenvironments often oversimplify the task complexity, thus failing to transfer to in-the-wild settings.\nCoversely, our training environment utilizes autonomous evaluation [26] with Gemini 1.5 Pro [7]\nto support diverse, open-ended tasks on parallel actual Android devices, at full scale unlike prior\nenvironments. This also contrasts other prior works that use single-threaded Android emulators [26,\n39, 19] and thus inefficient for support online RL at scale.\nReinforcement learning for LLM/VLMs. The majority of prior research employing RL for\nfoundation models concentrates on tasks that must be solved in a single turn, such as preference\noptimization [25, 58, 2] or reasoning [27]. However, optimizing for single-turn interaction from expert\ndemonstrations may result in sub-optimal strategies for multi-step problems [57, 38, 42], especially\namidst a high degree of stochasticity or non-stationarity. Therefore, we focus on building multi-turn\nRL algorithms that can learn from sub-optimal, online interaction data in this work. While prior\nworks have developed value-based RL algorithms for LLMs [42, 38, 1, 57, 50], they typically require\nmaintaining multiple models such as Q-networks, value-networks, and policy networks, along with\ntheir delayed target counterparts, and can be subjective to slow convergence and sensitivity to choices\nof hyper-parameters. In contrast, we focus on identifying the key design choices for instantiating a\nsimple yet effective RL algorithm for practitioners to incorporate to substantially improve full-scale\nAndroid device control. Our approach can serve as a base model for future research.\n3\nProblem Setup and Preliminaries\nProblem formulation. We are interested in pixel-based interaction with virtual devices. We scope\nour study in the control of Android devices: this is already significantly more challenging and more\ngeneral than previous learning-based environments that focus solely on web navigation [16, 56, 4],\nwhere the web browser itself is merely one application within our broader environment, and link-based\ndevice controls [47, 51] are inadequate for tasks like games that do not support link inputs.\nEach episode begins with the emulator initialized to the home screen. Subsequently, a task is selected\nfrom a predefined set of language instructions, some examples of which are shown in Appendix A.1.\nAn agent is then tasked with manipulating the emulator to fulfill this instruction. At each time step,\nthe agent receives a screenshot of the current screen as the observation. Following the action space\nin prior literature [31], the available actions include tapping and sliding based on normalized (x, y)\ncoordinates (ranging from 0 to 1 relative to the screen dimensions), typing text strings of variable\nlength, and pressing special buttons such as HOME, BACK, and ENTER, as illustrated in Figure 3.\nOur train and test instructions comes from General and Web Shopping subsets in AitW [31]. These\ntasks consist of information-gathering tasks like “What’s on the menu of In-n-Out?”, and shopping\ntasks on the web like “Go to newegg.com, search for razer kraken, and select the first entry”.\nChallenges of stochasticity. Real-world device contrl presents unique challenges of stochasticity ab-\nsent in simulated environments [56, 37] such as: (1) the non-stationarity of websites and applications,\nwhich undergo frequent updates, causing the online observations to be different from stale offline data,\n(2) various unpredictable distractors such as pop-up advertisements, login requests, and the stochastic\norder of search results. (3) technical challenges and glitches such as incomplete webpage loading or\ntemporary access restrictions to certain sites. Examples of scenarios with such stochasticity from\nour experiments are shown in Figure 3. We observe that these stochastic elements pose significant\n4\n\n\naction space\ntype\nclick\nslide\nhome\nback\nenter\nreal-world \nenvironment\nagent\nmodel\nopen-ended \nevaluator\nnon-stationary website\nload\nads\nunpredictable order\npop-up\nidentity\ndynamics\nFigure 3: Environment details. Top: actions space and dynamics of the environment. Bottom: examples of the\nread-world non-stationarity and dynamism of the environment.\nchallenges for pre-trained VLMs, including even those fine-tuned on device control data. As a\nconcrete example, Figure 4 shows an experiment result that illustrates the necessity of continuously\nadapting the models to the non-stationarity of websites and applications. After obtaining a good\ncheckpoint using our approach (DigiRL), that we will introduce in the next section, with autonomous\ndata from June.1 to June.3, we compare the performance of a frozen policy and a continuously\nupdating policy using fresh autonomous data from June.7 to June.11. We find that indeed the the\nperformance of the frozen policy gradually degrades over time due to the changes on websites and\napplications, while continuous online updates plays a key role in preventing this degradation.\nJune 1\nJune 3\nJune 7\nJune 11\nWalltime\n0.10\n0.15\n0.20\n0.25\n0.30\n0.35\n0.40\n0.45\n0.50\n0.55\n0.60\n0.65\n0.70\n0.75\nSuccess Rate\nLearning (Online)\nFrozen (Online)\nLearning (Online)\nFigure 4: Performance of our approach (DigiRL) in\ndifferent training modes on the Webshop subset. When\nutilizing a stale checkpoint, i.e., “frozen” (black+blue\ncurve) performance generally begins to degrade as time\nevolves, whereas autonomous online training (black+red\ncurve) via DigiRL allows us to retain performance de-\nspite non-stationarity and stochasticity.\nSetup for reliable and scalable online RL. As\nautonomous RL interleaves data collection and\ntraining, to maximize learning amidst stochas-\nticity, it is crucial to have a real-time data col-\nlection pipeline to collect enough experience\nfor gradient updates. While this is not possi-\nble in single-thread Android emulator environ-\nments [26, 39] due to latency, we parallelize our\nAndroid emulator using appropriate error han-\ndling as discussed in Appendix A.1. In addition,\nthe environment must provide a reward signal\nby judging whether the current observation in-\ndicates the agent has successfully completed the\ntask. To generalize our evaluator to support a\nwide range of tasks, we extend Pan et al. [26]’s\nend-to-end autonomous evaluator that does not\nrequire accessing the internal states of the emu-\nlator or human-written rules for each task. This\ncontrasts previous works that manually write\nexecution functions to verify the functional com-\npleteness of each task [16, 48, 37, 44]. We adopt Gemini 1.5 Pro [6, 7] as the backbone of the\nautonomous evaluator. We seed this model with few-shot rollouts and the associated human-labeled\nsuccess indicators to guide evaluation of novel queries. This pipeline enables a single evaluator that\ncan evaluate all AiTW tasks. The evaluator is highly aligned with human annotations (average error\nrate 2.8%), validated in Figure 8.\n4\nDigiRL: Autonomous RL for Building a Strong Device-Control Agent\nWe now present our autonomous RL framework for training device agents. We pose the device\ncontrol problem as a Markov decision process (MDP) and develop RL methods for this MDP. The\ncore of our approach is based on a simple and scalable off-policy RL method, advantage-weighted\nregression (AWR) [29], but we make crucial modifications to handle stochasticity and highly-variable\n5\n\n\ntask difficulty, through the use of value functions trained with appropriate losses, and an automatic\ncurriculum, induced by an instruction-level value function to maximize learning.\nDevice control and GUI navigation as a MDP. We conceptualize device control guided by nat-\nural language instructions as a finite horizon Markov Decision Process (MDP) represented by\nM = {S, A, T , µ0, R, H} and run policy gradient to solve this MDP. At the beginning, an initial\nstate s0 and a natural language instruction c are sampled from the initial state distribution µ0. A\nreward of 1 is given at the end if the agent successfully fulfills the task per the evaluator, otherwise\na reward of 0 is given. The trajectory terminates either when the agent accomplishes the task or\nwhen the maximum allowed number of interactions H is exceeded. States are represented using the\nlast two screenshots. To explain our approach in detail, we also include several standard definitions\nused in reinforcement learning (RL). The Q function for a policy π represents the expected long-\nterm return from taking a specific action at the current step and then following policy π thereafter:\nQπ(sh, ah, c) = Eπ\nhPH\nt=h r(st, at, c)\ni\n. The value function V π(sh, c) is calculated by averaging\nthe Q-value, Qπ(sh, ah, c), over actions ah drawn from the policy π. The advantage Aπ(sh, ah, c)\nfor a state-action pair is computed by subtracting the state’s value under the policy from its Q-value:\nAπ(sh, ah, c) = Qπ(sh, ah, c) −V π(sh, c).\n4.1\nBackbone of Our Approach: Off-Policy RL via Advantage-Weighted Regression\nThe starting point we choose to build our approach on is the advantage-weighted regression (AWR)\nalgorithm [29], which says that we can improve the policy reliably by regressing the policy towards\nexponentiated advantages induced by the reward function, as a proxy for optimizing the policy\ngradient while staying close to the previous policy [14, 35, 34]:\narg maxπ Eν [log π(a|s, c) · exp (A(s, a, c)/β)] ,\n(4.1)\nfor some positive parameter β and the distribution of past experience ν, and A(s, a, c) denotes the\nadvantage of a state-action pair (s, a) given a context c. To avoid tuning the hyperparameter β, we\nconsider an alternative that does “hard filtering” on the advantages instead of computing exp(A),\nsimilar to prior works [22, 43]. This leads to the following loss function for fine-tuning the model:\nL(π) = −Efilter(ν)[log π(a|s, c)].\n(4.2)\nTypically, these advantages are computed by running Monte-Carlo (MC) rollouts in the environment\nto estimate the value of a given state-action pair, and subtracting from it an estimate of the value\nof the state given by a learned value estimator alone. However, this approach is likely to produce\nhigh-variance advantages given the stochasticity of the device eco-system that affects MC rollouts.\n4.2\nObtaining Reliable Advantage Estimates from Doubly-Robust Estimators\nTo reliably identify advantageous actions given significant environment stochasticity, we construct a\nper-step advantage estimator, inspired by doubly-robust estimators [40, 36]:\nAstep(sh, ah, c) := λH−hr(sH, aH, c) + (1 −λH−hr(sH, aH, c))(V step(sh+1, c) + r(sh, ah, c) −V step(sh, c)),\n(4.3)\nwhere λ is a weighting hyper-parameter. This construction of the advantage estimator is a simplified\nversion of Generalized Advantage Estimation (GAE) [36] using only the next-step advantage estimator\nand final-step advantage estimator as there are no intermediate rewards in our problem. This construc-\ntion balances an advantage estimator with higher variance Monte-Carlo estimates λH−hr(sH, aH, c)\n(due to stochasticity) and an estimator with higher bias V step(sh+1, c) + r(sh, ah, c) −V step(sh, c)\n(due to imperfect fitting of the value function). We observed that combining both high-variance and\nhigh-bias estimators gave us a sweet-spot in terms of performance. To implement the step-level hard\nfiltering, we simply threshold this doubly robust estimator as Astep(sh, ah, c) > 1/H to decide which\nactions progress towards the goal.\n4.3\nAutomatic Curriculum using an Instruction-Level Value Function\nWhile the AWR update (Equation 4.1) coupled with a robust advantage estimator (Equation 4.3) is\nlikely sufficient on standard RL tasks, we did not find it to be effective enough for device control\nin preliminary experiments. Often this was the case because the task set presents tasks with highly-\nvariable difficulties that collecting more data on tasks that the agent was already proficient at affected\nsample efficieny negatively. In contrast, maximal learning signal can be derived by experiencing the\n6\n\n\ninstruction-level\nvalue function\nstep-level\nvalue function\nactor\nEquation (4.2)\nGo to walmart.com\n(difficulty: easy)\nGo to ebay.com, search  for \n\"asus zenbook\"\n(difficulty: medium)\nGo to ebay.com, search  for \n\"asus zenbook\"\n0.8\n0.2\n-0.01\n0.01\n0.10\n1\n0\n1\nGo to costco.com, search for \n\"bose soundsport free\", and \nselect the first entry\n(difficulty: hard)\ndiscarded\nTask\ndiscarded\ngo to state-\nlevel critic\nTask\nGo to ebay.com, search  for \n\"asus zenbook\"\nTask\nInstruction-level Value Function\nStep-level Value Function\nTrain w/ MLE loss\nFigure 5: Algorithm visualization. The two value function are first trained with original distribution of\ncollected trajectories according to Equation (4.5) and Equation (4.6), then used to filter the trajectories for\ntraining the actor. We use the MLE loss (Maximum Likelihood Estimation loss) to train the actor.\nmost informative tasks for the agent during training. To this end, we design an instruction-level value\nfunction V instruct(c) to evaluate if a given rollout can provide an effective learning signal:\nAinstruct(sh, ah, c) := PH\nt=hr(st, at, c) −V instruct(c) = r(sH, aH, c) −V instruct(c),\n(4.4)\nwhere PH\nt=h r(st, at, c) is a Monte-Carlo estimator of Q(sh, ah, c). The equality holds because the\nMDP formulation only provides rewards at the end of a rollout. Intuitively, if a rollout attains a\nhigh value of Ainstruct(sh, ah, c), it means the value function V instruct is small. Therefore, this rollout\nrepresents a valuable experience of the agent accomplishing a difficult task, and thus should be\nprioritized, akin to ideas pertaining to prioritized experience [32] or level replay [11]. When training\nthe actor with a buffer of historical off-policy data, we first perform a filtering step to identify the\ntop-p datapoints with highest Ainstruct(sh, ah, c). Then, we use it for AWR (Equation 4.1) with the\ndoubly-robust advantage estimator (Equation 4.3).\nImplementation details. Inspired by the findings in some recent works [5, 17] that modern deep\nlearning architectures like transformers [41] are better trained with cross-entropy losses instead of\nmean-squared losses, we utilize a cross-entropy objective based on the Monte-Carlo estimate of the\ntrajectory reward for training both of our value functions:\nL(V traj) = −Eν[r(sH, aH, c) log V traj(c) + (1 −r(sH, aH, c)) log(1 −V traj(c))],\n(4.5)\nL(V step) = −Eν[r(sH, aH, c) log V step(sh, ah, c) + (1 −r(sH, aH, c)) log(1 −V step(sh, ah, c))].\n(4.6)\nFinal algorithm. The final practical algorithm is shown in Figure 5. The instruction-level value\nfunction estimates the values of the trajectories, which is trained with loss shown in Equation (4.5).\nThe step-level value function estimates the values of states, which is trained with loss shown in Equa-\ntion (4.6). When training the actor, we first filter out trajectories and states using the value functions\nas shown in Equation (4.4) and Equation (4.3), then train the actor with the MLE loss shown in\nEquation (4.2) on the filtered data.\n5\nExperimental Evaluation\nThe goal of our experiments is to evaluate the performance of DigiRL on challenging Android device\ncontrol problems. Specifically, we are interested in understanding if DigiRL can produce agents that\ncan effectively learn from autonomous interaction, while still being able to utilize offline data for\nlearning. To this end, we perform a comparative analysis of DigiRL against several prior approaches,\nincluding state-of-the-art agents in Section 5.1. We also perform several ablation experiments to\nunderstand the necessity and sufficiency of various components of our approach in Section 5.2.\nBaselines and comparisons. We compare DigiRL with: (a) state-of-the-art agents built around\nproprietary VLMs, with the use of several prompting and retrieval-style techniques; (b) running\n7\n\n\nAitW General\nAitW Web Shopping\nTrain\nTest\nTrain\nTest\nPrompting\nSET-OF-MARKS\nGPT-4V\n5.2\n13.5\n3.1\n8.3\nGemini 1.5 Pro\n32.3\n16.7\n6.3\n11.5\nAPPAGENT\nGPT-4V\n13.5\n17.7\n12.5\n8.3\nGemini 1.5 Pro\n14.6\n16.7\n5.2\n8.3\nLearning\nSUPERVISED\nTRAINING\nCogAgent\n25.0\n25.0\n31.3\n38.5\nAutoUI\n12.5\n14.6\n14.6\n17.7\nOFFLINE\nFiltered BC\n51.7 ± 5.4\n50.7 ± 1.8\n44.7 ± 1.6\n45.8 ± 0.9\nOurs\n46.9 ± 5.6\n62.8 ± 1.0\n39.3 ± 6.0\n45.8 ± 6.6\nOFF-TO-ON\nFiltered BC\n53.5 ± 0.8\n61.5 ± 1.1\n53.6 ± 4.7\n57.8 ± 2.6\nOurs\n63.5 ± 0.0\n71.9 ± 1.1\n68.2 ± 6.8\n67.2 ± 1.5\nTable 1: Main comparisons of different agents across various settings. Each offline experiment is repeated\nthree times and the mean and standard deviation are reported. Each online experiment is repeated two times.\nResults are evaluated with our autonomous evaluator with the first 96 instructions in the train and test set.\nCorrelation of our correlation and human judgements can be found in Figure 8.\nimitation learning on static human demonstrations with the same instruction distribution, and (c)a\nfiltered BC approach [26]. For proprietary VLMs, we evaluate GPT-4V [24] and Gemini 1.5 Pro [7]\nboth zero-shot and when augmented with carefully-designed prompts. For the zero-shot setting, we\nuse the prompt from Yang et al. [47] and augment the observation with Set-of-Marks [55]. Set-of-\nMarks overlays a number for each interactable element over the screenshot, so that a VLM can directly\noutput the number of the element to interact with in plain text instead of attempting to calculate pixel\ncoordinates, which is typically significantly harder. We also compare with AppAgent [47], which first\nprompts the VLM to explore the environment, and appends the experience collected to the test-time\nprompt. We also compare with two state-of-the-art fine-tuning methods for Android device control:\nAutoUI (specifically AutoUI-Base [53]) and CogAgent [9]. AutoUI-Base uses an LM with 200M\nparameters, and a a vision encoder with 1.1B parameters. CogAgent has 11B parameters for its vision\nencoder and 7B for its LM. The supervised training corpus for both AutoUI-Base and CogAgent\ncontains AitW, including the instruction set and the emulator configuration we use.\nBase VLM and offline dataset. Both Filtered BC and DigiRL use trained AutoUI-Base checkpoints\nwith the image encoder frozen. The instruction and step-level value functions for DigiRL employ\nthis same frozen image encoder. The visual features output from the encoder are concatenated with\ninstruction features derived from RoBERTa [21]. A two-layer MLP is then used to predict the value\nfunction. In the offline phase, the offline dataset is collected by rolling out the initial AutoUI-Base\nsupervised trained checkpoint as policy. For fair comparisons, we keep the number of offline data\ncollected in the pure offline training roughly the same as the total number of data collected in the\noffline-to-online training. Due to the dynamic nature of the Internet-device eco-system, our offline\ndata was stale by the time we were able to run our offline-to-online experiments, and this presented\nadditional challenge in offline-to-online learning. In both General and Web Shopping subsets, offline\nexperiments make use of around 1500 trajectories while offline-to-online experiments start with\naround 500 offline trajectories and update with another 1000 online trajectories. In the offline phase,\nDigiRL skips instruction-level filtering and instead trains the actor with all successful trajectories to\nmake full use of the offline data. See a detailed breakdown of our dataset in Appendix A.1.\n5.1\nMain Results\nOur main results are summarized in Table 1 and Figure 6. We find that on both AitW General\nand AitW Web Shopping subsets, the agent trained via DigiRL significantly outperforms prior\nstate-of-the-art methods based on prompting and retrieval (AppAgent + GPT-4V/Gemini 1.5 Pro) or\ntraining on static demonstrations (CogAgent and AutoUI), by a large margin with more than 49.5%\nabsolute improvement (from 17.7% to 71.9% on the General subset and from 17.7% to 67.2% on\nthe Web Shopping subset). Notably, this improvement from DigiRL is realized fully autonomously\nwithout making use of human supervision (e.g. manually labeled rollouts or hand-written verifiers).\nAre inference-time prompting and retrieval techniques or supervised training enough for\ndevice control? Delving into Table 1, we observe that off-the-shelf proprietary VLMs, even when\n8\n\n\n0\n320\n640\n960\n#Trajectories\n0.1\n0.2\n0.3\n0.4\n0.5\n0.6\n0.7\n0.8\nSuccess Rate\n0\n320\n640\n960\n#Trajectories\n0.1\n0.2\n0.3\n0.4\n0.5\n0.6\n0.7\n0.8\nFiltered BC-1\nFiltered BC-2\nDigiRL-1\nDigiRL-2\nGPT-4V\nFigure 6: Offline-to-online training curves for Filtered BC and DigiRL. Curves are smoothed with expo-\nnential weighting over the x-axis. Left: AitW General. Right: AitW Web Shopping. Two runs for each model\nare started on two different dates with at least two days apart. Observe that DigiRL is able to improve faster\nwith a fewer number of samples. Since the data collection frequency is the bottleneck, these performance trends\ndirectly reflect performance trends against wall-clock time as well.\nFail to recover from mistakes\nGet stuck midway\nArrive at wrong goal\nFailure Mode\n0.0\n0.2\n0.4\n% in All Trajectories\nGeneral\nFail to recover from mistakes\nGet stuck midway\nArrive at wrong goal\nFailure Mode\n0.0\n0.2\n0.4\n% in All Trajectories\nWeb Shopping\nSet-Of-Marks\nGPT4V\nSet-Of-Marks\nGemini-1.5-Pro\nAppAgent\nGPT4V\nAppAgent\nGemini-1.5-Pro\nAutoUI\nCogAgent\nFiltered BC\nOffline\nDigiRL\nOffline\nFiltered BC\nOnline\nDigiRL\nOnline\nFigure 7: Failure modes for each approach on both the AiTW General and Web Shopping subsets. We found\nthat the failure mode RL training is most effective at reducing compared to model supervised trained on human\ndata is “Fail to recover from mistakes”. A more fine-grained decomposition can be found in Appendix D.\nsupplemented with the set-of-marks mechanism, do not attain satisfactory performance: both GPT-4V\nand Gemini 1.5 Pro achieve success rates under 20%. One possible cause could be the under-\nrepresentation of Android device data in the pre-training data. Moreover, inference-time adaptation\nstrategies such as AppAgent [47] show minimal improvement, with gains not exceeding 5% for either\nmodel. All this evidence suggests a limited scope for improvement without fine-tuning of some sort.\nAs illustrated in Figure 7, the primary failures of these VLMs stem from hallucinatory reasoning\nthat lead the VLMs to land on a relevant but wrong page. This suggests that while state-of-the-art\nVLMs excel at reasoning problems in code and math, their reliability in less-familiar domains, such\nas device control, remains inadequate. For example, for the instruction “Go to newegg.com, search\nfor alienware area 51, and select the first entry”, a GPT-4V based agent erroneously searched “alien\narea 51 ebay” in Google.com and decided that it had made progress towards the task (Figure 15).\nTraining on domain-specific human demonstrations, however, does boost performance, allowing\nthe smaller, specialized VLM, AutoUI with 1.5 billion parameters, to match or surpass the larger,\ngeneralist VLMs like GPT-4V and Gemini 1.5 Pro. Nonetheless, this supervised imitation learning\napproach still fall short, with success rates on both subsets remaining below 20%. This shortcoming\nis not fundamentally addressed via enhancements in model scale or architecture, as evidenced by\nCogAgent [9], with 18 billion parameters still achieving performances below 40% success rate. As\ndepicted in Figure 7, a predominant failure mode for these agents is an inability to rectify their own\nerrors. An example trajectory that we observed is that for the instruction “what’s on the menu of\nIn-n-Out”, the agent accidentally activated the voice input button, and failed to quit that page until\nthe step limit. In contrast, DigiRL is able to recover from the errors more efficiently( Appendix C.2).\n9\n\n\nSet-Of-Marks\nGPT4V\nSet-of-Marks\nGemini-1.5-Pro\nAppAgent\nGPT4V\nAppAgent\nGemini-1.5-Pro\nAutoUI\nCogAgent\nFiltered BC\nOffline\nDigiRL\nOffline\nFiltered BC\nOnline\nDigiRL\nOnline\nPolicy Model\n0\n50\n% Success Rate\n17.7\n13.5\n16.7\n16.7\n15.6\n17.7\n18.8\n16.7\n12.5\n14.6\n25.0\n25.0\n55.2\n53.1\n56.3\n63.5\n59.4\n61.5\n70.0\n72.9\nGeneral\nHuman\nGemini-1.5-Pro Evaluator\nSet-Of-Marks\nGPT4V\nSet-Of-Marks\nGemini-1.5-Pro\nAppAgent\nGPT4V\nAppAgent\nGemini-1.5-Pro\nAutoUI\nCogAgent\nFiltered BC\nOffline\nDigiRL\nOffline\nFiltered BC\nOnline\nDigiRL\nOnline\nPolicy Model\n0\n50\n% Success Rate\n11.4\n8.3\n15.6\n11.5\n13.5\n8.3\n13.5\n5.2\n18.8\n17.7\n42.6\n38.5\n45.8\n46.7\n57.3\n55.2\n61.5\n60.4\n68.8\n71.9\nWeb Shopping\nHuman\nGemini-1.5-Pro Evaluator\nFigure 8: Correlation between our autonomous evaluator and human judgements for all policy models on\nGeneral and Web Shopping subsets. For repeated offline and online runs, we report the correlation results for the\nrun with the highest autonomous evaluation success rate.\nComparison of different RL approaches. In Table 1 and Figure 6, we present a comparative\nanalysis of various autonomous approaches. Notably, both offline and offline-to-online configurations\ndemonstrate that our RL approach, when augmented with a continuous stream of autonomous\ninteraction data and reward feedback, substantially improves performance. This improvement is\nevident from an increase in the success rate from under 20% to over 40%, as the agent learns to\nadapt to stochastic and non-stationary device interfaces. Moreover, although the total sample sizes\nfor offline and offline-to-online settings are equivalent, the top-performing offline-to-online algorithm\nmarkedly surpasses its offline counterpart (75% versus 62.8% on the General subset). This highlights\nthe efficacy of autonomous environment interaction, and establishes the efficacy of DigiRL in learning\nfrom such uncurated, sub-optimal data. Lastly, DigiRL consistently outperforms the state-of-the-art\nalternative, Filtered BC, across both the General and Web Shopping subsets, improving from 61.5%\nto 71.9% and 57.8% to 61.4%, respectively, highlighting DigiRL’s performance and efficiency.\n5.2\nAnalysis and Ablations\nFailure modes analysis. We conduct an additional user study to annotate the failure modes for each\nagent as shown in Figure 7, and a more fine-grained breakdown can be found in Appendix D. At a\nhigh level, we classify the major failure modes of all agents into the following three categories: (1)\nFailure to recover from mistakes refers to the scenario where the agent made a mistake that led it to\nstates from which it failed to quickly recover and resume the task, such as a wrong search page. (2)\nGetting stuck midway refers to the failure mode where the agent gets distracted on the right track to\ncompleting the instruction and as a result fails to accomplish the task. For example, failing to click on\nthe right link or failing to search after typing the key words. (3) Arriving at wrong goal refers to the\nfailure mode where the agent arrives at a wrong page and mistakenly thinks that it had completed the\ntask. For e.g, the agent finds a macbook on costco.com instead of finding a macbook on ebay.com.\nWhile all the types of failure modes benefit from offline and offline-to-online RL training as shown\nin Figure 7, the most consistent and significant reduction is probably for the failure mode of failing\nto recover from mistakes. This is because while pre-trained models, generating plausible future\ntokens, can get distracted by the dynamic nature of the environment and, as a result, encounter at\nnever-before-seen states. With no clue of how to escape such states, these methods are unable to\nrecover and fail to solve the task. In contrast, by training on autonomously-collected rollouts, our\nagent DigiRL is able to learn from its own mistakes and reduces failures to recover over training.\nAblation study of each component in DigiRL. We conduct an ablation study on different components\nof DigiRL in Figure 9 (left). We find that all the components used by our approach are necessary: (1)\nusing cross-entropy for training the value functions boosts performance by around 12% (compare Ours\nand Ours w/ Regression); (2) using step-level advantages improves efficiency by 12% (comparing\nOurs and Ours w/o step-level advantage); (3) the use of automatic curriculum improves the speed\nof learning by around 25% (comparing Ours w/o step-level advantage and Filtered BC); (4) Ours\noutperforms vanilla AWR that does not employ a doubly-robust advantage estimator or curriculum.\nAdditionally, we also observe no degradation in performance as a result of “hard-filtering”, as show\nby nearly comparable performance of our approach and the best run of exponential filtering obtained\nvia an extensive tuning of the temperature hyperparameter τ in naïve AWR (comparing Ours and Ours\n10\n\n\n0\n100\n200\n300\n400\n500\n600\n#Trajectories\n0.20\n0.25\n0.30\n0.35\n0.40\n0.45\n0.50\n0.55\n0.60\n0.65\nSuccess Rate\nOurs\nOurs w/ regression\nOurs w/o step-level advantage\nVanilla AWR\nOurs w/ AWR reweighting\nFiltered BC\n8 16\n32\n64\n128\n#CPUs\n0\n1\n2\n3\n4\n5\nEmulation Speed (traj/min)\n0.36\n0.53 0.68\n0.74\n0.49\n0.99\n1.74\n3.55\nVanilla Emulator\nDistributed Emulator\nUpper Bound\nFigure 9: Left: Ablation study results on the AitW Web Shopping subset. Right: Emulation speed w.r.t\nnumber of CPUs used. The upper bound can only achieved when there is no communication and error handling\ncost. Our design of distributed emulator can significantly improve the efficiency of emulation compaared to the\nvanilla method of running all emulations over the same instance.\nw/ vanilla AWR reweighting), despite simplicity of implementation in the hard filtering approach.\nPutting together, these choices result in a new state-of-the-art RL approach for device control.\nEvaluation of our autonomous evaluator. In Figure 8, we present the findings from a user study\naimed at assessing the accuracy of our autonomous evaluator. Our results indicate that the success\nrates reported by our automatic evaluator are remarkably consistent with those assessed by human\nevaluators across almost all models, with differences less than 3%. Furthermore, we observed that\nevaluations on the Web Shopping subset are more precise compared to those on the General subset.\nThis increased accuracy likely stems from the fact that tasks in the General subset are formulated in\nfree-form language, which can introduce ambiguity, whereas the Web Shopping subset features a\nnarrower range of language expressions, reducing potential variability.\nSpeedup of emulation parallel. The performance boost with respect to the number of worker\nmachines is nearly linear, as demonstrated in Figure 9 (right), where we conduct experiments\nthat examine the scaling performance of our parallel emulator. Our distributed emulator that runs\nemulations across multiple servers can reliably collect data with up to 64 parallel emulators on 128\nCPUs with near-linear speedup. In contrast, a naive baseline that runs all parallel emulations on the\nsame server achieves much inferior performance (0.74 compared to 1.74 trajs/min using 64 CPUs).\n6\nDiscussion and Limitations\nIn this paper, we propose a novel autonomous RL approach, DigiRL, for training in-the-wild, multi-\nmodal, device-control agents that establish a new state-of-the-art performance on a number of Android\ncontrol tasks from Android-in-the-Wild dataset [31]. To achieve this, we first build a scalable and\nparallelizable Android environment with a robust VLM-based general-purpose evaluator that supports\nfast online data collection. We then develop a system for offline RL pre-training, followed by\nautonomous RL fine-tuning to learn via interaction, admist the stochasticity of the real-world Internet\nand device eco-system. Our agent achieves a 280% improvement over the previous state-of-the-art\nagents (from 17.7% to 68.2% in terms of task success rate), including AppAgent based on GPT-4V\nand Gemini 1.5 Pro, and supervised trained models such as AutoUI and CogAgent.\nDue to computational limitations, and despite the fact that the parallel emulator and autonomous\nevaluator can be easily extended to complicated tasks, our agent is trained only with tasks from AitW\ninstead of a all possible tasks on the device. Our design of the DigiRL algorithm aims for maximal\nimplementation simplicity, so we hope that our approach to serve as a base algorithm for future\nresearch to build on, including algorithmic research as well as expanding the space of tasks.\nAcknowledgements\nWe thank Yi Su, Izzedin Gur, Xinyang Geng, and Sandra Faust for feedback on an earlier version of\nthis paper and for informative discussions. This work is supported by NSF IIS-2246811 and ONR\n11\n\n\nN00014-21-1-2838, and Gemini 1.5 Pro credit donations for academic use and cloud resources from\nGoogle Cloud.\nReferences\n[1] Marwa Abdulhai, Isadora White, Charlie Snell, Charles Sun, Joey Hong, Yuexiang Zhai, Kelvin\nXu, and Sergey Levine. Lmrl gym: Benchmarks for multi-turn reinforcement learning with\nlanguage models, 2023.\n[2] Stephen Casper, Xander Davies, Claudia Shi, Thomas Krendl Gilbert, Jérémy Scheurer, Javier\nRando, Rachel Freedman, Tomasz Korbak, David Lindner, Pedro Freire, Tony Wang, Samuel\nMarks, Charbel-Raphaël Segerie, Micah Carroll, Andi Peng, Phillip Christoffersen, Mehul\nDamani, Stewart Slocum, Usman Anwar, Anand Siththaranjan, Max Nadeau, Eric J. Michaud,\nJacob Pfau, Dmitrii Krasheninnikov, Xin Chen, Lauro Langosco, Peter Hase, Erdem Bıyık,\nAnca Dragan, David Krueger, Dorsa Sadigh, and Dylan Hadfield-Menell. Open problems and\nfundamental limitations of reinforcement learning from human feedback, 2023.\n[3] Baian Chen, Chang Shu, Ehsan Shareghi, Nigel Collier, Karthik Narasimhan, and Shunyu\nYao. Fireact: Toward language agent fine-tuning. ArXiv, abs/2310.05915, 2023. URL https:\n//api.semanticscholar.org/CorpusID:263829338.\n[4] Alexandre Drouin, Maxime Gasse, Massimo Caccia, Issam H. Laradji, Manuel Del Verme, Tom\nMarty, Léo Boisvert, Megh Thakkar, Quentin Cappart, David Vazquez, Nicolas Chapados, and\nAlexandre Lacoste. Workarena: How capable are web agents at solving common knowledge\nwork tasks?, 2024.\n[5] Jesse Farebrother, Jordi Orbay, Quan Vuong, Adrien Ali Taïga, Yevgen Chebotar, Ted Xiao,\nAlex Irpan, Sergey Levine, Pablo Samuel Castro, Aleksandra Faust, Aviral Kumar, and Rishabh\nAgarwal. Stop regressing: Training value functions via classification for scalable deep rl, 2024.\n[6] 2023 Gemini Team. Gemini: A family of highly capable multimodal models, 2024.\n[7] 2024 Gemini Team. Gemini 1.5: Unlocking multimodal understanding across millions of tokens\nof context, 2024.\n[8] Dibya Ghosh, Jad Rahme, Aviral Kumar, Amy Zhang, Ryan P Adams, and Sergey Levine.\nWhy Generalization in RL is Difficult: Epistemic POMDPs and Implicit Partial Observability.\nNeurIPS, 2021.\n[9] Wenyi Hong, Weihan Wang, Qingsong Lv, Jiazheng Xu, Wenmeng Yu, Junhui Ji, Yan Wang,\nZihan Wang, Yuxuan Zhang, Juanzi Li, Bin Xu, Yuxiao Dong, Ming Ding, and Jie Tang.\nCogagent: A visual language model for gui agents, 2023.\n[10] Peter C Humphreys, David Raposo, Toby Pohlen, Gregory Thornton, Rachita Chhaparia, Alistair\nMuldal, Josh Abramson, Petko Georgiev, Alex Goldin, Adam Santoro, and Timothy Lillicrap.\nA data-driven approach for learning to control computers, 2022.\n[11] Minqi Jiang, Edward Grefenstette, and Tim Rocktäschel. Prioritized level replay. CoRR,\nabs/2010.03934, 2020. URL https://arxiv.org/abs/2010.03934.\n[12] Yiding Jiang, J Zico Kolter, and Roberta Raileanu. On the importance of exploration for\ngeneralization in reinforcement learning. Advances in Neural Information Processing Systems,\n36, 2024.\n[13] Carlos E. Jimenez, John Yang, Alexander Wettig, Shunyu Yao, Kexin Pei, Ofir Press, and\nKarthik Narasimhan. Swe-bench: Can language models resolve real-world github issues?, 2024.\n[14] Sham M. Kakade and John Langford. Approximately optimal approximate reinforcement\nlearning.\nIn International Conference on Machine Learning, 2002.\nURL https://api.\nsemanticscholar.org/CorpusID:31442909.\n[15] Raghav Kapoor, Yash Parag Butala, Melisa Russak, Jing Yu Koh, Kiran Kamble, Waseem\nAlshikh, and Ruslan Salakhutdinov. Omniact: A dataset and benchmark for enabling multimodal\ngeneralist autonomous agents for desktop and web, 2024.\n12\n\n\n[16] Jing Yu Koh, Robert Lo, Lawrence Jang, Vikram Duvvur, Ming Chong Lim, Po-Yu Huang,\nGraham Neubig, Shuyan Zhou, Ruslan Salakhutdinov, and Daniel Fried. Visualwebarena:\nEvaluating multimodal agents on realistic visual web tasks. arXiv preprint arXiv:2401.13649,\n2024.\n[17] Aviral Kumar, Rishabh Agarwal, Xinyang Geng, George Tucker, and Sergey Levine. Offline\nq-learning on diverse multi-task data both scales and generalizes, 2023.\n[18] Hanyu Lai, Xiao Liu, Iat Long Iong, Shuntian Yao, Yuxuan Chen, Pengbo Shen, Hao Yu,\nHanchen Zhang, Xiaohan Zhang, Yuxiao Dong, and Jie Tang. Autowebglm: Bootstrap and\nreinforce a large language model-based web navigating agent, 2024.\n[19] Juyong Lee, Taywon Min, Minyong An, Changyeon Kim, and Kimin Lee. Benchmarking\nmobile device control agents across diverse configurations, 2024.\n[20] Xiao Liu, Hao Yu, Hanchen Zhang, Yifan Xu, Xuanyu Lei, Hanyu Lai, Yu Gu, Hangliang Ding,\nKaiwen Men, Kejuan Yang, Shudan Zhang, Xiang Deng, Aohan Zeng, Zhengxiao Du, Chenhui\nZhang, Sheng Shen, Tianjun Zhang, Yu Su, Huan Sun, Minlie Huang, Yuxiao Dong, and Jie\nTang. Agentbench: Evaluating llms as agents, 2023.\n[21] Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy,\nMike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. Roberta: A robustly optimized BERT\npretraining approach. CoRR, abs/1907.11692, 2019. URL http://arxiv.org/abs/1907.\n11692.\n[22] Ashvin Nair, Murtaza Dalal, Abhishek Gupta, and Sergey Levine. Accelerating online re-\ninforcement learning with offline datasets.\nCoRR, abs/2006.09359, 2020.\nURL https:\n//arxiv.org/abs/2006.09359.\n[23] OpenAI, Ilge Akkaya, Marcin Andrychowicz, Maciek Chociej, Mateusz Litwin, Bob McGrew,\nArthur Petron, Alex Paino, Matthias Plappert, Glenn Powell, Raphael Ribas, Jonas Schneider,\nNikolas Tezak, Jerry Tworek, Peter Welinder, Lilian Weng, Qiming Yuan, Wojciech Zaremba,\nand Lei Zhang. Solving rubik’s cube with a robot hand, 2019.\n[24] 2023 OpenAI Team. Gpt-4 technical report, 2023.\n[25] Long Ouyang, Jeff Wu, Xu Jiang, Diogo Almeida, Carroll L. Wainwright, Pamela Mishkin,\nChong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, John Schulman, Jacob Hilton,\nFraser Kelton, Luke E. Miller, Maddie Simens, Amanda Askell, Peter Welinder, Paul Francis\nChristiano, Jan Leike, and Ryan J. Lowe. Training language models to follow instructions with\nhuman feedback. ArXiv, abs/2203.02155, 2022. URL https://api.semanticscholar.org/\nCorpusID:246426909.\n[26] Jiayi Pan, Yichi Zhang, Nicholas Tomlin, Yifei Zhou, Sergey Levine, and Alane Suhr. Au-\ntonomous evaluation and refinement of digital agents. arXiv preprint arXiv:2404.06474, 2024.\n[27] Richard Yuanzhe Pang, Weizhe Yuan, Kyunghyun Cho, He He, Sainbayar Sukhbaatar, and\nJason Weston. Iterative reasoning preference optimization, 2024.\n[28] Xue Bin Peng, Aviral Kumar, Grace Zhang, and Sergey Levine. Advantage-weighted regression:\nSimple and scalable off-policy reinforcement learning. CoRR, abs/1910.00177, 2019. URL\nhttp://arxiv.org/abs/1910.00177.\n[29] Xue Bin Peng, Aviral Kumar, Grace Zhang, and Sergey Levine. Advantage-weighted regression:\nSimple and scalable off-policy reinforcement learning, 2019.\n[30] Yujia Qin, Shihao Liang, Yining Ye, Kunlun Zhu, Lan Yan, Yaxi Lu, Yankai Lin, Xin Cong,\nXiangru Tang, Bill Qian, Sihan Zhao, Lauren Hong, Runchu Tian, Ruobing Xie, Jie Zhou,\nMark Gerstein, Dahai Li, Zhiyuan Liu, and Maosong Sun. Toolllm: Facilitating large language\nmodels to master 16000+ real-world apis, 2023.\n[31] Christopher Rawles, Alice Li, Daniel Rodriguez, Oriana Riva, and Timothy Lillicrap. Android\nin the wild: A large-scale dataset for android device control. arXiv preprint arXiv:2307.10088,\n2023.\n13\n\n\n[32] Tom Schaul, John Quan, Ioannis Antonoglou, and David Silver. Prioritized experience replay,\n2016.\n[33] Timo Schick, Jane Dwivedi-Yu, Roberto Dessì, Roberta Raileanu, Maria Lomeli, Luke Zettle-\nmoyer, Nicola Cancedda, and Thomas Scialom. Toolformer: Language models can teach\nthemselves to use tools, 2023.\n[34] John Schulman, Sergey Levine, Philipp Moritz, Michael I. Jordan, and Pieter Abbeel. Trust\nregion policy optimization. CoRR, abs/1502.05477, 2015. URL http://arxiv.org/abs/\n1502.05477.\n[35] John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal\npolicy optimization algorithms. CoRR, abs/1707.06347, 2017. URL http://arxiv.org/abs/\n1707.06347.\n[36] John Schulman, Philipp Moritz, Sergey Levine, Michael Jordan, and Pieter Abbeel. High-\ndimensional continuous control using generalized advantage estimation, 2018.\n[37] Tianlin Shi, Andrej Karpathy, Linxi Fan, Jonathan Hernandez, and Percy Liang. World of\nbits: An open-domain platform for web-based agents. In Doina Precup and Yee Whye Teh,\neditors, Proceedings of the 34th International Conference on Machine Learning, volume 70 of\nProceedings of Machine Learning Research, pages 3135–3144. PMLR, 06–11 Aug 2017. URL\nhttps://proceedings.mlr.press/v70/shi17a.html.\n[38] Charlie Snell, Ilya Kostrikov, Yi Su, Mengjiao Yang, and Sergey Levine. Offline rl for natural\nlanguage generation with implicit language q learning, 2023.\n[39] Daniel Toyama, Philippe Hamel, Anita Gergely, Gheorghe Comanici, Amelia Glaese, Zafarali\nAhmed, Tyler Jackson, Shibl Mourad, and Doina Precup. Androidenv: A reinforcement learning\nplatform for android. arXiv preprint arXiv:2105.13231, 2021.\n[40] Hado van Hasselt, Arthur Guez, and David Silver. Deep reinforcement learning with double\nq-learning. CoRR, abs/1509.06461, 2015. URL http://arxiv.org/abs/1509.06461.\n[41] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez,\nLukasz Kaiser, and Illia Polosukhin. Attention is all you need, 2023.\n[42] Siddharth Verma, Justin Fu, Mengjiao Yang, and Sergey Levine. Chai: A chatbot ai for\ntask-oriented dialogue with offline reinforcement learning, 2022.\n[43] Ziyu Wang, Alexander Novikov, Konrad Zolna, Jost Tobias Springenberg, Scott Reed, Bobak\nShahriari, Noah Siegel, Josh Merel, Caglar Gulcehre, Nicolas Heess, and Nando de Freitas.\nCritic regularized regression, 2021.\n[44] Tianbao Xie, Danyang Zhang, Jixuan Chen, Xiaochuan Li, Siheng Zhao, Ruisheng Cao, Toh Jing\nHua, Zhoujun Cheng, Dongchan Shin, Fangyu Lei, et al. Osworld: Benchmarking multimodal\nagents for open-ended tasks in real computer environments. arXiv preprint arXiv:2404.07972,\n2024.\n[45] An Yan, Zhengyuan Yang, Wanrong Zhu, Kevin Lin, Linjie Li, Jianfeng Wang, Jianwei Yang,\nYiwu Zhong, Julian McAuley, Jianfeng Gao, Zicheng Liu, and Lijuan Wang. Gpt-4v in\nwonderland: Large multimodal models for zero-shot smartphone gui navigation, 2023.\n[46] John Yang, Akshara Prabhakar, Karthik Narasimhan, and Shunyu Yao. Intercode: Standardizing\nand benchmarking interactive coding with execution feedback, 2023.\n[47] Zhao Yang, Jiaxuan Liu, Yucheng Han, Xin Chen, Zebiao Huang, Bin Fu, and Gang Yu.\nAppagent: Multimodal agents as smartphone users. arXiv preprint arXiv:2312.13771, 2023.\n[48] Shunyu Yao, Howard Chen, John Yang, and Karthik Narasimhan. Webshop: Towards scalable\nreal-world web interaction with grounded language agents, 2023.\n[49] Aohan Zeng, Mingdao Liu, Rui Lu, Bowen Wang, Xiao Liu, Yuxiao Dong, and Jie Tang.\nAgenttuning: Enabling generalized agent abilities for llms, 2023.\n14\n\n\n[50] Yuexiang Zhai, Hao Bai, Zipeng Lin, Jiayi Pan, Shengbang Tong, Yifei Zhou, Alane Suhr,\nSaining Xie, Yann LeCun, Yi Ma, and Sergey Levine. Fine-tuning large vision-language models\nas decision-making agents via reinforcement learning. arXiv preprint arXiv:2405.10292, 2024.\n[51] Chaoyun Zhang, Liqun Li, Shilin He, Xu Zhang, Bo Qiao, Si Qin, Minghua Ma, Yu Kang,\nQingwei Lin, Saravan Rajmohan, et al. Ufo: A ui-focused agent for windows os interaction.\narXiv preprint arXiv:2402.07939, 2024.\n[52] Jiwen Zhang, Jihao Wu, Yihua Teng, Minghui Liao, Nuo Xu, Xiao Xiao, Zhongyu Wei, and\nDuyu Tang. Android in the zoo: Chain-of-action-thought for gui agents, 2024.\n[53] Zhuosheng Zhang and Aston Zhang. You only look at screens: Multimodal chain-of-action\nagents, 2023.\n[54] Ziniu Zhang, Shulin Tian, Liangyu Chen, and Ziwei Liu. Mmina: Benchmarking multihop\nmultimodal internet agents. arXiv preprint arXiv:2404.09992, 2024.\n[55] Boyuan Zheng, Boyu Gou, Jihyung Kil, Huan Sun, and Yu Su. Gpt-4v(ision) is a generalist\nweb agent, if grounded, 2024.\n[56] Shuyan Zhou, Frank F. Xu, Hao Zhu, Xuhui Zhou, Robert Lo, Abishek Sridhar, Xianyi Cheng,\nYonatan Bisk, Daniel Fried, Uri Alon, and Graham Neubig. Webarena: A realistic web\nenvironment for building autonomous agents. ArXiv, abs/2307.13854, 2023. URL https:\n//api.semanticscholar.org/CorpusID:260164780.\n[57] Yifei Zhou, Andrea Zanette, Jiayi Pan, Sergey Levine, and Aviral Kumar. Archer: Training\nlanguage model agents via hierarchical multi-turn rl. arXiv preprint arXiv:2402.19446, 2024.\n[58] Daniel M. Ziegler, Nisan Stiennon, Jeffrey Wu, Tom B. Brown, Alec Radford, Dario Amodei,\nPaul F. Christiano, and Geoffrey Irving. Fine-tuning language models from human preferences.\nCoRR, abs/1909.08593, 2019. URL http://arxiv.org/abs/1909.08593.\n15\n\n\nAppendices\nA\nEnvironment details\nA.1\nPost-processing of AitW\nThe Android in the Wild (AiTW) task set is a large-scale dataset for android device control, containing\nfive subsets: GoogleApps, Install, Web Shopping, General, and Single, where we select the General\nand Web Shopping subsets. Single subset is not considered here because all tasks in Single can be\ncompleted within one step and thus this subset fails to examine the multi-step challenges that we are\ninterested in this paper. Install and GoogleApps are not considered due to security reasons as those\ntasks require an active Google account and parallel emulations can flag security concerns.\nGeneral. The General set focuses on searching for information and basic application usage. For\nexample, it contains searching for latest news in Chile, search for flights from NYC to Sydney,\nopening Gmail, etc. We use all 545 tasks in the training set for training and the first 96 tasks in the\ntest set for testing due to computational and budget constraints. The maximum allowed number of\nsteps for this subset is 10. Offline data is collected by rolling our the initial AutoUI policy with tasks\nfrom the training set. The offline data used for the offline-to-online setting contains 608 trajectories\nwhile the offline data used for the offline setting contains 1552 trajectories. Some task examples are\nshown in Table 3.\nTask Example\nHow do I get to the nearest Verizon Store?\nHow much does a 2 bedroom apartment rent for in Denver?\nSearch for flights from Barcelona to Boston\nWhat’s a good restaurant in New York?\nWhat’s on the menu at Burger King?\nTable 2: Examples of task descriptions in the AiTW General task set.\nWeb Shopping. The Web Shopping subset comprises search instructions on various shopping\nwebsites, like searching for razer blader on ebay. As some websites (e.g. Amazon) and operations\n(e.g. adding items to cart) frequently require captcha verifications, we post-process the Web Shopping\nsubset to exclude such operations and websites and also make the task easy to evaluate for our\nautonomous evaluator. The resulting task set involves navigating through five websites (costco.com,\nbestbuy.com, target.com, walmart.com, newegg.com) and three basic operations (go to website,\nsearch in the website, and select items from the searched results). Our post-processed training set\ncontains 438 tasks and our testing set contains 96 tasks. Example tasks after post-processing can\nbe found in Table 3. The maximum allowed number of steps for this subset is 20. Offline data is\ncollected by rolling our the initial AutoUI policy with tasks from the training set. The offline data\nused for the offline-to-online setting contains 528 trajectories while the offline data used for the\noffline setting contains 1296 trajectories.\nDifficulty\nTask Example\n1\nGo to costco.com\nGo to walmart.com\n2\nGo to costco.com, search for \"bose soundsport free\"\nGo to walmart.com, search for \"logitech g910\"\n3\nGo to costco.com, search for \"bose soundsport free\" and select the first entry\nGo to walmart.com, search for \"logitech g910\" and select the first entry\nTable 3: Examples of task descriptions in the AiTW Webshopping task set.\n16\n\n\n0\n200\n400\n600\n800\n#Trajectories\n0.1\n0.2\n0.3\n0.4\n0.5\n0.6\n0.7\nSuccess Rate\nFiltered BC-20\nFiltered BC-10\nDigiRL-20\nDigiRL-10\nGPT-4V\nFigure 10: Success rate with different horizon length (H ∈{10, 20})under different methods on\nthe AiTW Google Search task set.\nAitW General\nAitW Web Shopping\nAll Trajectories\nSuccessful Trajectories\nAll Trajectories\nSuccessful Trajectories\nDigiRL Run1\n6.31\n4.40\n11.35\n7.23\nDigiRL Run2\n6.64\n5.04\n10.86\n6.55\nFiltered BC Run1\n8.08\n6.56\n12.05\n6.88\nFiltered BC Run2\n7.36\n6.13\n14.72\n9.62\nTable 4: Average rollout length of the DigiRL agent compared to filtered BC. Darker green means shorter\nrollout length. On both AitW General and AitW Web Shopping test subsets, we find that DigiRL consistently\nproduces shorter length rollouts than filtered BC.\nB\nOther Quantitative Experiments\nB.1\nHorizon Limit\nWe investigate the horizon limit of filtered BC and DigiRL on the AitW General subset. As most\ntasks can be effectively solved within 10 steps, we specify two horizon limits: a sufficient horizont\nH = 10, and a redundant horizon H = 20. Results in Figure 10 show that a redundant horizon\nintroduces significantly faster learning speed for both filtered BC and DigiRL, presumbaly because\nlonger horizon means more opportunity to try in a single trajectory. In both horizon settings, we\nobserve the DigiRL offers a significant speedup of around 100 trajectories over Filtered BC.\nB.2\nTrajectory Length\nWe investigate the rollout length of DigiRL compared to filtered BC. Results in Table 4 demonstrate\nthat DigiRL consistently achieves shorter average rollout lengths compared to filtered BC across both\nsubsets. This observation holds true whether considering all rollouts for computing this correlation or\nonly investigating this correlation on rollouts that eventually succeed. This indicates the capability of\nDigiRL to solve tasks in a more efficient and directed manner. Qualitative examples can be found\nin Figure 14.\nC\nQualitative Examples\nC.1\nRandom sample of trajectories for different agents\nIn Figures 11 and 12, we provide trajectories of DigiRL, AutoUI, and GPT-4V randomly sampled\nfrom our test set to offer a qualitative understanding of the agents’ performance. As shown in these\nexamples, DigiRLcan efficiently carry out in-the-wild device control tasks and less likely to get stuck\nor get to a wrong page compared to AutoUI and GPT-4V.\n17\n\n\nDigiRL:\nAutoUI:\nGPT-4V\nWhat are the new products by Samsung?

\nGot \nstuck\n✘\nClick\n Show me some nice wallpapers for my tablet

\nDigiRL:\nAutoUI:\nGPT-4V\nSkipped\nStops \nEarly\n✘\nFigure 11: Agents’ trajectory on two randomly sampled tasks on the General split of AitW.\n18\n\n\nGo to costco.com, search for 'macbook pro', and select the first entry

\nDigiRL:\nAutoUI:\nGPT-4V\nEarly \nstop\n✘\nGot \nstuck\n✘\n✘\nGot \nstuck\nGo to newegg.com, search for 'duracell triple a’\nDigiRL:\nAutoUI:\nGPT-4V\nSkipped\nSkipped\nSkipped\nWrong\nPage\n✘\n✘\nCould not\nsearch\nFigure 12: Agents’ trajectory on two randomly sampled tasks on the WebShop split of AitW.\n19\n\n\nGo to bestbuy.com, search for 'macbook'\nDigiRL:\nAutoUI:\nSkipped\nSkipped\n✘\nGot \nStuck\nFigure 13: Error recovery cases. In bestbuy.com, we systematically find DigiRL able to recover\nfrom its own mistakes, while AutoUI fails to do so.\nC.2\nError Recovery\nWe observe that DigiRL is able to recover from its own mistakes. As shown in Figure 13, we find\nthat DigiRL explores ways to get back to the original screen in order to perform a search. As a\ncomparison, AutoUI fails to reset to the original screen and gets stuck at the diverged screen. Under\nthe hood, we find DigiRL trying to maximize the state value, which usually induces it to reset to the\noriginal screen (that has a large value to success).\nC.3\nTrajectory Length\nQualitative example on the number of steps in trajectories of DigiRL and filtered BC are shown\nin Figure 14. We find consistent cases where DigiRL has shorter trajectory length than filtere BC.\nC.4\nReasoning failure of GPT-4V\nThe performance of GPT-4V failed on AiTW tasks predominantly due to not being able to carry out\ncontrol actions as it plans on a high level, and then not being able to recover from these mistakes.\nMoreover, one of the main reasons why it is not able to recover from a mistake is that it might\nhallucinate and make itself believe that it is a wrong app or website. Indeed, GPT-4V constructs\na plan of further actions when provided a task from either Web Shopping or General dataset of\nAiTW. Then, when it makes a misclick and fails to successfully proceed in an intermediate step,\nit might think that it actually solved that intermediate step and is in the correct app or website to\nexecute further actions, causing the overall trajectory to fail. An example of this is provided in\nFigure 15. Here, we ask the model to search for an item in a webshopping website, in particular in\n“newegg.com”. However, the model fails to proceed to that website due to not being able to precisely\nlocating the search button. Then, instead of trying to go to that website again, the model thinks it is\nalready in that webshopping website, and mistakes the search bar of Google with the search bar of\n“newegg.com”. Hence, the rest of the trajectory also fails. Another slightly different phenomenon is\nillustrated in Figure 16. Here, the model is able to proceed to the correct website and search for an\nitem, but this time it fails to tap on the search button on the website and clicks to an advertisement\n20\n\n\nGo to ebay.com, search for \"lenovo thinkpad\"\nDigiRL\nFiltered BC\nSearch for flights from Seoul to Mexico city\nDigiRL\nFiltered BC\nFigure 14: Examples where DigiRL has shorter trajectory length than online filtered BC.\ninstead. Consequently, the model fools itself to think it successfully searched the item, and scrolls\nthe page hoping to find that item, but it cannot do so because in reality it views the results of the\nadvertisement. The primary reason of these failures is the challenge of grounding the control actions\nin GUI interfaces to realize the intermediary goals laid out by GPT-4V model’s thoughts. As an\nexample, we provide an illustration of trying to set up an alarm task in Figure 17. Here, in the last\nframe, it fails to execute the precise movements in the necessary amount of rounds to correctly set up\nthe alarm to the desired time, and in the last frame we see that the action taken does not align with\nthe thought process of the model.\nD\nFine-grained failure modes\nIn Figure 18, we present a more fine-grained breakdown for all six failure modes provided in the user\nstudy. Those failure modes include:\n• Failure to recover from mistakes refers to the scenario where the agent made a mistake that\nled it to states from which it failed to quickly recover and resume the task, such as a wrong\ngoogle search page.\n• Failure to click on the right link or failure to click refers to the failure mode where the agent\neither fails to locate the element that it tries to click on and keeps clicking on the nearby\nregion, or fails to start typing in the string when it is supposed to do so.\n• Failure to take reasonable attempts at all refers to the failure mode where there is no clear\nreason that the agent fails to complete the task and does not seem to be on the right track\nthroughout the trajectory.\n• Quit or press HOME early refers to the failure mode where the agent decided to finish the\ntask or press HOME to start over before the task is actually finished.\n21\n\n\nGo to newegg.com, search for “alienware area 51” and select the first entry.\n...\nThought: To \ncomplete the given \ntask, I should launch \nthe Google Chrome \nbrowser to navigate \nto newegg.com. The \nUI element labeled \nwith the number 5 is \ndocumented as an \napp launcher for \nGoogle Chrome. \n\nAction: tap(5)\nThought: To \ncomplete the given \ntask, the next step I \nshould do is enter the \nURL \"newegg.com\" \ninto the address bar.\n\nAction: \ntext(\"newegg.com\")\nThought: To \ncomplete the given \ntask, the next step I \nshould do is to \nnavigate to the \nNewegg website by \ntapping the search or \ngo button associated \nwith the entered \nURL. \n\nAction: tap(11)\nThought: To \ncomplete the given \ntask, I should clear \nthe current text in \nthe address bar and \nenter the search term \n'alienware area 51' \nsince the goal is to \nsearch for that \nspecific product on \nnewegg.com.\n\nAction: tap(2)      \nThought: To \ncomplete the given \ntask, the next step I \nshould do is clear the \ncurrent text in the \naddress bar and \nenter the search term \n'alienware area 51' \nAction: \ntext(\"alienware area \n51\")\nsince the browser \nhas already \nnavigated to the \nNewegg website. \n\nThought: To \ncomplete the given \ntask, the next step I \nshould do is to \nsubmit the search \nquery 'alienware area \n51' \nSince the \nsearch term is \nalready entered, I \nneed to tap the \nsearch or go button \nto initiate the search. \n\non the Newegg \nwebsite. \nAction: tap(10)\nFigure 15: Failure of GPT-4V, with its thoughts and link-based actions given. A typical cause of\nfailure is that it cannot tap on the correct “search” button after entering a query and mistakenly tapped\nonto the “x” symbol in the search bar as the “search” button. Here the goal is: Go to newegg.com,\nsearch for “alienware area 51” and select the first entry. As seen in red emboldened actions, it fails to\npress search button and deletes the query instead. Also, as seen in red highlighted parts in thoughts, it\nthinks it is in “newegg.com” website even though it is not.\nThought: To complete \nthe given task, the next \nstep I should do is type \n\"costco.com\" into the \nsearch bar since I have \nalready initiated the \nprocess to navigate to \ncostco.com by tapping \non the Google search \nbar.\n\nAction: \ntext(\"costco.com\")\nThought: To complete \nthe given task, the next \nstep I should do is enter \nthe search term 'acer \npredator' into the search \nbar using the text input \nfunction.\n\nAction: text(\"acer \npredator\")\nThought: To complete the \ngiven task, \nAction: \nI should \ncontinue to swipe up on \nthe product listings to \nfind the 'acer predator' \nitem.\n\nswipe(21, \"up\", \n\"medium\")\nThought: To complete the \ngiven task, I should scroll \nthrough the product \nlistings to find the 'acer \npredator' item. Since the \nfirst entry is what we \nneed to select, \nAction: \nI should \nswipe up on the element \nlabeled '26' to view more \nproducts.\n\nswipe(26, \"up\", \n\"medium\")\nThought: To complete \nthe given task, the next \nstep I should do is to \ninitiate the search by \ntapping the search \nbutton on the \nkeyboard. \n\nAction: tap(24)\nThought: To complete \nthe given task, the next \nstep I should do is \nnavigate to the Costco \nwebsite. Since the \naddress \"costco.com\" is \nalready entered in the \naddress bar, I need to \ninitiate the web search \nor load the page by \ntapping the search \nbutton on the keyboard.\n\nAction: tap(4)\nGo to costco.com, search for “acer predator”, and select the first entry.\n...\nFigure 16: Failure of GPT-4V, with its thoughts and link-based actions given. This time the reason\nfor failure is misclick on the wrong button. The task is “Go to costco.com, search for “acer predator”,\nand select the first entry”. Notice that up until the fourth frame in this Figure, the trajectory goes\ncorrect. But then it clicks on the generic advertisements on the Costco.com website, and it cannot\nrecover back. It continues to scroll the page and takes wrong actions thereafter.\n22\n\n\nSet an alarm for 4pm.\nThought: To set an \nalarm for 4pm, I \nneed to find and \nopen the alarm or \nclock app. Since the \nalarm app icon is not \nimmediately visible \non the current \nscreen, I may need to \nswipe up or tap on \nthe app drawer icon \n(if present) to view \nmore apps and locate \nthe alarm or clock \napp. \n\nAction: swipe(1, \"up\", \n\"medium\")\nThought: To set an \nalarm for 4pm, I \nneed to open the \nClock app. The next \nstep is to tap on the \nClock app icon.\n\nAction: tap(25)\nThought: To set an \nalarm for 4pm, I \nneed to navigate to \nthe alarm section of \nthe Clock app if not \nalready there, and \nthen use the button \nto add a new alarm.\n\nAction: tap(1)\nThought: To set a \nnew alarm for 4 PM, \nI should tap the \nbutton to add a new \nalarm. \n\nAction: tap(14)\nThought: To \ncomplete the given \ntask, I need to set the \nalarm for 4:00 PM. \nThe hour is close to \nthe desired time, so I \nshould first switch \nfrom AM to PM using \nthe PM button.\n\nAction: tap(4)\nThought: To \ncomplete the given \ntask, I should set the \nminutes to \"00\" to \nmake the alarm for \n4:00 PM. \n\nAction: tap(1)\nFigure 17: Failure of GPT-4V, with an example task on the AiTW general test set. The task is “Set\nan alarm for 4pm”. Here, GPT-4V is able to successfully navigate to the clock app, and the alarm\nsettings of that app. However, it cannot take the correct precise actions to set the alarm quickly\nenough, and it fails due to maximum rounds reached. In the last round, notice that the action of tap(1)\ncontradict with its own thought process of setting minutes to “00”.\nSet-Of-Marks\nGPT4V\nSet-Of-Marks\nGemini-1.5-Pro\nAppAgent\nGPT4V\nAppAgent\nGemini-1.5-Pro\nAutoUI\nCogAgent\nFiltered BC\nOffline\nDigiRL\nOffline\nFiltered BC\nOnline\nDigiRL\nOnline\nSet-Of-Marks\nGPT4V\nSet-Of-Marks\nGemini-1.5-Pro\nAppAgent\nGPT4V\nAppAgent\nGemini-1.5-Pro\nAutoUI\nCogAgent\nFiltered BC\nOffline\nDigiRL\nOffline\nFiltered BC\nOnline\nDigiRL\nOnline\nGeneral\nWeb Shopping\nFail to recover from mistakes\nFail to click on the right link or fail to type\nFail to take reasonable attempts at all\nQuit or press HOME early\nStops at wrong but relevant page\nTechnical issues\nTask success\nFigure 18: Failure modes decomposition for each policy model for both General and Web Shopping\nsubsets.\n• Stops at wrong but relevant page refers to the failure mode where the agent arrives at a wrong\npage and mistakenly thinks that it had completed the task. For example, the agent finds a\nmacbook on costco.com while the instruction asked it to find a macbook on ebay.com.\n• Technical issues refer to the failure mode that either the task is impossible (e.g. the tasks\nasks to open Amazon app but this app is not installed) or the agent is temporarily blocked\nfrom a certain website due to frequent visits.\nThe translation between fine-grained failure modes and coarse-grained failure modes is presented in\nTable 5.\nE\nExperiment machines\nOur main experiments are conducted on VM instances from Google Cloud Platform. Each VM\ninstance comes with 1x Tesla T4 GPU and 16x Intel(R) Xeon(R) CPU.\n23\n\n\nFine-Grained Failure\nCoarse-Grained Failure\nFail to recover from mistakes\nFail to recover from mistakes\nFail to click on the right link or fail to type\nGet stuck midway\nFail to take reasonable attempts at all\nGet stuck midway\nQuit or Press HOME early\nArrive at wrong goal\nStops at wrong but relevant page\nArrive at wrong goal\nTechnical Issues\nNone\nTable 5: Examples of task descriptions in the AiTW Webshopping task set.\nhost machine\nworker machines\nemulators\naggregate \ntrajectories\ndistribute updated policy\nFigure 19: Multi-machine parallel emulator execution. The host machine is equipped with GPU\naccelerators and the worker machines are equipped only with CPUs. The policy update is executed on\nthe worker machine and the trajectory collections are executed distributedly on the worker machines\nand aggregated by the host machine.\nF\nSetup for parallel environment\nRunning multiple emulators in parallel can be challenging due to the inefficiency in thread syn-\nchronization and frequent fault propagation when one emulator runs into an unknown error. To\naddress this challenge, we set up a server-client system where all emulator processes are running in\nindependent server processes. Each emulator process communicates with the main training process\nthrough different UIAutomotor servers. The main training process sends high-level instructions to\nUIAutomotor servers (such as reset and step), while UIAutomotor servers parse high-level instruc-\ntions into low-level UI commands (such as typing a character and tapping at a coordinate) and such\nUI commands are executed by the emulator processes. When an exception is thrown in the emulator,\nthe UIAutomotor examines if it is recoverable (e.g. an UI command takes too long to execute in the\nemulator) and reset the emulator process if it is not. When an exception is thrown in the UIAutomotor\nserver, the main training process stops and resets the UIAutomotor server to ensure data correctness.\nThis design can easily be scaled up to a multi-machine setting. As illustrated in Figure 19, one host\nmachine equipped with GPU accelerator has a local copy of the current policy πt, and distributes\nthe policy to all worker machines equipped with only one GPU and multiple CPUs. Each worker\nmachine will then collect trajectories of different tasks using πt. After all collection processes are\nsynchronized, the host machine gathers all the trajectories together to update the policy to πt+1. This\nprocess keeps iterating until the policy converges.\nG\nAutonomous evaluator details\nOur autonomous evaluator gives a reward to each observation we get. The observation is composed\nof the current screenshot of device and the task. The evaluator gives a reward of 1 if the screenshot\nshows a completion of the task, and will terminate the POMDP as a result result.\nThe optimized prompt is shown in Figure 20 and Figure 21 for General and Web Shopping subsets\nrespectively.\n24\n\n\nPrompt\nYou're an expert in evaluating whether the Screenshot successfully completes the Task.\n\n=====Examples=====\nScreenshot: {train_1.png}\nTask: Open the settings.\nQ: What should I expect to see on the screenshot if I've opened the settings?\nA: I should expect to see I'm in the settings app. The screenshot shows the home screen of a mobile device, with \nvarious app icons displayed, including the settings app icon, but the settings app is not opened.\nStatus: failure\n\nScreenshot: {train_2.png}\nTask: Find hotels in washington dc\nQ: What should I expect to see on the screenshot if I've searched for hotels in Washington, DC?\nA: I should expect to see I'm in a search results page for hotels in Washington, DC. The screenshot shows a Google \nsearch page with the search field populated with the query \"hotels in washington dc\" and a list of suggested searches \nrelated to hotels in Washington, DC, but it does not show any search results for hotels in Washington, DC.\nStatus: failure\n\nScreenshot: {train_3.png}\nTask: What's a good restaurant in Portland?\nQ: What should I expect to see on the screenshot if I've searched for a good restaurant in Portland?\nA: I should expect to see I'm in a search results page for a good restaurant in Portland. The screenshot shows a \nGoogle search page with a search input field for \"good restaurant in portland\" and a map results preview showing \nbusiness locations near Portland, like \"Li Pigeon\", \"Portland City Grill\", and \"Higgins\",\nStatus: success\n\n... (more cases)\n\n=====Your Turn=====\nScreenshot: {test.png}\nTask: {task_this_traj}\nRespond in this format:\nQ: What should I expect to see on the screenshot if I've <repeat the task>?\nA: I should expect to see <first expectation, then what's in the given screenshot.>\nStatus: success or failure (don't return anything else)\nStart with \"Q:\".\n\nResponse\nQ: What should I expect to see on the screenshot if I've searched for the price of a 12' ladder at Home Depot?
\nA: I should expect to see the price of a 12' ladder at Home Depot; the screenshot shows a search result page for the p\nrice of a 12' ladder, with some product advertisements showing prices from Home Depot.
\nStatus: success\n\nImage Sources\ntrain_1.png\ntrain_2.png\ntrain_3.png\ntest.png\nFigure 20: Prompt for our autonomous evaluator for tasks in AitW General subset.\n25\n\n\nPrompt\nYou're an expert in evaluating whether the Screenshot successfully completes the Task.\n\n=====Examples=====\nScreenshot: {train_1.png}\nTask: Go to bestbuy.com\nQ: What should I expect to see on the screenshot if I've gone to bestbuy.com?\nA: I should expect to see I'm in the Best Buy website, which usually shows the best buy logo with some featured prod\nucts and categories. The screenshot shows I'm searching for \"bestbuy.com\" in the Google search (with some search s\nuggestions) instead of being in the Best Buy website.\nStatus: failure\n\nScreenshot: {train_2.png}\nTask: Go to ebay.com, search for \"corsair k70\"\nQ: What should I expect to see on the screenshot if I've gone to ebay.com AND searched for \"corsair k70\"?\nA: I should expect to see I'm in the eBay website and search results for \"corsair k70\". The screenshot shows I'm in the \neBay ebay website with some search suggestions for \"corsair k70\", but it does not show search results of the product, \nwhich usually includes price and the product details.\nStatus: failure\n\nScreenshot: {train_3.png}\nTask: Go to ebay.com, search for \"lenovo thinkpad\"\nQ: What should I expect to see on the screenshot if I've gone to ebay.com AND searched for \"lenovo thinkpad\"?\nA: I should expect to see I'm in the eBay website and search results for \"lenovo thinkpad\". The screenshot shows I'm \nin the eBay website and have several search results for \"lenovo thinkpad\".\nStatus: success\n\n... (more cases)\n\n=====Your Turn=====\nScreenshot: {test.png}\nTask: {task_this_traj}\nRespond in this format:\nQ: What should I expect to see on the screenshot if I've <repeat the task>?\nA: I should expect to see <first expectation, then what's in the given screenshot.>\nStatus: success or failure (don't return anything else)\nStart with \"Q:\".\n\nResponse\nQ: What should I expect to see on the screenshot if I've searched for the price of a 12' ladder at Home Depot?
\nA: I should expect to see the price of a 12' ladder at Home Depot; the screenshot shows a search result page for the p\nrice of a 12' ladder, with some product advertisements showing prices from Home Depot.
\nStatus: success\n\nImage Sources\ntrain_1.png\ntrain_2.png\ntrain_3.png\ntest.png\nFigure 21: Prompt for our autonomous evaluator for tasks in AitW Web Shopping subset.\n26\n\n\nH\nZero-shot Baseline Details\nFigure 22 shows the prompt that we used for testing the Set-of-Marks performance for GPT-4V and\nGemini 1.5 Pro. This prompt is directly taken from Yang et al. [47].\nPrompt\n\n\"You are an agent that is trained to perform some basic tasks on a smartphone. You will be given a \\nsmartphone \nscreenshot. The interactive UI elements on the screenshot are labeled with numeric tags starting from 1. The \n\\nnumeric tag of each interactive element is located in the center of the element.\\n\\nYou can call the following \nfunctions to control the smartphone:\\n\\n1. tap(element: int)\\nThis function is used to tap an UI element shown on \nthe smartphone screen.\\n\\\"element\\\" is a numeric tag assigned to an UI element shown on the smartphone screen.\n\\nA simple use case can be tap(5), which taps the UI element labeled with the number 5.\\n\\n2. text(text_input: \nstr)\\nThis function is used to insert text input in an input field/box. text_input is the string you want to insert and \nmust \\nbe wrapped with double quotation marks. A simple use case can be text(\\\"Hello, world!\\\"), which inserts the \nstring \\n\\\"Hello, world!\\\" into the input area on the smartphone screen. This function is usually callable when you \nsee a keyboard \\nshowing in the lower half of the screen.\\n\\n3. long_press(element: int)\\nThis function is used to \nlong press an UI element shown on the smartphone screen.\\n\\\"element\\\" is a numeric tag assigned to an UI element \nshown on the smartphone screen.\\nA simple use case can be long_press(5), which long presses the UI element \nlabeled with the number 5.\\n\\n4. swipe(element: int, direction: str, dist: str)\\nThis function is used to swipe an UI \nelement shown on the smartphone screen, usually a scroll view or a slide bar.\\n\\\"element\\\" is a numeric tag assigned \nto an UI element shown on the smartphone screen. \\\"direction\\\" is a string that \\nrepresents one of the four \ndirections: up, down, left, right. \\\"direction\\\" must be wrapped with double quotation \\nmarks. \\\"dist\\\" determines \nthe distance of the swipe and can be one of the three options: short, medium, long. You should \\nchoose the \nappropriate distance option according to your need.\\nA simple use case can be swipe(21, \\\"up\\\", \\\"medium\\\"), which \nswipes up the UI element labeled with the number 21 for a \\nmedium distance.\\n\\n5. grid()\\nYou should call this \nfunction when you find the element you want to interact with is not labeled with a numeric tag and \\nother \nelements with numeric tags cannot help with the task. The function will bring up a grid overlay to divide the \n\\nsmartphone screen into small areas and this will give you more freedom to choose any part of the screen to tap, \nlong \\npress, or swipe.\n\nThe task you need to complete is to How much does a 2 bedroom apartment rent for in Denver?. \n\nYour past actions to proceed with this task are summarized as follows: None\n\nNow, given the documentation and the following labeled screenshot, you need to think and call the function needed \nto proceed with the task. Your output should include three parts in the given format: \nObservation: <Describe what you observe in the image>\nThought: <To complete the given task, what is the next step I should do>\nAction: <The function call with the correct parameters to proceed with the task. When you are certain that the task \nis successfully done and the goal is reached as of the current observation, you should output FINISH. You cannot \noutput anything else except a function call or FINISH \\nin this field.>\nSummary: <Summarize your past actions along with your latest action in one or two sentences. Do not include the \nnumeric \\ntag in your summary>\\nYou can only take one action at a time, so please directly call the function.\"\nFigure 22: Set-of-Marks prompting. The boldened inputs can be changed according to our goal. The\ntask changes for every different task. The past actions change as we take actions (it is None now\nsince this is the prompt for the first round).\nI\nHyperparameters\nHyperparameters for both Filtered BC and DigiRL are carefully tuned through binary search on the\ntraining set of General and Web Shopping subsets. The final choice of hyperparameters for both\nmethods can be found in Table 6. As shown in the table, the only hyperparameters introduced by\nDigiRL are supervised training hyperparameters for the value function and instruction value function\n(including number of iterations and learning rate) and GAE λ.\n27\n\n\nTable 6: Hyperparameters for All Experiments\nMethod\nHyperparameter\nOffline\nOffline-to-Online\nFiltered\nBC\nactor lr\n3e-3\n3e-3\nbatch size\n128\n128\nrollout trajectories\n-\n16\nreplay buffer size\n-\n5000\nrollout temperature\n-\n1.0\nmaximum gradient norm\n0.01\n0.01\nactor updates per iteration\n20\n20\nnumber of iterations for offline actor updates\n10\n10\nDigiRL\nactor lr\n3e-3\n3e-3\nvalue function lr\n3e-3\n3e-3\ninstruction value function lr\n3e-3\n3e-3\ninstruction value function lr\n3e-3\n3e-3\nbatch size\n128\n128\nrollout trajectories\n-\n16\nreplay buffer size\n-\n5000\nrollout temperature\n-\n1.0\nmaximum gradient norm\n0.01\n0.01\nGAE λ\n0.5\n0.5\nactor updates per iteration\n20\n20\nvalue function updates per iteration\n5\n5\ninstruction value function updates per iteration\n-\n5\nnumber of iterations for offline actor updates\n10\n10\nnumber of iterations for offline value function updates\n20\n20\nnumber of iterations for offline instruction value function updates\n-\n20\nTable 7: Hyperparameters for DigiRL and Filtered BC on both General and Web Shopping subset of\nAitW..\n28","difficulty":"hard","domain":"Single-Document QA","length":"short","question":"Based on the passage, which of the following statements about the DigiRL framework's interaction with the emulator is correct?","sub_domain":"Academic"}

Source: https://huggingface.co/datasets/zai-org/LongBench-v2

initial import

Posting: /agents

GET /api/v1/write?intent=publish&task_id=4b6915f4-4f34-5707-8051-baeea918473b&body={url_encoded_text}&agent_name={optional_name}&nonce={optional_random_id}