# LongBench v2 / 66faac15bb02136c067c73f3

task_id: 92d2892e-ce0b-58a1-b42f-365a9eae8514
task_key: train--66faac15bb02136c067c73f3
task_revision_id: 3

{"choice_A":"Both AndroidArena and B-moca have designed two types of tasks: single app tasks and multi app tasks. AndroidArena also includes tasks that restrict the use of a specific app.","choice_B":"The testing method designed by B-moca is universal for text models, multimodal models, and video models, while Android Arena uses Deep-Q learning to train agents, which can be easily extended to multimodal models, although this article does not cover this aspect.","choice_C":"AndroidArena's test standards focus on operation sequences, while B-moca focuses on execution results on the simulator. Specifically, the Task Reward of AndroidArena calculates the LCS between the standard answer and the model operation sequence, and adds them up weighted.","choice_D":"Both AndroidArena and B-moca need to use real Android machines for testing.","context":"BENCHMARKING MOBILE DEVICE CONTROL AGENTS ACROSS DIVERSE CONFIGURATIONS\nJuyong Lee1\nTaywon Min2\nMinyong An3\nChangyeon Kim1\nKimin Lee1\n1KAIST\n2Seoul National University\n3Yonsei University\nABSTRACT\nDeveloping autonomous agents for mobile devices can significantly enhance user\ninteractions by offering increased efficiency and accessibility. However, despite\nthe growing interest in mobile device control agents, the absence of a commonly\nadopted benchmark makes it challenging to quantify scientific progress in this area.\nIn this work, we introduce B-MoCA: a novel benchmark designed specifically\nfor evaluating mobile device control agents. To create a realistic benchmark, we\ndevelop B-MoCA based on the Android operating system and define 60 common\ndaily tasks. Importantly, we incorporate a randomization feature that changes\nvarious aspects of mobile devices, including user interface layouts and language\nsettings, to assess generalization performance. We benchmark diverse agents,\nincluding agents employing large language models (LLMs) or multi-modal LLMs\nas well as agents trained from scratch using human expert demonstrations. While\nthese agents demonstrate proficiency in executing straightforward tasks, their\npoor performance on complex tasks highlights significant opportunities for future\nresearch to enhance their effectiveness. Our source code is publicly available at\nhttps://b-moca.github.io.\n1\nINTRODUCTION\nAutonomous agents controlling digital devices have great potential benefits. For example, these\nagents can improve the accessibility of user interactions, especially for users with physical disabilities\nor those facing challenges in operating devices, or boost productivity by automating tedious jobs.\nThis leads to increased interest in developing agents for mobile device control, and diverse approaches\nhave been introduced, including agents based on large language models (LLMs; Wen et al. 2023; Yan\net al. 2023) and agents trained with human demonstrations (Sun et al., 2022; Li et al., 2023), toward\nassistive agents that can understand the screen layout of the devices and manipulate the user interface\n(UI) to follow human instructions.\nDespite recent progress in developing mobile device control agents based on real systems, such as\nAndroid emulators (Toyama et al., 2021; Shvo et al., 2021; Zhang et al., 2023), prior works often\noverlook several important properties. One is testing generalization ability across diverse device\nconfigurations, which is crucial in deploying agents in real devices. Moreover, practical tasks essential\nfor life (such as creating an alarm or making emergency calls) are often neglected because of the\nchallenges in defining a wide range of practical tasks with robust success criteria in various device\nsettings. The lack of a unified benchmark encompassing these important properties has impeded\nscientific progress in this field.\nIn this work, we introduce B-MoCA: a Benchmark designed for evaluating Mobile device Control\nAgents across diverse configurations, based on Android emulators (see Figure 1). A key feature of\nB-MoCA is supporting numerous customization to mirror diverse device configurations,including\nvariations in icon placements, sizes, wallpapers, languages, and device types. Utilizing this feature,\nusers can easily create diverse environments with various configurations to evaluate generalization\nability. Additionally, we define 60 practical tasks grounded in realistic scenarios, such as opening\nspecific applications, initializing searches over the web, and adjusting device settings. To ensure\nreliable evaluation, B-MoCA provides rule-based success detectors, which are based on pre-defined\ntask completion criteria.\n1\narXiv:2404.16660v1  [cs.HC]  25 Apr 2024\n\n\nICLR 2024 Workshop on Generative Models for Decision Making\nMobile Device \nEnvironment\nDaily Tasks\nDiverse Device Setups\nMobile Device \nControl Agent\nLLM with\nText Action\nMLLM with \nText Action\nor\nAlgorithmic Designs\nVision-Language\nModel with\nUI-based Action \nor\nFigure 1: Illustration of B-MoCA. We present a realistic benchmark for assessing the performances\nof mobile device control agents in executing everyday tasks. To analyze generalization ability, we\nintroduce a randomization feature that changes various device attributes. We benchmark agents\nleveraging LLMs or MLLMs as well as agents with vision-language models trained from scratch.\nWe benchmark various methods for building mobile device control agents in B-MoCA. The baselines\ninclude agents employing text-only large language models (LLMs) or multi-modal LLMs (MLLMs),\nwhich benefit from extensive knowledge obtained through pre-training. We consider both closed-\nsource models, such as GPT-4 (Achiam et al., 2023) and Gemini (Gemini et al., 2023), and open-\nsource models, such as Llama 2 (Touvron et al., 2023) and Llama 3. Additionally, we train agents\nfrom scratch that directly interact with device UIs using behavior cloning (BC; Pomerleau 1988).\nIn our experiments, we find that the agents exhibit fundamental skills in mobile device control, such\nas solving straightforward tasks or completing tasks in training environments. However, they struggle\nin more challenging scenarios, such as handling more difficult tasks or generalizing to unseen device\nconfigurations. Specifically, the agents employing LLMs or MLLMs show high robustness across\ndiverse device configurations, while they fall short on multiple sequential decision-making. Agents\ntrained with BC, on the other hand, successfully mimic expert behaviors but lack generalization\nability in test environments with unseen device configurations. We study the effect of different design\nchoices on leveraging foundation models, including few-shot learning and the visual prompting\nmethod. We also analyze the effect of using pre-trained representation models or utilizing different\nnumbers of training device environments while training agents from scratch. Our extensive analyses\nreveal the limitations of existing methods in mobile device control, calling for future research.\nWe open-source all the source codes and relevant materials for easy reproduction of our environments\nand experiments. We hope B-MoCA helps future researchers identify challenges in building assistive\nagents and easily compare the efficacy of their methods over the prior work.\n2\nB-MOCA\nIn this section, we introduce B-MoCA: a benchmark designed to evaluate the performance of mobile\ndevice control agents on diverse device configurations in executing common daily tasks.\n2.1\nDESIGN FACTORS\nTo create a realistic benchmark for mobile device control agents, we build our benchmark based on\nAndroid, a widely used open-source operating system. In this benchmark, we frame device control\nas a sequential decision-making problem, reflecting the multi-step nature of the real interactions\n(Section 2.2). Designing a meaningful benchmark for mobile device control poses a significant\nchallenge, particularly in defining practical tasks like opening applications or adjusting device\nsettings. To address this, we consider 60 basic tasks that involve commonly used applications like\nChrome and Calendar, ensuring relevance to everyday life. Each task is equipped with a success\ndetector to evaluate the agent’s performance in accurately completing the task (Section 2.3).\n2\n\n\nICLR 2024 Workshop on Generative Models for Decision Making\nFigure 2: Examples of the home screen images from environments in B-MoCA. The randomized\nfeatures span icon location, font size, wallpaper, language, and device type and challenge the\ngeneralization ability of agents.\nGiven the diverse nature of user mobile device setups, such as variations in icon placements, wall-\npaper choices, languages, and device types, it is important to test the generalization abilities of\ndevice-control agents across diverse setups. To assess generalization performance, we incorporate\na randomization feature in our benchmark. This feature is designed to simulate various real-world\nscenarios by changing various aspects of mobile devices, such as user interface layouts and wallpapers\n(Section 2.4).\n2.2\nPROBLEM FORMULATION\nIn B-MoCA, we formulate the device management task as a sequential decision-making problem,\nwhere an agent interacts with an environment. Formally, given a task instruction c, an agent receives\nan observation ot and takes an action at based on its policy at ∼π(·|ot, c) at each timestep t. The\nenvironment (i.e., Android emulator) returns a success signal rt and the environment transitions to\nthe next observation ot+1.\nObservations, which capture the UI elements, can be represented as either screen pixels, screen\ndescriptions derived from the Android view hierarchy, or a combination of both. The action space\ncomprises a dual-gesture, similar to Rawles et al. (2023), which consists of a pair of (x, y) screen\nlocations for touch and lift. The dual-gesture action is identified as tapping the screen when\nthe two locations are identical within a specified threshold or swiping the screen when the distance\nbetween the two locations exceeds this threshold. Additionally, the agent can press navigation buttons\n(i.e., back, home, and overview) by touching the corresponding button locations on the screen. We\nnote that our benchmark supports text-based actions, enabling the utilization of the LLMs or MLLMs\n(see Section 3.1 for details).\nWe refer the readers for further details on the environment implementation to Appendix A.1.\n2.3\nDAILY TASKS\nOur B-MoCA includes 60 tasks essential for managing digital devices, providing functionalities\nuseful in daily routines. Each task is designed to be grounded in realistic situations, such as setting the\nalarm or enabling airplane mode. The tasks span various applications and require agents to interact\nwith diverse UI elements, such as application icons, checkboxes, toggle switches, input fields, and\nsliders. For a comprehensive list of tasks, we refer readers to Appendix B.1.\nTask completion is determined by a rule-based success detector implemented using Android Debug\nBridge (ADB). This success detector monitors logs from ADB and identifies the successful completion\nbased on pre-defined criteria. These criteria are established by examining ADB logs from human\ndemonstrations for each task and selecting the log produced when the target task is completed. With\nthe pre-defined criteria, then, the success detector automatically finds the matching regular expression\nin the ADB logs to signal the task completion. The success signal is with the value of +1 when the\ntask is completed, and 0 otherwise. An episode terminates as a success if the success detector signals\ncompletion, or as a failure if the agent exceeds a maximum step limit without meeting the criteria.\n3\n\n\nICLR 2024 Workshop on Generative Models for Decision Making\nTask instruction\nVision-Language-UI\nAgents\nLarge Language Model \nAgents\nMultimodal \nLarge Language Model\nAgents\nXML\nParser\nText observation\nText action\nAction\nConverter\nXML\nParser\nText action\nMultimodal \nobservation\nAction\nConverter\nImage observation\nDual-gesture action\nFigure 3: Illustration of baseline agents. LLM agents and MLLM agents interact with environments\nthrough additional XML parser and action converter, to obtain text descriptions and manipulate UIs\nwith text actions. VLUI agents directly leverage the UIs with screen images and dual-gesture actions.\n2.4\nENVIRONMENT RANDOMIZATION\nIn mobile device control, developing agents that can generalize across various device setups is crucial.\nTo evaluate their generalization ability, B-MoCA incorporates a randomization feature that changes\nicon placements and sizes, wallpapers, languages, and device types. Users can select the device type\nfrom a device list that includes popular models like Pixel 3, Pixel 4, Pixel 6, and WGXA Tablet. They\ncan also specify the locales to set the language and region, choose wallpapers from a selection of\ncustom images, and activate dark mode for further environmental variation. Moreover, the sizes of\nicons and text can vary between small, medium, and large. Lastly, applications can be randomly\nplaced on the home screen to simulate real-world usage patterns.\nUsing randomization features, we create 45 unique environments in B-MoCA, with examples shown\nin Figure 2. To assess the generalization ability, we divide the 45 distinct environments into two\nsets: 35 for training and 10 for testing. We employ domain randomization (Tobin et al., 2017) to\ntrain agents, enabling them to perform tasks robustly across diverse device configurations. We then\nevaluate the performance on test environments, which include unseen device setups. A detailed list of\nenvironment device configurations we prepare is available in Appendix A.2.\n3\nBASELINES\nIn this work, we benchmark various approaches for building mobile device control agents: LLM\nagents, MLLM agents, and Vision-Language-UI (VLUI) agents (see Figure 3). LLM agents and\nMLLM agents are developed using foundation models like LLMs and MLLMs, respectively (Sec-\ntion 3.1). VLUI agents, which consist of vision-language encoders, are trained from scratch using\nhuman expert demonstrations (Section 3.2).\n3.1\nLLM AGENTS AND MLLM AGENTS\nUtilizing foundation models such as LLMs and MLLMs, which contain extensive knowledge and have\nemergent capabilities, becomes a major direction in developing mobile device control agents (Wen\net al., 2023; Yan et al., 2023). In this work, we benchmark two types of agents that employ different\nfoundation models: LLMs (e.g., GPT-4) and MLLMs (e.g., GPT-4V). LLM agents utilize only the\ntext descriptions of the screen layout to generate text actions, while MLLM agents process both text\nand visual inputs.\nTo facilitate the interactions of LLM and MLLM agents with an Android emulator, we define an XML\nparser (Zhang et al., 2023; Yang et al., 2023b). This XML parser converts the UI elements, from the\nAndroid view hierarchy of the screen presented in XML format, into a list of text descriptions. The\ndescription includes the location of the bounding box, if necessary. Additionally, we define a set of\npossible action options, as detailed in Table 1, that can be converted into a corresponding dual-gesture\n4\n\n\nICLR 2024 Workshop on Generative Models for Decision Making\nAction option\nDescription\ndual-gesture(*)\nOperate a dual-gesture action\nwith arguments (*).\ntap(numeric tag)\nTap UI element labeled\nwith numeric tag.\nswipe(direction)\nSwipe to direction.\npress(\"HOME\")\nPress home button.\npress(\"BACK\")\nPress back button.\npress(\"OVERVIEW\")\nPress overview button.\nTable 1: A set of action options for text-based\nagents. Additional options are converted into\ncorresponding dual-gesture actions.\nRole: You are an agent that is trained to perform daily tasks\non digital devices, such as smartphones [...]\nAction space: You need to select an action option [...]\nGoal: [...]\n(Optional) Few-shot examples: [...]\nOutput format: Your output should follow the given format\n• Description: Describe what you observe in the input\n• Thought: To complete the given task, what is the next step\n• Action: The function call with the correct parameters\nObservation: [...]\n1\nFigure 4: An overview of prompt for the text-based\nagents, with abbreviated relevant information as\n[...]. The complete prompt is at Appendix C.1.\naction.1 These action options include tapping the UI element by choosing the numeric tags, swiping\nthe screen in pre-defined directions (up, down, left, right), and pressing the button with the names.\nWith these text-based observations and actions, we prompt the foundation models to explain the\nagents’ role, action space definition, goal, (optional) few-shot examples, and current observation. Our\nprompts, outlined in Figure 4, also incorporate the Chain-of-Thought technique (Wei et al., 2022) to\nenhance the reasoning ability of the agents by enforcing a certain output format.\n3.2\nVLUI AGENTS\nDespite the promising results of LLMs, leveraging these foundation models presents several chal-\nlenges such as the necessity of auxiliary interfaces or difficulties in fine-tuning. Thus, we also\ninvestigate another type of agent that can be trained from scratch: VLUI agents, named after the\nvision-language model with UI actions. Characterized by their direct interaction with device UIs in\na human-like manner, these agents can significantly benefit from the easy incorporation of human\ndemonstrations for training, potentially improving learning efficiency.\nTo be detailed, VLUI agents take a task instruction and screen images as the input and produce a\ndual-gesture action as the output. Input embeddings are extracted using vision and language encoders\nand a transformer (Vaswani et al., 2017) module is utilized to process these embeddings and generate\nthe dual-gesture actions. Specifically, we train a deterministic multi-task policy πθ(at|ot, c) using\nBC (Pomerleau 1988; Schaal 1996). The parameters θ of the policies are optimized to imitate the\nhuman expert demonstrations D = {(ot, a∗\nt , c)} by minimizing the following objective with mean\nsquared error function L(·):\nX\n(ot,a∗\nt ,c)∼D\nL(πθ(at|ot, c), a∗\nt ).\nWe refer readers to Appendix C.2 for more details on the architecture of VLUI agents.\n4\nEXPERIMENTS\nWe design our experiments to investigate the following research questions:\n• Can baseline agents perform daily tasks on mobile devices? (Section 4.2)\n• What are the distinctive characteristics of each agent? (Section 4.2)\n• What are the effects of different design choices for LLM or MLLM agents? (Section 4.3)\n• How crucial is the pre-training or training data diversity for VLUI agents? (Section 4.4)\n1To convert text actions to dual-gesture actions, we define the action converter. We analyze the efficacy of\nthe action options in Appendix E.1.\n5\n\n\n0\n30\n60\n90\nSuccess Rates (%)\nVLUI\nMLLM (GPT-4V)\nMLLM (Gemini-Pro-V)\nLLM (GPT-4)\nLLM (Gemini-Pro)\nAirplane\n0\n30\n60\n90\nSuccess Rates (%)\nAlarm1\n0\n30\n60\n90\nSuccess Rates (%)\nAlarm2\n0\n30\n60\n90\nSuccess Rates (%)\nVLUI\nMLLM (GPT-4V)\nMLLM (Gemini-Pro-V)\nLLM (GPT-4)\nLLM (Gemini-Pro)\nBrightness\n0\n30\n60\n90\nSuccess Rates (%)\nCall 911\n0\n30\n60\n90\nSuccess Rates (%)\nLanguage\nFigure 5: Average success rates of the baseline agents in the test environments. We report the mean\nand standard error across three runs. LLM agents are in three-shot learning, and MLLM agents are\nwithout SoM prompting and in one-shot learning for Gemini-Pro-V (due to maximum context\nlength) or three-shot learning for GPT-4V. The text-based agents with GPT-4 or GPT-4V show the\nbest performances on Airplane, Alarm1, and Birhgtness, while VLUI agents show better\nperformances on Alarm2, Call 911, and Language.\n4.1\nEXPERIMENTAL SETUP\nIn our experiments, we evaluate LLM agents, MLLM agents, and VLUI agents using six representa-\ntive tasks: named, Airplane, Alarm1, Alarm2, Brightness, Call 911, and Language.\nThese tasks are selected to cover navigating multiple pages in target applications and manipulating\ndiverse UI elements which vary in configuration across different device settings. For example, on\nAlarm2, the agents need to reach the alarm tab in the clock application and adapt to varying shapes\nof clock UI in a shape of either rectangle or circle with different size options. We display exemplary\nexpert demonstrations on these tasks in Appendix B.2. For each task, the task instruction is as follows:\n• Airplane: “turn on airplane mode”\n• Alarm1: “turn on alarm at 9 am”\n• Alarm2: “create an alarm at 10:30 am”\n• Brightness: “decrease the screen brightness in setting”\n• Call 911: “call 911”\n• Language: “go to the ‘add a language’ page in setting”\nFor LLM agents, we employ the closed-source models Gemini-Pro (Gemini et al., 2023) and GPT-4\n(GPT-4-0125-preview; Achiam et al. 2023).2 We study LLM agents with both zero-shot\nand few-shot learning cases. For few-shot learning, we sample examples from 210 human expert\ndemonstrations (see Appendix D.1 for dataset collection). For MLLM agents, we leverage Gemini-\nPro-V and GPT-4V (GPT-4-vision-preview). We report MLLM agents in only few-shot\nlearning and investigate visually grounding the agents with Set-of-Mark (SoM) prompting (Yang\net al., 2023a). We provide more details on the configurations for LLM and MLLM Agents in\nAppendix C.3. For VLUI agents, we train multi-task policies where each policy performs all six tasks.\nThe policies are trained with BC using the 210 human expert demonstrations.3 We refer the readers\nto Appendix C.4 for more details on the training procedures of VLUI agents.\nFor each evaluation, we measure the success rates of the agents in the 10 test environments and\ncompute the average success rates. These success rates are automatically computed by the rule-based\nsuccess detector. We report the mean and standard error across three different runs.\n4.2\nMAIN RESULTS\nFigure 5 shows the success rates of LLM agents, MLLM agents, and VLUI agents in test envi-\nronments. LLM agents and MLLM agents utilize their pre-trained base knowledge and few-shot\n2We include experiments with open-source models of Llama 2 (Touvron et al., 2023), Llama 3, and\nAgentLM (Zeng et al., 2023) in Appendix E.2.\n3We also include experimental results of VLUI agents trained with offline reinforcement learning by employ-\ning the success signals as rewards in Appendix E.3.\n6\n\n\nICLR 2024 Workshop on Generative Models for Decision Making\n(a)\n(b)\n(c)\nExecuted \nAction\nDesired\nAction\nFigure 6: The common failure modes of the agents.\n(a) LLM agents fail to complete sequential steps,\n(b) MLLM agents miss details in the images, and\n(c) VLUI agents tap the wrong icon locations.\nLLM\n(zero-shot)\nLLM\n(few-shot)\nMLLM\n(w/o SoM)\nMLLM\n(w/ SoM)\nAirplane\n53 ± 03\n73 ± 12\n80 ± 06\n83 ± 03\nAlarm1\n42 ± 13\n67 ± 03\n60 ± 15\n62 ± 09\nAlarm2\n00 ± 00\n00 ± 00\n23 ± 03\n17 ± 03\nBrightness\n73 ± 12\n73 ± 09\n87 ± 03\n83 ± 03\nCall 911\n00 ± 00\n03 ± 03\n53 ± 03\n33 ± 09\nLanguage\n27 ± 06\n43 ± 09\n43 ± 09\n47 ± 17\nTable 2: Success rates of text-based agents with\ndifferent prompting methods. While few-shot\nexamples help LLM agents with GPT-4, we ob-\nserve no significant gain from SoM prompting\nfor MLLM agents with GPT-4V.\nexamples to complete simple tasks with high performances (e.g., more than 70% on Airplane and\nBrightness with GPT-4 or GPT-4V), but their success rates significantly drop as the tasks become\ncomplex (e.g., less than 30% on Alarm2 even with GPT-4 or GPT-4V). VLUI agents, on the other\nhand, imitate the behaviors of experts and exhibit average success rates of higher than 50% on all\ntasks, except 47% on Alarm2. However, all methods still show low performances (less than 60%)\non complex tasks (i.e., Alarm2 and Call 911), which calls for new algorithms.\nWe provide more remarks on each agent type below.\nRobustness of LLM agents and MLLM agents\nBoth types of agents employing foundation\nmodels have shown robust performances in diverse device configurations. It is straightforward that\nthese agents are robust to the randomization over the visual appearances, such as icon locations or\nfont size, as the locations of the UI elements are described in the Android view hierarchy. In addition,\nLLM agents with both Gemini-Pro and GPT-4 are robust to language changes, with descriptions\nof UI elements in different languages. Particularly, these agents generalize to languages in test\nenvironments, e.g., Korean and Egyptian Arabic, which are not included in the few-shot examples.\nRemaining challenges for LLM agents\nWhile exhibiting robust performances across diverse\ndevice settings, several limitations of LLM agents are observed. First, the agents face difficulties with\nlong-horizon tasks, which require completing a precise sequence of multiple actions. For example,\non Call 911, the agents often make mistakes while typing the sequence of 9-1-1, as shown in\nFigure 6(a). Second, the agents struggle to leverage few-shot examples adaptively. For instance,\non Brightness, we observe LLM agents naively copying the few-shot examples from different\ndevice configurations without adjusting them to the current environment.\nEfficacy of multi-modal input for MLLM agents\nWe confirm the effectiveness of image input\nwith MLLM agents employing GPT-4V, as large increases in success rates are observed on Alarm2\nand Call 911 compared to LLM agents with GPT-4. However, MLLM agents share the challenges\nof LLM agents in accurately executing complex tasks. Moreover, they still fall short in understanding\ndetails of visual input, such as the small interface for setting AM/PM on Alarm2 as shown in\nFigure 6(b). MLLM agents with Gemini-Pro-V show significantly lower performances than LLM\nagents with Gemini-Pro, assumably due to the longer context length of multi-modal inputs. These\nresults indicate the remaining headroom in leveraging multi-modal inputs more efficiently.\nGeneralization ability of VLUI agents\nWe observe training VLUI agents with BC can lead to\nhigh performances on many complex tasks where MLLM agents fail. These agents perform robustly\nto unseen wallpapers, as being trained with multiple different background images. Also, they can\ngeneralize their actions to unseen devices, e.g., Pixel 4, even though they are trained only on a single\ndevice type, i.e., Pixel 3. However, VLUI agents begin failing to complete the tasks with severe\nvisual changes induced by unseen device configurations. While they exhibit higher than 90% success\nrates in training environments, the performance degrades to less than 70% in test environments\n(see Appendix D.2 for more details). Specifically, they suffer from handling unseen locations of UI\nelements, as shown in Figure 6(c). We believe these findings reveal the importance of diversity in\ntraining data from randomized environments (see Section 4.4 for more discussions).\n7\n\n\nICLR 2024 Workshop on Generative Models for Decision Making\nAirplane\nAlarm1\nAlarm2\nBrightness\nCall911\nLanguage\n10\n40\n70\nSuccess Rates (%)\n Randomly initialized\n Pre-trained\nFigure 7: Success rates of VLUI agents with vi-\nsual encoders randomly initialized or pre-trained.\nPre-training helps the performances of the agents.\nAirplane\nAlarm1\nAlarm2\nBrightness\nCall911\nLanguage\n10\n40\n70\nSuccess Rates (%)\n 7 Envs\n 21 Envs\n 35 Envs\nFigure 8: Success rates of VLUI agents with vary-\ning numbers of training environments. The suc-\ncess rates escalate with more environments.\n4.3\nINVESTIGATIONS ON DESIGN CHOICES FOR LLM AGENTS AND MLLM AGENTS\nThe performance of LLM agents heavily relies on how the input prompts are tailored. Considering\nonly the leaf UI elements of Android view hierarchy to describe the screen layout, similar to prior\nwork (Li et al., 2020; Yang et al., 2023b) for example, might result in meaningless descriptions in\ncertain applications (e.g., the setting application on Airplane and Language). In this work, we\nhave leveraged the text attributes of all the available nodes to avoid such collapse, while we believe\nthere can be more simple yet expressive representation methods.\nIn addition, we observe that few-shot examples can significantly improve the performance of LLM\nagents with GPT-4 compared to zero-shot cases. As shown in Table 2, equipping prompt with\nfew-shot examples improves the performance from 42% to 67% on Alarm1 and from 27% to 43%\non Language. However, employing few-shot examples does not always help agents, as shown on\nAlarm2 or Brightness. We note that naive exploitation of expert demonstrations might lead to\nexcessive increases in computational cost and highlight the necessity of efficient few-shot prompting.\nMoreover, we investigate the effect of common visual prompting methods for MLLM agents with\nGPT-4V. To enhance the visual grounding ability of MLLMs, prior studies (Yan et al., 2023; Yang\net al., 2023b) have actively adopted SoM prompting, where each UI element in the input image is\nmarked with numeric tags. However, we find that SoM prompting can often significantly degrade the\nperformance of MLLM agents on Alarm2 and Call 911 as shown in Table 2. We hypothesize\nthat the numeric tags may cause confusion when overlaid on UI elements with numbers, such as dial\nbuttons or clock interfaces. For examples of the inputs used in SoM prompting, see Appendix D.3.\n4.4\nEFFECTS OF PRE-TRAINED ENCODERS AND DATA DIVERSITY FOR VLUI AGENTS\nThe main challenge of VLUI agents is the lack of generalization ability as mentioned in Section 4.2.\nHence, we examine the different algorithmic designs for the representation model of VLUI agents and\nthe effects of training diversity on performance robustness. We also include an additional experiment\nwith varying model sizes of visual encoders in Appendix E.4.\nFirst, we compare VLUI agents in two different designs: visual encoders with parameters randomly\ninitialized and visual encoders pre-trained with ImageNet (Krizhevsky et al., 2017). As shown in\nFigure 7, we observe significant improvements in success rates with pre-training, e.g., from 37% to\n63% on Language. These results demonstrate the benefit of employing pre-trained representation\nmodels, and we expect further improvements can be induced by leveraging more Android-specific\nimages for pre-training (Sun et al., 2022; Rawles et al., 2023).\nFurthermore, we train VLUI agents by progressively increasing the number of training environments\n(see Appendix D.1 for more details of the experiment setting). As shown in Figure 8, as the number\nof training environments increases, the performance of VLUI agents escalates. Specifically, the\nagents exhibit success rates of 20%, 40%, and 63% on Language with the number of training\nenvironments 7, 21, and 35, respectively. We believe this verifies the efficacy of the environment\nrandomization feature incorporated in our benchmark toward more practical agents.\n8\n\n\nICLR 2024 Workshop on Generative Models for Decision Making\n5\nRELATED WORK\nFoundation models for decision-making system\nInspired by the strong emergent properties of\nfoundation models (Brown et al., 2020; Wei et al., 2022), many researches have adopted LLMs to\ndevelop decision-making system (Yao et al., 2023; Shinn et al., 2023). In robot learning, for example,\nLLMs have been widely equipped for reasoning, planning, manipulation, and navigation (Driess et al.,\n2023; Liang et al., 2023; Huang et al., 2023). Furthermore, agents with LLMs have shown capabilities\nof performing interesting tasks in numerous simulated worlds, including game environments (Wang\net al., 2023; Tan et al., 2024) and virtual reality (Qian et al., 2023; Yang et al., 2024). In recent days,\nfocusing on practicalness, solving computer tasks with foundation models has also been actively\nexplored (Nakano et al., 2021; Furuta et al., 2023). We further study the abilities of foundation\nmodels to control mobile devices toward assistive agents in real life.\nDeveloping assistive agent for device control\nFor agents that effectively understand and manipu-\nlate the UI elements, a large body of work has leveraged the structural information, such as document\nobject model in HTML or Android view hierarchy (Branavan et al., 2010; Gur et al., 2019). In\naddition, methods for equipping agents with the ability to understand information-rich screen images\nhave been widely investigated, mainly with vision-based reinforcement learning (Liu et al., 2018;\nHumphreys et al., 2022; Shaw et al., 2023). Recently, diverse strategies to build device control\nagents with foundation models are introduced, including prompting methods (Wen et al., 2023; Kim\net al., 2023), instruction-tuning (Furuta et al., 2023), fine-tuning with images (Zhan & Zhang, 2023;\nHong et al., 2023), and visual prompting (Yan et al., 2023; Yang et al., 2023b). Here, we present an\nelaborate analysis of the main methods for building mobile device control agents.\nBenchmark for decision-making agents\nThere have been continuous efforts to build reliable bench-\nmarks for sequential decision-making in video games (Bellemare et al., 2013), locomotion (Brockman\net al., 2016), and robotic manipulation (James et al., 2020). Lately, researchers have proposed bench-\nmarks for solving device control tasks, viewing it as another decision-making problem. For example,\nYao et al. (2022) and Zhou et al. (2024) have presented benchmark simulating web platforms, while\nToyama et al. (2021), Shvo et al. (2021), and Zhang et al. (2023) have suggested RL environments\nadopting Android emulators. In this work, inspired by special-purpose benchmarks quantifying the\nrobustness of the agents (Cobbe et al., 2020; Stone et al., 2021), we newly propose a benchmark with\nthe randomization feature.\n6\nDISCUSSION & CONCLUSION\nWe present B-MoCA, a new benchmark designed for evaluating mobile device control agents. Our\nbenchmark provides diverse tasks applicable to everyday routines and environments that simulate\nnumerous device configurations. We conduct extensive experiments and demonstrate that B-MoCA\ncan serve as a standardized platform for developing different types of agents in a unified setting.\nFinally, we mention several limitations and promising future directions of this work:\n• Tasks with text typing While we define the action spaces with dual-gesture actions, text typing\nby touching the soft keyboard demands excessively long interactions. In the future, we plan\nto include tasks requiring text typing, such as web search or e-mail sending, with advanced\ninterfaces.\n• Open-ended tasks and reward modeling Since the ADB-based success detector does not capture\nthe semantics of agent behaviors, tasks with ambiguous success criteria are hard to evaluate.\nAlternatively, we believe employing the reward model learned from demonstrations (Fan et al.,\n2022) can be used for integrating open-ended tasks.\n• More on LLM agents Foundation models can be employed in different ways, such as using them\nas a high-level planner to operate a set of pre-defined APIs (Chen & Li, 2024) or neural network\npolicies (Ahn et al., 2022) as low-level actors. Also, as training VLUI agents with demonstrations\nresults in high performances, fine-tuning LLMs is highly promising.\nToward practical mobile device control agents, we hope that B-MoCA stands as a valuable platform\nwith helpful resources for future innovations.\n9\n\n\nICLR 2024 Workshop on Generative Models for Decision Making\nIMPACT STATEMENT\nThis study proposes a benchmark designed to assess interactive mobile device management agents,\nwith social opportunities to enhance user accessibility and aid those facing disabilities. We caution\nusers about privacy concerns while we try to eliminate such potentials during task designs. Noting\nthe importance of research for preventing malicious usages of device control agents, we emphasize\nB-MoCA as a useful test bed.\nACKNOWLEDGMENTS\nWe thank Dongjun Lee, Kyuyoung Kim, and Ahjeong Seo for providing sincere suggestions for\nimproving our work. This work was supported by Institute for Information & communications\nTechnology Promotion (IITP) grant funded by the Korea government (MSIT) (No.2019-0-00075\nArtificial Intelligence Graduate School Program (KAIST)).\nREFERENCES\nJosh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Aleman,\nDiogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, et al. Gpt-4 technical report.\narXiv preprint arXiv:2303.08774, 2023.\nMichael Ahn, Anthony Brohan, Noah Brown, Yevgen Chebotar, Omar Cortes, Byron David, Chelsea\nFinn, Chuyuan Fu, Keerthana Gopalakrishnan, Karol Hausman, et al. Do as i can, not as i say:\nGrounding language in robotic affordances. In The Conference on Robot Learning, 2022.\nM. G. Bellemare, Y. Naddaf, J. Veness, and M. Bowling. The arcade learning environment: An\nevaluation platform for general agents. Journal of Artificial Intelligence Research, 47:253–279,\n2013.\nSRK Branavan, Luke Zettlemoyer, and Regina Barzilay. Reading between the lines: Learning to map\nhigh-level instructions to commands. In Association for Computational Linguistics, 2010.\nGreg Brockman, Vicki Cheung, Ludwig Pettersson, Jonas Schneider, John Schulman, Jie Tang, and\nWojciech Zaremba. Openai gym. arXiv preprint arXiv:1606.01540, 2016.\nTom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal,\nArvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language models are\nfew-shot learners. In Conference on Neural Information Processing Systems, 2020.\nWei Chen and Zhiyuan Li. Octopus v2: On-device language model for super agent. arXiv preprint\narXiv:2404.01744, 2024.\nKarl Cobbe, Chris Hesse, Jacob Hilton, and John Schulman. Leveraging procedural generation to\nbenchmark reinforcement learning. In International conference on machine learning, 2020.\nDanny Driess, Fei Xia, Mehdi SM Sajjadi, Corey Lynch, Aakanksha Chowdhery, Brian Ichter, Ayzaan\nWahid, Jonathan Tompson, Quan Vuong, Tianhe Yu, et al. Palm-e: An embodied multimodal\nlanguage model. International Conference on Machine Learning, 2023.\nLinxi Fan, Guanzhi Wang, Yunfan Jiang, Ajay Mandlekar, Yuncong Yang, Haoyi Zhu, Andrew Tang,\nDe-An Huang, Yuke Zhu, and Anima Anandkumar. Minedojo: Building open-ended embodied\nagents with internet-scale knowledge. In Conference on Neural Information Processing Systems,\n2022.\nHiroki Furuta, Ofir Nachum, Kuang-Huei Lee, Yutaka Matsuo, Shixiang Shane Gu, and Izzeddin Gur.\nInstruction-finetuned foundation models for multimodal web navigation. In International Confer-\nence on Learning Representations 2023 Workshop on Mathematical and Empirical Understanding\nof Foundation Models, 2023.\nTeam Gemini, Rohan Anil, Sebastian Borgeaud, Yonghui Wu, Jean-Baptiste Alayrac, Jiahui Yu, Radu\nSoricut, Johan Schalkwyk, Andrew M Dai, Anja Hauth, et al. Gemini: a family of highly capable\nmultimodal models. arXiv preprint arXiv:2312.11805, 2023.\n10\n\n\nICLR 2024 Workshop on Generative Models for Decision Making\nIzzeddin Gur, Ulrich Rueckert, Aleksandra Faust, and Dilek Hakkani-Tur. Learning to navigate the\nweb. In International Conference on Learning Representations, 2019.\nWenyi Hong, Weihan Wang, Qingsong Lv, Jiazheng Xu, Wenmeng Yu, Junhui Ji, Yan Wang, Zihan\nWang, Yuxiao Dong, Ming Ding, et al. Cogagent: A visual language model for gui agents. arXiv\npreprint arXiv:2312.08914, 2023.\nChenguang Huang, Oier Mees, Andy Zeng, and Wolfram Burgard. Visual language maps for robot\nnavigation. In International Conference on Robotics and Automation, 2023.\nPeter C Humphreys, David Raposo, Tobias Pohlen, Gregory Thornton, Rachita Chhaparia, Alistair\nMuldal, Josh Abramson, Petko Georgiev, Adam Santoro, and Timothy Lillicrap. A data-driven\napproach for learning to control computers. In International Conference on Machine Learning,\n2022.\nStephen James, Zicong Ma, David Rovick Arrojo, and Andrew J. Davison. Rlbench: The robot\nlearning benchmark & learning environment. IEEE Robotics and Automation Letters, 2020.\nGeunwoo Kim, Pierre Baldi, and Stephen McAleer. Language models can solve computer tasks.\nConference on Neural Information Processing Systems, 2023.\nDiederik P. Kingma and Jimmy Ba. Adam: A method for stochastic optimization. In International\nConference for Learning Representations, 2017.\nIlya Kostrikov, Ashvin Nair, and Sergey Levine. Offline reinforcement learning with implicit\nq-learning. In International Conference on Learning Representations, 2022.\nAlex Krizhevsky, Ilya Sutskever, and Geoffrey E Hinton. Imagenet classification with deep convolu-\ntional neural networks. Communications of the ACM, 60(6):84–90, 2017.\nSascha Lange, Thomas Gabel, and Martin Riedmiller. Batch reinforcement learning. In Reinforcement\nlearning: State-of-the-art, pp. 45–73. Springer, 2012.\nSergey Levine, Aviral Kumar, George Tucker, and Justin Fu. Offline reinforcement learning: Tutorial,\nreview, and perspectives on open problems. arXiv preprint arXiv:2005.01643, 2020.\nWei Li, Fu-Lin Hsu, Will Bishop, Folawiyo Campbell-Ajala, Oriana Riva, and Max Lin. Uinav: A\nmaker of ui automation agents. arXiv preprint arXiv:2312.10170, 2023.\nYang Li, Gang Li, Luheng He, Jingjie Zheng, Hong Li, and Zhiwei Guan. Widget captioning:\nGenerating natural language description for mobile user interface elements. In Conference on\nEmpirical Methods in Natural Language Processing, 2020.\nJacky Liang, Wenlong Huang, Fei Xia, Peng Xu, Karol Hausman, Brian Ichter, Pete Florence, and\nAndy Zeng. Code as policies: Language model programs for embodied control. In International\nConference on Robotics and Automation, 2023.\nEvan Zheran Liu, Kelvin Guu, Panupong Pasupat, Tianlin Shi, and Percy Liang. Reinforcement\nlearning on web interfaces using workflow-guided exploration. In International Conference on\nLearning Representations, 2018.\nHaotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee. Visual instruction tuning. In Conference\non Neural Information Processing Systems, 2023.\nAshvin Nair, Abhishek Gupta, Murtaza Dalal, and Sergey Levine. Awac: Accelerating online\nreinforcement learning with offline datasets. arXiv preprint arXiv:2006.09359, 2020.\nReiichiro Nakano, Jacob Hilton, Suchir Balaji, Jeff Wu, Long Ouyang, Christina Kim, Christopher\nHesse, Shantanu Jain, Vineet Kosaraju, William Saunders, et al. Webgpt: Browser-assisted\nquestion-answering with human feedback. arXiv preprint arXiv:2112.09332, 2021.\nJan Peters and Stefan Schaal. Reinforcement learning by reward-weighted regression for operational\nspace control. In International Conference on Machine learning, 2007.\n11\n\n\nICLR 2024 Workshop on Generative Models for Decision Making\nDean A Pomerleau. Alvinn: An autonomous land vehicle in a neural network. In Conference on\nNeural Information Processing Systems, 1988.\nChen Qian, Xin Cong, Wei Liu, Cheng Yang, Weize Chen, Yusheng Su, Yufan Dang, Jiahao Li,\nJuyuan Xu, Dahai Li, Zhiyuan Liu, and Maosong Sun. Communicative agents for software\ndevelopment. arXiv preprint arXiv:2307.07924, 2023.\nChristopher Rawles, Alice Li, Daniel Rodriguez, Oriana Riva, and Timothy P Lillicrap.\nAn-\ndroidinthewild: A large-scale dataset for android device control.\nIn Conference on Neural\nInformation Processing Systems Datasets and Benchmarks Track, 2023.\nStefan Schaal. Learning from demonstration. Conference on Neural Information Processing Systems,\n1996.\nPeter Shaw, Mandar Joshi, James Cohan, Jonathan Berant, Panupong Pasupat, Hexiang Hu, Urvashi\nKhandelwal, Kenton Lee, and Kristina Toutanova. From pixels to ui actions: Learning to follow\ninstructions via graphical user interfaces. In Conference on Neural Information Processing Systems,\n2023.\nNoah Shinn, Beck Labash, and Ashwin Gopinath. Reflexion: an autonomous agent with dynamic\nmemory and self-reflection. arXiv preprint arXiv:2303.11366, 2023.\nMaayan Shvo, Zhiming Hu, Rodrigo Toro Icarte, Iqbal Mohomed, Allan D. Jepson, and Sheila A.\nMcIlraith. Appbuddy: Learning to accomplish tasks in mobile apps via reinforcement learning. In\nCanadian Conference on Artificial Intelligence, 2021.\nAustin Stone, Oscar Ramirez, Kurt Konolige, and Rico Jonschkowski. The distracting control suite–a\nchallenging benchmark for reinforcement learning from pixels. arXiv preprint arXiv:2101.02722,\n2021.\nLiangtai Sun, Xingyu Chen, Lu Chen, Tianle Dai, Zichen Zhu, and Kai Yu. Meta-gui: Towards\nmulti-modal conversational agents on mobile gui. Conference on Empirical Methods in Natural\nLanguage Processing, 2022.\nMingxing Tan and Quoc V. Le. Efficientnet: Rethinking model scaling for convolutional neural\nnetworks. In International Conference on Machine Learning, 2019.\nWeihao Tan, Ziluo Ding, Wentao Zhang, Boyu Li, Bohan Zhou, Junpeng Yue, Haochong Xia, Jiechuan\nJiang, Longtao Zheng, Xinrun Xu, et al. Towards general computer control: A multimodal agent\nfor red dead redemption ii as a case study. arXiv preprint arXiv:2403.03186, 2024.\nJosh Tobin, Rachel Fong, Alex Ray, Jonas Schneider, Wojciech Zaremba, and Pieter Abbeel. Do-\nmain randomization for transferring deep neural networks from simulation to the real world. In\nInternational Conference on Intelligent Robots and Systems, 2017.\nHugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay\nBashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, et al. Llama 2: Open foundation\nand fine-tuned chat models. arXiv preprint arXiv:2307.09288, 2023.\nDaniel Toyama, Philippe Hamel, Anita Gergely, Gheorghe Comanici, Amelia Glaese, Zafarali Ahmed,\nTyler Jackson, Shibl Mourad, and Doina Precup. Androidenv: A reinforcement learning platform\nfor android. arXiv preprint arXiv:2105.13231, 2021.\nAshish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz\nKaiser, and Illia Polosukhin. Attention is all you need. In Conference on Neural Information\nProcessing Systems, 2017.\nGuanzhi Wang, Yuqi Xie, Yunfan Jiang, Ajay Mandlekar, Chaowei Xiao, Yuke Zhu, Linxi Fan, and\nAnima Anandkumar. Voyager: An open-ended embodied agent with large language models. In\nConference on Neural Information Processing Systems, 2023.\nJason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Fei Xia, Ed Chi, Quoc V Le, Denny\nZhou, et al. Chain-of-thought prompting elicits reasoning in large language models. In Conference\non Neural Information Processing Systems, 2022.\n12\n\n\nICLR 2024 Workshop on Generative Models for Decision Making\nHao Wen, Yuanchun Li, Guohong Liu, Shanhui Zhao, Tao Yu, Toby Jia-Jun Li, Shiqi Jiang, Yunhao\nLiu, Yaqin Zhang, and Yunxin Liu. Empowering llm to use smartphone for intelligent task\nautomation. arXiv preprint arXiv:2308.15272, 2023.\nAn Yan, Zhengyuan Yang, Wanrong Zhu, Kevin Lin, Linjie Li, Jianfeng Wang, Jianwei Yang, Yiwu\nZhong, Julian McAuley, Jianfeng Gao, et al. Gpt-4v in wonderland: Large multimodal models for\nzero-shot smartphone gui navigation. arXiv preprint arXiv:2311.07562, 2023.\nJianwei Yang, Hao Zhang, Feng Li, Xueyan Zou, Chunyuan Li, and Jianfeng Gao. Set-of-mark\nprompting unleashes extraordinary visual grounding in gpt-4v. arXiv preprint arXiv:2310.11441,\n2023a.\nJihan Yang, Runyu Ding, Ellis Brown, Xiaojuan Qi, and Saining Xie. V-irl: Grounding virtual\nintelligence in real life. arXiv preprint arXiv:2402.03310, 2024.\nZhao Yang, Jiaxuan Liu, Yucheng Han, Xin Chen, Zebiao Huang, Bin Fu, and Gang Yu. Appagent:\nMultimodal agents as smartphone users. arXiv preprint arXiv:2312.13771, 2023b.\nShunyu Yao, Howard Chen, John Yang, and Karthik Narasimhan. Webshop: Towards scalable\nreal-world web interaction with grounded language agents. Conference on Neural Information\nProcessing Systems, 2022.\nShunyu Yao, Jeffrey Zhao, Dian Yu, Nan Du, Izhak Shafran, Karthik Narasimhan, and Yuan Cao.\nReact: Synergizing reasoning and acting in language models. In International Conference on\nLearning Representations, 2023.\nAohan Zeng, Mingdao Liu, Rui Lu, Bowen Wang, Xiao Liu, Yuxiao Dong, and Jie Tang. Agenttuning:\nEnabling generalized agent abilities for llms. arXiv preprint arXiv:2310.12823, 2023.\nZhuosheng Zhan and Aston Zhang. You only look at screens: Multimodal chain-of-action agents.\narXiv preprint arXiv:2309.11436, 2023.\nDanyang Zhang, Lu Chen, and Kai Yu. Mobile-env: A universal platform for training and evaluation\nof mobile interaction. arXiv preprint arXiv:2305.08144, 2023.\nShuyan Zhou, Frank F Xu, Hao Zhu, Xuhui Zhou, Robert Lo, Abishek Sridhar, Xianyi Cheng,\nYonatan Bisk, Daniel Fried, Uri Alon, et al. Webarena: A realistic web environment for building\nautonomous agents. In International Conference on Learning Representations, 2024.\n13\n\n\nICLR 2024 Workshop on Generative Models for Decision Making\nAppendix:\nBenchmarking Mobile Device Control Agent across Diverse Configurations\nA\nENVIRONMENT DETAILS\nA.1\nENVIRONMENT IMPLEMENTATION AND INTERFACE\nEnvironment\nB-MoCA is based on Android OS for real-system interactive evaluation. The\nenvironment is simulated with Android virtual devices, containing the device hardware profile,\nsystem image, storage area, and other relevant properties. The dynamics of the environments, such as\nthe transition rules, are governed by Android OS and applications.\nEach environment is represented as an Android device, running on top of the Android emulator.\nTo be more specific, we define each environment as a snapshot, a stored image of the Android\nvirtual device. Each snapshot is built by saving an image of the target device after the configurations.\nThese configurations include randomizing the features of environments by placing icons in random\nlocations, setting dots per inch (DPI), modifying wallpapers, and changing the language. In addition,\nour configuration process includes adjusting several device settings for accurate evaluation, such as\nchanging the database of applications.\nTo facilitate interactions between the environment and agents, we develop a set of interfaces. These\ninterfaces encompass various functionalities: to provide the task descriptions in text to the agents, to\ncapture screenshots of the virtual device, to provide the Android view hierarchy in XML format and\nparse the text description of the screen, to extract dual-gesture actions from text-based actions, and to\ndeliver the dual-gesture action to the Android emulator.\nInteraction Frequency\nThe Android emulators run asynchronously independent of the agent\nthat is interacting with the environments. However, this asynchronicity between the agent and the\nenvironment may cause several issues such as incomplete transition of the environments or delayed\nsuccess signals. To alleviate the issue, we adjust the interaction frequency between agents and\nenvironments. Specifically, this adjustment is operated by forcing the agent to wait a pre-defined\ntime before fetching the screen information from the environment. In our experiments, we fix the\ninteraction frequency during evaluation to be 1/3Hz across all types of agents.\nObservation space\nThe observation space is comprised of either a screen image, a text description\nof the screen in XML formats based on the Android view hierarchy, or both.\nThe screen images are used for multi-modal large language model (MLLM) agents and vision-\nlanguage-UI (VLUI) agents. Each image is resized into a resolution of 1024 × 2048 for MLLM\nagents and 128 × 256 for VLUI agents.\nThe text descriptions are used for agents with LLMs and MLLMs. To build the text description, the\nAndroid debug bridge (ADB) UI Automator is employed for acquiring the Android view hierarchy\nin XML format. A pre-defined parser, then, converts the list of UI elements with attributes in an\nXML file into a set of text descriptions of UI elements. The description includes the numeric tag\nof UI elements, a short description of the elements including class name or content descriptions\n(e.g., “com.google.android.apps.nexuslauncher.id title weather text Sunny,1◦C” for a view in the\nhome screen describing the weather). Also, we optionally provide the bounding box x-y coordinates\nspecifying the location of the elements, such as the slider interface. Moreover, we define the parser to\ncapture the descriptions of all the nodes in the Android view hierarchy. This is because we observe\nthat many UI elements are omitted if we only parse leaf nodes, resulting in meaningless descriptions\nas discussed in Section 4.3.\nAction space\nThe action space of the agents is defined as a set of dual-gesture actions {a| a =\n(ytouch, xtouch, ylift, xlift) ∈R4}, similar to Rawles et al. (2023). Each value of dual-gesture action a is\nnormalized to be in between [−1, 1] with respect to the screen resolutions. The former two values\nspecify the location of the screen to touch, while the latter two values determine the location of the\nscreen to lift. This definition enables interpreting useful actions in digital device control, i.e., tapping\n14\n\n\nICLR 2024 Workshop on Generative Models for Decision Making\nor swiping the screens, in a precise and compressive manner. Also, our interface allows pressing the\nnavigator buttons available by touching the screen to support the essential actions for manipulating\nAndroid devices.\nSpecifically, we implement an interface that determines whether the action is a tap, swipe, or pressing\nof navigation buttons i.e., back, home, and overview. The action parsing interface converts the action\ninto taps, swipes, or pressing buttons following the rule as follows:\n• The action is tapping, if d((xtouch, ytouch), (xlift, ylift)) < threshold\n– The tapping is to press BACK button, if (xtouch, ytouch) = (0.95, 0.22)\n– The tapping is to press HOME button, if (xtouch, ytouch) = (0.95, 0.50)\n– The tapping is to press OVERVIEW button, if (xtouch, ytouch) = (0.95, 0.78)\n• The action is swiping, if d((xtouch, ytouch), (xlift, ylift)) ≥threshold,\nwhere the threshold value is defined as 0.14. This value is adjustable by users, while we find that\nthe value of 0.14 ensures proper interactions over UI elements, e.g., tapping the target application\nicon, in all of our experiments. These specific values are tested to be consistent across different device\ntypes, ensuring that the positions correspond to the correct buttons in all B-MoCA environments.\nFor LLM agents and MLLM agents, we further define action options. Following the action space\ndefinition, the action options are designed to be compatible with a dual-gesture action. We prompt\nthe LLM agents to output actions among six possible options: raw dual-gesture action, tap, swipe,\npress(“HOME”), press(“BACK”), and press(“OVERVIEW”). These action options are converted into\na corresponding dual-gesture action by an additional action extractor we define as below:\n• For the dual-gesture action, we convert the text action into the four floating points by rounding\neach value into the second decimal point.\n• For tap actions, the LLM agent outputs an integer value specifying the numeric tag assigned to\nthe UI element. Given the tapping action with a numeric tag, the parser converts the action into a\ntapping dual-gesture action with the bounding box information of the chosen UI element.\n• For swipe actions, a direction ‘up’, ‘down’, ‘left’ and ‘right’ is converted into (0.8, 0.5, 0.2, 0.5),\n(0.2, 0.5, 0.8, 0.5), (0.5, 0.2, 0.5, 0.8), and (0.5, 0.8, 0.5, 0.2), respectively.\n• For the action press(“HOME”), press(“BACK”), and press(“OVERVIEW”), we convert the\noutputs to dual-gesture actions in the same way as VLUI agents.\nDuring the evaluation, we ignore the action in the wrong format by skipping the transition of the\nenvironments but penalizing the agents by incrementing the steps taken, yet we observe both Gemini\nand GPT models (as well as vision version of them) rarely make mistakes on the format.\nA.2\nTRAINING AND TEST ENVIRONMENTS CONFIGURATIONS\nWe construct 45 unique environments in B-MoCA, where 35 environments are for training and 10\nenvironments are for testing. Each environment is provided with a unique identification (ID) number,\nto distinguish the environments easily. Table 3 shows the list of the device configurations and the\nhome screen images of exemplary environments.\nTo construct environments, we use popular device types: Pixel 3, Pixel 4, Pixel 4 XL, Pixel 6, and\nWGXA Tablet. For training environments, only Pixel 3 is employed. For evaluation environments,\nwe use all device types Pixel 3, Pixel 4, Pixel 4 XL, Pixel 6, and WGXA Tablet. In these models,\nwe alter the icon and font sizes by changing the dots per inch (DPI) values of the devices. For each\ndevice type, we prepare three different sizes that users can select. We, then, change the wallpaper\nwith 13 images collected from a free license image website. These wallpaper image files are shared\nin the open-source repository. We also customize the background images with the dark theme mode.\nIf the dark theme mode is activated, the device provides screen images with light-dark color reversed.\nFor instance, the wallpaper of the application list page is white in the default setting, while it becomes\nblack with dark theme mode activated. Furthermore, we incorporate changes in locale, specifying\nthe language and location of the devices. 12 different locales are used for 35 training environments,\nwhile we include three more locales for the test environments.\n15\n\n\nICLR 2024 Workshop on Generative Models for Decision Making\nTable 3: The device configuration of each environment with the home screen image\nID\n000\n001\n002\n003\n004\nDevice type\nPixel 3\nPixel 3\nPixel 3\nPixel 3\nPixel 3\nDPI\n330\n330\n440\n440\n550\nLocale\nen-US\nen-US\nen-US\nen-US\nen-US\nWallpaper\n000.jpg\n000.jpg\n000.jpg\n000.jpg\n000.jpg\nDark theme\n-\n-\n-\n-\n-\nID\n005\n006\n007\n008\n009\nDevice type\nPixel 3\nPixel 3\nPixel 3\nPixel 3\nPixel 3\nDPI\n440\n440\n330\n440\n550\nLocale\nen-US\nen-US\nen-US\nen-US\nen-US\nWallpaper\n000.jpg\n000.jpg\n001.jpg\n002.jpg\n001.jpg\nDark theme\n-\n-\n✓\n✓\n-\nID\n010\n011\n012\n013\n014\nDevice type\nPixel 3\nPixel 3\nPixel 3\nPixel 3\nPixel 3\nDPI\n330\n440\n550\n440\n440\nLocale\nen-US\nen-US\nen-US\nen-US\nen-US\nWallpaper\n002.jpg\n008.jpg\n003.jpg\n010.jpg\n013.jpg\nDark theme\n-\n✓\n✓\n-\n✓\nContinued on next page\n16\n\n\nICLR 2024 Workshop on Generative Models for Decision Making\nTable 3: The device configuration of each environment with the home screen image (Continued)\nID\n015\n016\n017\n018\n019\nDevice type\nPixel 3\nPixel 3\nPixel 3\nPixel 3\nPixel 3\nDPI\n330\n440\n440\n550\n330\nLocale\nen-US\nen-US\nen-US\nen-US\nen-US\nWallpaper\n008.jpg\n007.jpg\n004.jpg\n010.jpg\n013.jpg\nDark theme\n-\n-\n✓\n✓\n-\nID\n020\n021\n022\n023\n024\nDevice type\nPixel 3\nPixel 3\nPixel 3\nPixel 3\nPixel 3\nDPI\n440\n330\n440\n550\n330\nLocale\nen-US\nes-US\nes-US\nfr-CA\nfr-CA\nWallpaper\n004.jpg\n001.jpg\n002.jpg\n001.jpg\n002.jpg\nDark theme\n-\n✓\n✓\n-\n-\nID\n025\n026\n027\n028\n029\nDevice type\nPixel 3\nPixel 4\nPixel 4\nPixel 4\nPixel 5\nDPI\n440\n550\n440\n440\n330\nLocale\nzh-hans-CN\nzh-hans-CN\nhi-IN\nja-JP\nru-MD\nWallpaper\n008.jpg\n003.jpg\n010.jpg\n013.jpg\n008.jpg\nDark theme\n✓\n✓\n-\n✓\n-\nContinued on next page\n17\n\n\nICLR 2024 Workshop on Generative Models for Decision Making\nTable 3: The device configuration of each environment with the home screen image (Continued)\nID\n030\n031\n032\n033\n034\nDevice type\nPixel 3\nPixel 3\nPixel 3\nPixel 3\nPixel 3\nDPI\n440\n440\n550\n330\n440\nLocale\nar-AE\nde-DE\nak-GH\npt-BR\npt-PT\nWallpaper\n007.jpg\n004.jpg\n010.jpg\n013.jpg\n004.jpg\nDark theme\n-\n✓\n✓\n-\n-\nID\n100\n101\n102\n103\n104\nDevice type\nPixel 3\nPixel 3\nPixel 3\nPixel 3\nPixel 3\nDPI\n440\n330\n440\n550\n440\nLocale\nen-US\nen-US\nen-US\nen-US\nfr-CA\nWallpaper\n000.jpg\n000.jpg\n009.jpg\n012.jpg\n009.jpg\nDark theme\n-\n-\n✓\n-\n✓\nID\n109\n105\n106\n107\n108\nDevice type\nWXGA\nTablet\nPixel 3\nPixel 4\nPixel 5\nPixel 6\nDPI\n160\n550\n440\n550\n700\nLocale\nar-EG\nko-KR\nen-US\nen-US\nur-PK\nWallpaper\n012.jpg\n009.jpg\n012.jpg\n005.jpg\n011.jpg\nDark theme\n-\n✓\n-\n✓\n-\n18\n\n\nICLR 2024 Workshop on Generative Models for Decision Making\nB\nTASK DETAILS\nB.1\nLIST OF DAILY TASKS\nB-MoCA presents 60 daily tasks that are common in everyday life. The tasks are designed to operate\nin diverse environments seamlessly and cover commonly used applications. Daily tasks effectively\nsimulate a wide range of essential skills for mobile device control problems, such as manipulating UI\nelements (including application icons, checkboxes, and sliders), and can be employed for evaluating\nmobile device control agents’ capabilities in performing tasks that mirror our daily activities.\nIn Table 4, we include the detailed list of tasks with the maximum step limit and the success criteria.\nThe success criteria are defined in the form of regular expression and are employed by the rule-based\nsuccess detector. The success criteria (filter) specifies the target application or activity, and the success\ncriteria (regex) refers to the regular expression we use. We also define the maximum step limits,\nwhich are set for the rigorous evaluation of the agents’ proficiency on each task.\nB.2\nEXAMPLE OF DEMONSTRATION ON REPRESENTATIVE TASKS\nIn our experiments, we select six representative tasks. The tasks are selected to cover a wide range of\nfunctionalities, such as navigating pages (e.g., tab in the clock application or different setting pages\nin the setting application) and manipulating various UI elements (e.g., checkbox, slider, time pickers,\netc.). On each task, we display the successful demonstration in Figure 9.\nTable 4: Comprehensive list of tasks.\nStep\nlimit\nTask instruction\nSuccess criteria (filter)\nSuccess criteria (regex)\n4\n“open the calendar app”\nActivityTaskManager\nˆ(.*?)START(.*?)com.android.calendar\n4\n“open the camera app”\nActivityTaskManager\nˆ(.*?)Start proc(.*?)com.android.camera\n4\n“open the chrome app”\nActivityTaskManager\nˆ(.*?)START(.*?)com.google.android.apps.chrome\n4\n“open the clock app”\nActivityTaskManager\nˆ(.*?)START(.*?)com.android.deskclock\n4\n“open the contact app”\nActivityTaskManager\nˆ(.*?)Start proc(.*?)com.android.contacts\n4\n“open the file manager app”\nActivityTaskManager\nˆ(.*?)START(.*?)files.FilesActivity\n4\n“open the gmail app”\nActivityTaskManager\nˆ(.*?)START(.*?)com.google.android.gm\n4\n“open the map app”\nActivityTaskManager\nˆ(.*?)START(.*?)com.google.android.maps.MapsActivity\n4\n“open the message app”\nActivityTaskManager\nˆ(.*?)START(.*?)com.google.android.apps.messaging\n4\n“open the phone app”\nDialer\nˆ(.*?)MainActivity.onCreate\n4\n“open the photos app”\nActivityTaskManager\nˆ(.*?)START(.*?)com.google.android.apps.photos\n4\n“open the play music app”\nActivityTaskManager\nˆ(.*?)START(.*?)com.android.music\n4\n“open the setting app”\nActivityManager\nˆ(*.?)Start proc(.*?)com.android.settings.Settings\n4\n“open the youtube app”\nActivityTaskManager\nˆ(.*?)START(.*?)com.google.android.youtube\n4\n“turn on alarm at 9 am”\nAlarmClock\nˆ(.*?)Created new alarm instance\n5\n“delete alarm at 9 am”\nAlarmClock\nˆ(.*?)Removed alarm\n5\n“go to the alarm page in clock”\nAlarmClock\nˆ(.*?)Events: [Alarm] [Show Tab] [Tap]\n5\n“go to the stopwatch page in clock”\nAlarmClock\nˆ(.*?)Events: [Stopwatch] [Show Tab] [Tap]\n5\n“go to the timer page in clock”\nAlarmClock\nˆ(.*?)Events: [Timer] [Show Tab] [Tap]\n5\n“list audio files in file manager”\nDirectoryFragment\nˆ(.*?)Showing directory(.*?)audio(.*?)root\n5\n“list image files in file manager”\nDirectoryFragment\nˆ(.*?)Showing directory(.*?)images\n5\n“list video files in file manager”\nDirectoryFragment\nˆ(.*?)Showing directory(.*?)videos\nContinued on next page\n19\n\n\nICLR 2024 Workshop on Generative Models for Decision Making\nTable 4: Comprehensive list of tasks. (Continued)\n5\n“list download files in file manager”\nDirectoryFragment\nˆ(.*?)Showing directory(.*?)download\n5\n“activate the insert page in contact”\nActivityTaskManager\nˆ(.*?)START(.*?)INSERT(.*?)ContactEditorActivity\n5\n“activate the edit page in contact”\nActivityTaskManager\nˆ(.*?)START(.*?)EDIT(.*?)ContactEditorActivity\n5\n“activate the search bar in chrome”\nAndroidIME\nˆ(.*?)LatinIme.onActivate(.*?)android.chrome\n5\n“activate the search bar in map”\nAndroidIME\nˆ(.*?)LatinIme.onActivate(.*?)apps.map\n5\n“activate the search bar in youtube”\nAndroidIME\nˆ(.*?)LatinIme.onActivate(.*?)android.youtube\n5\n“activate the search bar in google”\nAndroidIME\nˆ(.*?)LatinIme.onActivate(.*?)android.googlequicksearchbox\n5\n“activate the search bar in message”\nAndroidIME\nˆ(.*?)LatinIme.onActivate(.*?)apps.messaging\n5\n“start chatting in message”\nBugleUsageStatistics\nˆ(.*?)BUGLE CREATE(.*?)DEFAULT\n5\n“press the call button in dial”\nTelecom\nˆ(.*?)LogUtils(.*?)EventRecord added as Call\n5\n“turn on airplane mode”\nPhoneGlobals\nˆ(.*?)Turning radio off(.*?)airplane\n5\n“turn off airplane mode”\nPhoneGlobals\nˆ(.*?)Turning radio on(.*?)airplane\n5\n“turn on wifi”\nWifiService\nˆ(.*?)setWifiEnabled(.*?)com.android.settings(.*?)enable=true\n5\n“turn off wifi”\nWifiService\nˆ(.*?)setWifiEnabled(.*?)com.android.settings(.*?)enable=false\n5\n“start the stopwatch in clock”\nAlarmClock\nˆ(.*?)Start\n5\n“pause the stopwatch in clock”\nAlarmClock\nˆ(.*?)Pause\n5\n“reset the stopwatch in clock”\nAlarmClock\nˆ(.*?)Reset\n5\n“go to search history in chrome”\nActivityTaskManager\nˆ(.*?)START(.*?)chrome.browser.history.HistoryActivity\n5\n“go to trash page in photo”\nActivityTaskManager\nˆ(.*?)START(.*?)apps.photos(.*?)TrashPhotosActivity\n5\n“go to smart pairing page in youtube”\nActivityTaskManager\nˆ(.*?)START(.*?)youtube.mdx.smartpairing.PairWithTvActivity\n6\n“increase media volume in setting”\nvol.Events\nˆ(.*?)MEDIA\n6\n“increase call volume in setting”\nvol.Events\nˆ(.*?)CALL\n6\n“increase ring volume in setting”\nvol.Events\nˆ(.*?)MUSIC\n6\n“increase alarm volume in setting”\nvol.Events\nˆ(.*?)ALARM\n6\n“decrease screen brightness in set-\nting”\nDisplayPowerController\nˆ(.*?)Brightness(.*?)changing(.*?)manual\n6\n“toggle dark theme in setting”\nSettingsProvider\nˆ(.*?)content(.*?)settings(.*?)dark(.*?)mode\n6\n“toggle vibrate for calls in setting”\nSettingsProvider\nˆ(.*?)vibrate(.*?)when(.*?)ringing\n6\n“go to app info list in setting”\nSettingsActivity\nˆ(.*?)Switching(.*?)android.settings(.*?)ManageApplications\n6\n“go to bluetooth setting”\nPrefCtrlListHelper\nˆ(.*?)android.settings.bluetooth.BluetoothDevice\n7\n“go to ‘add a language’ page in set-\nting”\nActivityTaskManager\nˆ(.*?)LocalePicker\n9\n“call 911”\nTelecom\nˆ(.*?)Emergency number detected\n10\n“turn off the call in process”\nTelecom\nˆ(.*?)InCallController(.*?)onCallRemoved\n11\n“create alarm at 06:30 am”\nConditionProviders.SCP\nˆ(.*?)nextUserAlarmTime(.*?)06:30:00\n11\n“create alarm at 10:30 am”\nConditionProviders.SCP\nˆ(.*?)nextUserAlarmTime(.*?)10:30:00\n11\n“create alarm at 13:30 pm”\nConditionProviders.SCP\nˆ(.*?)nextUserAlarmTime(.*?)13:30:00\n11\n“create alarm at 17:30 pm”\nConditionProviders.SCP\nˆ(.*?)nextUserAlarmTime(.*?)17:30:00\n11\n“create alarm at 20:30 pm”\nConditionProviders.SCP\nˆ(.*?)nextUserAlarmTime(.*?)20:30:00\n11\n“create alarm at 23:30 pm”\nConditionProviders.SCP\nˆ(.*?)nextUserAlarmTime(.*?)23:30:00\n20\n\n\nICLR 2024 Workshop on Generative Models for Decision Making\n(a) Airplane\n(b) Alarm1\n(c) Alarm2\n(d) Brightness\n(e) Call 911\n(f) Language\nFigure 9: Examples of human expert demonstrations of six representative tasks. The blue and red\ncursors linked with a white arrow identify the swiping action, while the red cursor alone identifies\nthe tapping action.\nC\nAGENT DETAILS\nC.1\nPROMPT DETAILS FOR LLM AGENTS AND MLLM AGENTS\nFor the agents employing LLMs or MLLMs, we use a complete prompt format described in Table 5.\nThe role description informs the agents with general instructions about the problem, i.e., device\ncontrol problem. The possible actions are provided as callable functions, options of tapping an\nelement in the list, swiping the screen, and pressing the three navigation buttons over the screen with\naction press(“BACK”), action press(“HOME”), and action press(“OVERVIEW”). The output format\nis designed to integrate the Chain-of-Thought (CoT) technique (Wei et al., 2022).\n21\n\n\nICLR 2024 Workshop on Generative Models for Decision Making\nYou are an agent that is trained to perform daily tasks on digital devices, such as smartphones. You are given a\ngoal of task instruction to accomplish and a description of screen from Android view hierarchy, which contains\nelements’ numeric tag and description.\nBased on the goal of task instruction and UI elements list, you need to select an action option by calling one of\nthe following functions to control the digital device:\n1. dual-gesture(touch y: float, touch x: float, lift y: float, lift x: float): This function is used to operate\na dual-gesture action. A dual-gesture comprises of four floating point numeric values, in between 0 and 1\nindicating a normalized location of the screen in each of x-y coordinates. A dual-gesture action is interpreted\nas touching the screen at the location of (touch y, touch x) and lifting at the location of (lift y, lift x). The\ndual-gesture action indicates a tapping action if the touch and lift locations are identical but a swiping action if\nthey differ. A simple use case is dual-gesture(0.5, 0.5, 0.5, 0.5) to tap the center of the screen.\n2. tap(numeric tag: int): This function is used to tap an UI element shown on the digital device screen. ”numeric\ntag” is a tag assigned to an UI element shown on the digital device screen. A simple use case can be tap(5),\nwhich taps the UI element labeled with the number 5.\n3. swipe(direction: str): This function is used to swipe on the digital device screen, ”direction” is a string that\nrepresents one of the four directions: up, down, left, right. ”direction” must be wrapped with double quotation\nmarks. A simple use case is swipe(”up”) which can be used to open the app list in the home screen.\n4. press(”HOME”): to press home button.\n5. press(”BACK”): to press back button.\n6. press(”OVERVIEW”): to press overview button.\nGoal: [task instruction].\n[few shot prompt]\nNow, given the parsed uiautomator xml, you need to think and call the function needed to proceed with the task.\nYour output should include three parts in the given format:\n- Description: <Describe what you observe in the input>\n- Thought: <To complete the given task, what is the next step I should do>\n- Action: <The function call with the correct parameters to proceed with the task. You cannot output anything\nelse except a function call>\nYou can only take one action at a time, so please directly call the function.\nPlease never take action beside options provided.\nTable 5: Prompts used for the LLM Agents and MLLM agents. Parts for [...] are filled in according\nto different experiments, while the few-shot examples are optional.\nFor few-shot learning of agents with foundation models, we include a pre-defined number of examples\nhinting correct actions to take. Specifically, to build a prompt with few-shot examples, the [few shot\nprompt] part in Table 5 is replaced with the text illustrating the human demonstration. Table 6 shows\nan exemplary few show prompts, with one transition of the human expert demonstration.\nBelow illustrates exemplary human demonstration(s), with format:\n- Instruction: <The instruction of task>\n- Observation: <An observation from environment>\n- Action: <An action taken by the human expert>\n- Next Observation: <The next observation from environment after the action is taken>\n- Reward: <A reward after action is executed>.\n- Instruction: turn on alarm at 9 am\n- Observation: [’numeric tag’: 0, ’description’: ’android.view.View Appslist’, [...] ’numeric tag’: 27, ’descrip-\ntion’: ’android.widget.FrameLayout’]\n- Action: swipe(”up”)\n- Next observation: [’numeric tag’: 0, ’description’: ’android.view.View Appslist’, [...] ’numeric tag’: 30,\n’description’: ’android.widget.FrameLayout’]\n- Reward: 0.0\nTable 6: An exemplary few-shot prompt with one transition of human expert demonstration for\ntext-based agents. The abbreviated [...] parts are filled with a list of descriptions for UI elements.\n22\n\n\nICLR 2024 Workshop on Generative Models for Decision Making\nC.2\nARCHITECTURE DESIGN FOR VLUI AGENTS\nThe network architecture for VLUI agents is composed of three components: encoder, attention\nmodule, and action head. Given the task instruction c and the visual screen ot ∈R3×256×128 at each\ntimestep t, VLUI agents generate action at ∈R4 in the form of dual-gesture action.\nVLUI agents use visual and text encoders to represent screen images ot and task instruction c,\nrespectively. The visual encoder embeds visual feature eot ∈Rd from the observation ot, and the\ntext encoder extracts features ec ∈Rd from the task instruction c. For the visual encoder, we use\nEfficientNet-b0 (Tan & Le, 2019) pre-trained with ImageNet followed by an adaption layer using a\nfully connected layer to adapt the output channel to hidden dimension d (Liu et al., 2023). For the\ntext encoder, we use a pre-trained text encoder of Text-to-Text Transfer Transformer (Zhan & Zhang,\n2023) which is trained with a dataset composed of demonstrations for solving Android device control\nproblems (Rawles et al., 2023). The text encoder is kept frozen during the training process. The\nhidden dimension d is set to equal the value of 768 for both visual embedding and text embedding.\nThe attention module, then, fuses the visual feature eot and text feature ec into a single vision-\nlanguage embedding efused ∈Rd. Especially, we use Multi-head attention layer (Vaswani et al., 2017)\nfor cross-attention, with ec given as query and eot given as key and value. Given the fused feature\nefused, the action heads predict the action at. The action head consists of fully connected (FC) layers\nwith the last layer having an output dimension of 4, accounting for the dimension of a ∈R4. The\nsequence of three FC layers follows output dimensions of (1024, 1024, 4). We apply the tanh layer to\nthe predicted action, observing improved performances with normalization of the action values.\nC.3\nCONFIGURATION DETAILS FOR LLM AGENTS AND MLLM AGENTS\nFor the experiments in Section 4.2 and Section 4.3, we set the configurations for the foundation\nmodels. We use a temperature of 0.1, a top-p of 1, and a top-k of 1 for Gemini-Pro and Gemini-\nPro-V. We set the temperature to be 0.0 and top-p with the default value of 1 (as altering only either\ntemperature or top-p from the default setting is suggested) for GPT-4 and GPT-4V.\nC.4\nTRAINING DETAILS FOR VLUI AGENTS\nFor the experiments in Section 4.2 and Section 4.4, we train VLUI agents with behavior cloning (BC)\nover 4K steps with a batch size of 512, sampled from a collection of 210 human demonstrations.\nWe use the Adam optimizer (Kingma & Ba, 2017) with a learning rate of 3e-4 and adopt a cosine\nannealing learning rate scheduler. Each training is conducted on a single NVIDIA RTX A6000 GPU\nand takes approximately one hour.\nD\nEXPERIMENT DETAILS\nD.1\nDATASET COLLECTION\nFor the few-show learning of LLM and MLLM agents and training of VLUI agents, we collect\nhuman expert demonstrations. The collectors (graduate students) are instructed to complete the six\nrepresentative tasks in each training environment. The definitions of action space for the collected\ndemonstration are in two modes: the action space defined with action options and the action space\nas a set of dual-gesture actions. The end of each episode is determined by the ADB-based success\ndetector, and we exclude the demonstrations with failures.\nFor the experiments in Section 4.2, we exploit training environments with identifying (ID) numbers\nfrom 000 to 034. Hence, a total number of 210 trajectories of demonstrations are prepared. For agents\nusing foundation models, each transition (task instruction, observation, action, next observation,\nreward) in the trajectories is sampled as a few-shot example, similar to prior methods (Zhang et al.,\n2023; Rawles et al., 2023). For VLUI agents, each triplet (task instruction, observation, action)\nin the trajectories is used as a data point for composing the training batch. For the experiments\nin Section 4.4, we leverage varying numbers of training environments 7, 21, and 35 where the\ncorresponding identifying (ID) numbers of the environments are from 000 to 006, from 000 to 021,\nand from 000 to 034, respectively. The total number of demonstrations for each setting is 42, 126,\nand 210, respectively.\n23\n\n\nICLR 2024 Workshop on Generative Models for Decision Making\nD.2\nPERFORMANCES OF VLUI AGENTS IN TRAINING ENVIRONMENTS\nAirplane\nAlarm1\nAlarm2\nBrightness\nCall 911\nLanguage\n30\n60\n90\nSuccess Rates (%)\nTrain\nTest\nFigure 10: Success rates of VLUI agents trained with BC on training and test environments. The\ndifferences between the success rates demonstrate the headroom for the generalization ability.\nFigure 10 displays the differences in the success rates of VLUI agents in training and test environments.\nThe challenges with diverse device configurations degenerate the performances of the VLUI agents,\nfrom higher than 90% in the training environments to less than 70% in the test environments.\nD.3\nEXAMPLES OF VISUAL INPUTS WITH SOM PROMPTING FOR MLLM AGENTS\nIn Section 4.3, we investigate the effects of SoM prompting that several prior works (Yan et al., 2023;\nYang et al., 2023b) adopted. Figure 11 presents several examples of visual inputs used for analysis.\n(a)\n(b)\n(c)\n(d)\nFigure 11: Examples of visual inputs for MLLM agents with SoM prompting. The overlaid numeric\ntags for selecting icons at (a) the home screen or (b) the menu screen of applications list can be\nbeneficial for MLLM agents, while the tags on (c) dial buttons or (d) clock UI may confuse them.\nE\nADDITIONAL EXPERIMENTS\nE.1\nMLLM AGENTS WITH DUAL-GESTURE ACTIONS\nWe have implemented additional interfaces for agents employing foundation models and action\noptions, as introduced in Section 3.1. However, it is still questionable whether defining action options\nis truly desirable for these agents. To answer this, we conduct a comparison between agents only\ngenerating actions in the dual-gesture action format and the agents using additional action options.\nIn this experiment, we examine MLLM agents employing GPT-4V in a zero-shot manner on two\nselected tasks (Airplane and Alarm 1) with only one run for simplicity.\nAs shown in Table 7, we observe that the agents benefit from employing additional action options. In\nthe experiments, the agents without additional options exhibit several successful trials on Alarm 1,\nby including the bounding box location of all UI elements in the observation prompt for these agents.\nHowever, we observe that the agents lack generating diverse dual-gesture actions but only perform\ntapping actions. With these results, we examine the proficiency of LLM agents and MLLM agents\nwith the action options in Section 4.2.\n24\n\n\nICLR 2024 Workshop on Generative Models for Decision Making\nMLLM agents\n(dual-gesture actions)\nMLLM agents\n(action options)\nAirplane\n00\n30\nAlarm 1\n30\n50\nTable 7: Success rates of MLLM agents with different action spaces. MLLM agents (dual-gesture\nactions) generate the actions in only dual-gesture action format, and MLLM agents (action options)\nleverage the additional action options we define.\nE.2\nLLM AGENTS WITH OPEN-SOURCE MODELS\nWhile employment of foundation models for mobile device control agents is gaining interests (Wen\net al., 2023; Yang et al., 2023b), many approaches still rely on closed-source models. However,\nleveraging closed-source foundation models lies with severe limitations, such as difficulties in fine-\ntuning. Instead, one can employ open-source LLMs which can benefit from high flexibility in usage.\nIn this experiment, we examine the proficiency of LLM agents with open-source models.\nWe study open-source models: Llama2-chat (abbreviated as Llama2) (Touvron et al., 2023), Llama3,\nand AgentLM (Zeng et al., 2023). Llama2 and Llama3 are open-source models that have shown\ncompatible performances with several closed-sourced models, and AgentLM is an instruction-tuned\nversion of Llama2 in a collection of numerous agent tasks (including web tasks). For Llama2 and\nAgentLM, we use 7b and 13b size models. For Llama3, we use an 8b size model. We set the\ntemperature value to be 0.1 and report across three different runs.\nTable 8 show the success rates of LLM with open-source models. We observe that these agents\nseverely lack the proficiency in performing tasks that we select. While the agents can perform\nsub-tasks of opening the target application on the home screen or entering the menu screen in some\ntrials, as observed in the rollouts, they fail to complete the instructed tasks in limited allowed steps.\nThe open-source models also struggle with generating actions in the format we instruct, while closed-\nsource models rarely generate actions in the wrong format. With these pilot test results, we primarily\nfocus on examining the efficacy of training agents from scratch with VLUI agents.\nLLM agents\n(Llama2-7b)\nLLM agents\n(Llama2-13b)\nLLM agents\n(AgentLM-7b)\nLLM agents\n(AgentLM-13b)\nLLM agents\n(Llama3-8b)\nLLM agents\n(Gemini-Pro)\nLLM agents\n(GPT-4)\nAirplane\n00 ± 00\n00 ± 00\n00 ± 00\n00 ± 00\n63 ± 00\n87 ± 07\n53 ± 03\nAlarm 1\n00 ± 00\n00 ± 00\n00 ± 00\n00 ± 00\n00 ± 00\n27 ± 03\n42 ± 13\nBrightness\n00 ± 00\n00 ± 00\n00 ± 00\n00 ± 00\n17 ± 03\n05 ± 03\n73 ± 12\nTable 8: Success rates of LLM agents with open-source models Llama2 and AgentLM in zero-shot\nscenario. The agents do not complete any tasks that LLM agents with closed-source models of GPT-4\nor Gemini (in zero-shot) have achieved.\nE.3\nVLUI AGENTS WITH REINFORCEMENT LEARNING\nWe study VLUI agents trained using reinforcement learning (RL) algorithms, by using the success\nsignal rt as a sparse reward. Formally, the RL agent is trained to maximize the expected return,\ndenoted as follows:\nEπθ\n\u0014\nT\nX\nt=0\nγtrt\n\u0015\n,\nwhere γ ∈[0, 1] is the discount factor and T is the length of episode. In this experiment, we focus\non offline RL setting (Lange et al., 2012; Levine et al., 2020), where the agent learns from a pre-\ncollected dataset generated by some behavior policy. Specifically, We utilize the implicit Q-learning\n(IQL; Kostrikov et al. 2022) algorithm.\nIQL is one of the popular actor-critic algorithms. The actor network π parameterized with θ\ninfers the action at at each time step t, given the observation ot and task instruction ct, The critic\nnetwork Q parameterized with ϕ estimates the value of action predicted by the actor. IQL leverages\nexpectile regression for robust value estimation and improves the policy using advantage-weighted\n25\n\n\nICLR 2024 Workshop on Generative Models for Decision Making\nregression (Peters & Schaal, 2007; Nair et al., 2020). In particular, IQL introduces a separate value\nnetwork V parameterized with ψ for robust learning. The loss function for the critic in IQL is\nformulated as:\nLV (ψ) = E(ot,a∗\nt )∼D\nh\nLτ\n2\nQ ˆ\nϕ(ot, at) −Vψ(ot)\n\u0001i\n,\nLQ(ϕ) = E(ot,a∗,rt,ot+1)∼D\nhrt + γ · Vψ(ot+1) −Qϕ(ot, a∗\nt )\n\u00012i\n,\nwith Lτ\n2(u) defined to be |τ −1(u < 0)| · u2, a value function V parameterized with ψ, Q-function\nQ parameterized with ϕ, and the dataset of human demonstrations D = {(ot, a∗\nt , rt, ot+1)}. Then,\nthe actor is trained with advantage-weighted behavioral cloning objective defined as:\nLπ(θ) = E(ot,a∗\nt )∼D\nh\nexp\nβ · A(ot, at)\n\u0001\n·\nat −a∗\nt\n\u00012i\n,\nwith action prediction at predicted by the actor network π, the advantage A(ot, at) = Q ˆ\nϕ(ot, at) −\nVψ(ot), and an inverse temperature β ∈[0, ∞).\nThe policy architecture for VLUI agents follows the same architecture of VLUI agents trained with\nBC, described in Appendix C.2. Similarly, the hyperparameters for optimizers and other training\ndetails remain the same, except that we iterate the training over 20K steps to ensure the convergence\nof training. For the training, we employ 35 training environments, namely 210 successful human\nexpert demonstrations.\nFigure 12 shows the success rates of VLUI agents trained with IQL, compared with VLUI agents\ntrained with BC. We observe that the agents trained with IQL do not exhibit compatible performances\nwith the agents trained with BC across all tasks. We assume these results originated from training\ninstability due to sparse rewards, as more training steps are required for the convergence of IQL\ntraining. However, as observed in Kostrikov et al. (2022), we expect that offline RL can provide\npotential benefits over vanilla BC training, such as utilizing failure demonstrations. We leave training\nVLUI agents with IQL more efficiently and with higher proficiencies as future work.\nE.4\nVLUI AGENT WITH REPRESENTATION MODEL OF VARYING CAPACITY\nWe conduct the effect of representation models with varying capacities on the robustness of VLUI\nagents. Specifically, we compare the VLUI agents equipped with visual encoders using EfficientNet-\nb0, EfficientNet-b3, and EfficientNet-b7 (with increasing numbers of parameters with values of 5.3M,\n12M, and 66M, respectively), which are pre-trained with ImageNet.\nFigure 13 demonstrates the experimental results. To illustrate, our experiment indicates no significant\nimprovements by employing representation models with increased model sizes. With these results,\nwe expect that increasing the training data diversity is more desirable than increasing the model sizes\nin our current benchmark setting. Also, we add that the higher model capacity can be beneficial for\ndeveloping multi-task policies with a greater number of tasks.\nAirplane\nAlarm1\nAlarm2\nBrightness\nCall911\nLanguage\n10\n50\n90\nSuccess Rates (%)\n BC\n IQL\nFigure 12: Success rates of VLUI agents trained\nwith BC and IQL. Training with IQL does not\nresult in as high performances as training with\nBC, presumably due to training instability.\nAirplane\nAlarm1\nAlarm2\nBrightness\nCall911\nLanguage\n10\n50\n90\nSuccess Rates (%)\nEfficientNet-b0\nEfficientNet-b3\nEfficientNet-b7\nFigure 13: Success rates ov VLUI agents with\nvarying size of visual encoders. We do not ob-\nserve significant benefits by increasing the model\ncapacities of representation models.\n26\n\n\nUnderstanding the Weakness of Large Language Model Agents\nwithin a Complex Android Environment\nMingzhe Xing∗\nPeking University\nRongkai Zhang\nPeking University\nHui Xue\nMicrosoft Research\nQi Chen\nMicrosoft Research\nFan Yang\nMicrosoft Research\nZhen Xiao†\nPeking University\nABSTRACT\nLarge language models (LLMs) have empowered intelligent agents\nto execute intricate tasks within domain-specific software such as\nbrowsers and games. However, when applied to general-purpose\nsoftware systems like operating systems, LLM agents face three pri-\nmary challenges. Firstly, the action space is vast and dynamic, posing\ndifficulties for LLM agents to maintain an up-to-date understanding\nand deliver accurate responses. Secondly, real-world tasks often re-\nquire inter-application cooperation, demanding farsighted planning\nfrom LLM agents. Thirdly, agents need to identify optimal solutions\naligning with user constraints, such as security concerns and prefer-\nences. These challenges motivate AndroidArena, an environment\nand benchmark designed to evaluate LLM agents on a modern op-\nerating system. To address high-cost of manpower, we design a\nscalable and semi-automated method to construct the benchmark.\nIn the task evaluation, AndroidArena incorporates accurate and\nadaptive metrics to address the issue of non-unique solutions. Our\nfindings reveal that even state-of-the-art LLM agents struggle in\ncross-APP scenarios and adhering to specific constraints. Addition-\nally, we identify a lack of four key capabilities, i.e., understanding,\nreasoning, exploration, and reflection, as primary reasons for the\nfailure of LLM agents. Furthermore, we provide empirical analysis\non the failure of reflection, and improve the success rate by 27% with\nour proposed exploration strategy. This work is the first to present\nvaluable insights in understanding fine-grained weakness of LLM\nagents, and offers a path forward for future research in this area. En-\nvironment, benchmark, and evaluation code for AndroidArena are\nreleased at https://github.com/AndroidArenaAgent/AndroidArena.\n1\nINTRODUCTION\nLarge language models (LLMs) have shown great potentials in\nunderstanding hidden intent from human and commonsense rea-\nsoning [27]. This makes it possible to utilize LLM as agent [29, 32],\nan intelligent entity capable of making decisions and executing\n∗This work is done during the internship at Microsoft Research.\n†Corresponding author.\nPermission to make digital or hard copies of all or part of this work for personal or\nclassroom use is granted without fee provided that copies are not made or distributed\nfor profit or commercial advantage and that copies bear this notice and the full citation\non the first page. Copyrights for components of this work owned by others than the\nauthor(s) must be honored. Abstracting with credit is permitted. To copy otherwise, or\nrepublish, to post on servers or to redistribute to lists, requires prior specific permission\nand/or a fee. Request permissions from permissions@acm.org.\nConference acronym ’XX, June 03–05, 2018, Woodstock, NY\n© 2018 Copyright held by the owner/author(s). Publication rights licensed to ACM.\nACM ISBN 978-1-4503-XXXX-X/18/06\nhttps://doi.org/XXXXXXX.XXXXXXX\nactions based upon the perceived state of environment. An exam-\nple is for LLMs to interact with domain-specific software, such as\ndatabases [13], games [28] and browsers [40], for task completion.\nMore recently, new LLM-based agents have emerged to interact\nwith general-purpose software systems, such as operating systems\nalong with their installed APPs, to accomplish more complex open-\ndomain tasks [35, 36]. These tasks range from simple actions like\nsetting reminders to more intricate activities like financial manage-\nment and staying connected with loved ones. Complex scenarios in\noperating systems typically manifest the following characteristics:\n1) a vast and ever-changing action space due to real-time inter-\nnet data exchange, APP installations, and upgrades; 2) an increasing\ndemand for cross-APP collaboration as user tasks become more\ninterconnected and multifaceted; and 3) heightened consideration\nfor personal interests and security concerns.\nThese characteristics motivate us to establish a new environment\nand comprehensive benchmark to study the boundaries of LLM\nagent’s capability within a complex software system. In this paper,\nwe introduce AndroidArena, an environment built on the Android\noperating system, accompanied by an evaluation benchmark con-\ntaining annotated ground truth action sequences. AndroidArena\nsupports real-time internet data exchange and dynamic APP man-\nagement, and enables seamless operations across various APPs.\nThese features facilitate the evaluation of LLM agents in a vast\nand dynamic action space and cross-APP scenarios. Additionally, we\npropose a scalable method for semi-automatically constructing an\ninstruction benchmark, ensuring comprehensive coverage of APP\nfunctionalities. Our open-source benchmark, informed by the afore-\nmentioned characteristics, evaluates tasks not only within a single\nAPP but also complex tasks requiring collaboration across multiple\nAPPs. It further considers tasks subject to constraints such as user\npreferences and security considerations.\nEvaluating tasks within a complex operating system is non-\ntrivial [14], primarily due to the fact that the feasible action se-\nquence for a task is often non-unique. This presents a significant\nchallenge to precisely evaluating agents in multi-step decision-\nmaking scenarios. To address this issue, we devise adaptive met-\nrics to evaluate task completion accurately. The evaluation\nresults reveal that all state-of-the-art (SOTA) LLM agents fall short\nin cross-APP scenarios, with a success rate of less than 60%, and\nstruggle to fully adhere to specific constraints. Notably, GPT-3.5 [18]\nachieves a 6x higher success rate than LLaMA2-70B [25]. Through\nmeticulous case analysis to understand the causes of failure, we\nidentify and abstract four key planning capabilities of LLM\nagents, inspired by reinforcement learning (RL) [24]: understand-\ning, reasoning, exploration, and reflection. We design metrics\narXiv:2402.06596v1  [cs.AI]  9 Feb 2024\n\n\nConference acronym ’XX, June 03–05, 2018, Woodstock, NY\nMingzhe Xing, Rongkai Zhang, Hui Xue, Qi Chen, Fan Yang, and Zhen Xiao\nTable 1: The comparison between our AndroidArena benchmark and existing benchmarks.\nBenchmark\nOnline\nEvaluation\nRealistic\nEnvironment\nScalably\nGenerated\nCollaborative Tasks\nbetween APPs\nTasks with\nConstraints\nMineCraft [28]\n✓\n✗\n✗\n✗\n✗\nMind2Web [7]\n✗\n✓\n✗\n✗\n✗\nAITW [20]\n✗\n✓\n✗\n✗\n✗\nAndroidEnv [26]\n✓\n✓\n✗\n✗\n✗\nWebArena [40]\n✓\n✓\n✗\n✗\n✗\nAndroidArena\n✓\n✓\n✓\n✓\n✓\nto measure these fine-grained capabilities, showing improve-\nment directions for LLM agents. LLaMA2 exhibits weaknesses\nacross all dimensions, and even advanced models like GPT-4 [18]\nare no exemptions, exhibiting weak reflection and exploration abil-\nities. Empirical analysis predominantly attributes the weakness\nin reflection to low-quality trajectories and sparsity in environ-\nment feedback. Moreover, we find that by integrating historical\nvisited information into the prompt and balancing exploration and\nexploitation by the agent, the success rate of specific APPs can\nimprove by 27%, and the exploration performance is enhanced.\nIn summary, we make the following contributions.\n• We open-source AndroidArena, a benchmark based on the An-\ndroid operating system, to evaluate daily tasks requiring cross-\nAPP collaboration, as well as considerations for constraints such\nas security. Additionally, our scalable and semi-automated ap-\nproach reduces the cost of benchmark construction.\n• Our findings indicate that STOA models underperform in daily\ntasks and are not ready for direct product integration. We pro-\npose fine-grained metrics that reveal failure causes and highlight\nfour areas for future research: understanding, reasoning, reflec-\ntion, and exploration. Initial analysis show the failure reasons of\nreflection, and 27% of improvement when enhancing exploration.\n2\nBACKGROUND\n2.1\nFrameworks of LLM Agent\nWith the emergence of LLMs, the study of LLM agents has begun\nto thrive. Early research work [1, 11, 19] prompt LLMs to directly\ngenerate actions based on environment observations. ReAct [38]\nis a pioneer work to integrate reasoning and acting in LLM for\ngeneral task solving. It first generates reasoning traces based on\nhistory context, subsequently producing actions to interact with\nthe environment. Building upon this task-solving paradigm, subse-\nquent agents have been proposed to enhance capabilities in various\ndimensions. Reflexion [22] summarizes textual feedback from the\nenvironment and then incorporates it as additional context for the\nLLM agent in the subsequent episode. The self-reflective context\nacts as a semantic gradient signal, offering the agent a concrete\ndirection to improve upon, and facilitates the learning process from\nprior mistakes for enhancing task performance. This paper focuses\non evaluating the abilities of LLM agents and understanding their\nweaknesses. We adopt ReAct as the basic agent strategy and Reflex-\nion as an approach to assess the agent’s ability of self-reflection.\n2.2\nExisting Operating System Task Benchmark\nOperating Systems (OS) serve as crucial environments with which\nhumans interact daily, and numerous benchmarks have emerged to\nevaluate the performances of agents within OS. AITW [20] stands\nout as a static image dataset that offers human demonstrations of\ndevice interactions. However, the static nature of AITW prevents\nagents from obtaining a reproducible environment. On the other\nhand, AndroidEnv [26] provides support for dynamic interactions\nwith APP. Despite this, it only supports single APP’s interaction in\neach environment instance, which limits its capability to evaluate\ncomplex and realistic tasks. WebArena [40] creates tasks simulating\nhuman behavior on web browsers. However, it is also limited to\nautomate tasks on a single website, which poses a constraint on its\napplicability. While these works have been a source of inspiration,\nthey also highlight the significant challenges of evaluating tasks\non OS. Tasks performed by real-world users are often more com-\nplex and demanding, requiring the collaboration of multiple APPs.\nAdditionally, agents need to consider various constraints such as\nsecurity and user preference. Therefore we propose AndroidArena,\na reproducible mobile environment that allows for cross-APP ac-\ncess. Alongside this, we introduce a new dataset that encapsulates\nthe richness, difficulty, and constraints of instructions. The detail\ncomparison with other works is listed in Table 1.\n2.3\nExisting Metrics for Multi-step Decision\nTo assess the performances of LLM agents, a range of metrics\nare proposed. SmartPlay [31] and LASER [16] employ success\nrate and reward to evaluate task completion. However, these met-\nrics cannot reflect the detailed completion within individual tasks.\nTPTU [21] and Mind2Web [7] incorporate a step-wise action align-\nment method, offering a more nuanced analysis of task completion.\nNevertheless, when applied to multi-step decision-making scenar-\nios where the feasible action sequence for completing a task is not\nunique, this kind of step-wise matching method may introduce in-\naccuracies. In this paper, we propose an adaptive way to accurately\nassess the task completion, and a set of fine-grained ability evalu-\nation metrics to understand the weaknesses of agents, providing\nvaluable insights into improvement directions for LLM agents.\n3\nANDROIDARENA ENVIRONMENT\nIn this section, we introduce the AndroidArena environment, dis-\ntinguished by its vast and dynamic action space, along with its\ncapability to facilitate cross-APP and constrained task execution.\n\n\nUnderstanding the Weakness of Large Language Model Agents within a Complex Android Environment\nConference acronym ’XX, June 03–05, 2018, Woodstock, NY\nSearch contacts\nZara\nJohn\nBob\nAlice\nContacts\nFix & manage\nJ\n+\nZ\nJ\nB\nA\n(a) APP screenshot.\n(b) Compressed observation.\nFigure 1: An example of the Contacts APP page and its corre-\nsponding compressed observation.\nWe begin by offering a formal definition of the mobile task automa-\ntion process, followed by an overview of the system implementa-\ntion. Subsequently, we explore the intricacies of the action space,\nhighlighting its dynamic and expansive nature.\n3.1\nLLM Agent for Mobile Task Automation\nGiven a task presented with a user instruction in natural language,\nthe agent is responsible for making action decisions to complete\nthis instruction on the phone. This process can be formulated as a\nContextual Markov Decision Process (CMDP) [9] ⟨C, S, A, T,𝑟⟩.\nContext 𝑐∈C is the mobile task explicitly expressed as a textual\ninstruction. State 𝑠∈S is the current observed phone state, i.e., the\ndisplayed content on the screen. Action 𝑎∈A can be performed on\nthe current phone screen, e.g., clicks or typing. Transition function\nT (𝑠′|𝑠,𝑎) represents the change in the phone on performing an\naction. Reward 𝑟is awarded for successful completion of the task.\nImplementation. Our implementation is based on UIAutoma-\ntor [8], a UI testing framework that enables direct operations on UI\ncomponents. With UIAutomator, we offer flexible configurations\nto render APP page content (i.e., the observation space) in two\nmodes: 1) the phone screenshot, a pixel-based representation as\nperceived by humans, and 2) the textual XML description of the\nphone screen (depicted in Fig. 8 in §A). It is important to note that,\ngiven the focus of this work on LLM agents, we exclusively utilize\nthe text modality, while acknowledging that our implementation is\ncapable of supporting multi-modal models. Each UI component in\nthe screen corresponds to an XML entry, containing its role (e.g.,\na button), text content, and properties (e.g., if clickable) informa-\ntion. A statistic conducted on eight popular APPs indicates an\naverage token count for the XML exceeding 10,000. Consequently,\ndirectly feeding the XML into the LLM is impractical due to con-\ntext length limitations. To address this challenge, we propose a\ntwo-stage heuristic compression method, involving the removal\nof decision-irrelevant XML tags and the merging of non-visible or\nnon-functional nodes (the detailed algorithm is provided in §A)\nto compress the XML. As illustrated in Fig. 1(b), the compressed\nobservation maintains the hierarchical structure of the original\nXML, enabling the LLM agent to comprehend the UI layout via\ntext. Subsequent to compression, each entry is assigned a unique\nID (e.g.,[nd725]), facilitating agents in locating the UI element. Our\nAtlanta International\nAirport\nConstraint: I prefer to use Email\nrather than instant messaging APPs\nTell Bob the meeting is moved to tomorrow \nExtract the email titled \"flight confirmation\",\nand find the direction to the airport.\nCross-APP Task\nConstrained Task\nFigure 2: Examples of cross-APP and constrained tasks.\nproposed method achieves a compression ratio of 86.6% across sev-\neral tested APPs (please see Table 6 in §A). Motivated by previous\nresearch [12] showing superior performance by regarding LLM as\nreward functions, we employ GPT-4 to quantify the reward 𝑟, and\nvalidate its effectiveness through experiments in §6.2.\n3.2\nVast and Dynamic Action Space\nUnlike prior environments [26, 40] focusing on a single APP and\nonly supporting specific actions, our action space is vast and dy-\nnamic. It is attributed to the fact that a typical APP may feature\nhundreds of UI elements available for manipulation, and these UI\ncomponents exhibit variability owing to real-time internet data ex-\nchange. The vast and dynamic natures are further amplified when\nconsidering all the APPs within AndroidArena. Our designed ac-\ntions can be categorized into four groups: 1) APP-level actions\nare responsible for installing, launching, and stopping APPs; 2)\nComponent-level actions directly operate the UI components such\nas clicking, typing, and swiping etc; 3) System-level actions include\nturning the screen on and off, adjusting the volume, and taking\nscreenshots etc; and 4) Task-level action is issued when the agent\ndeems the task should finish. The complete action space is in §B.\n4\nSCALABLE MOBILE TASK GENERATOR\nThe tasks executed in AndroidArena environment are distinguished\nfrom other benchmarks by incorporating cross-APP collaboration\nand constrained tasks scenarios, commonly encountered in real-life\nbut ignored in existing benchmarks. Even for single-APP tasks,\nexisting benchmarks are either small-scale [36] or derived from\nthe PixelHelp forum [20, 35], a platform dedicated to discussing\nphone-related issues, thus deviating from routine tasks. These short-\ncomings underscore the necessity for a benchmark that exhibits\nhigher scalability and aligns closely with human experiences,\nwhile accounting for cross-APP and constrained tasks.\nThis section outlines our proposed Mobile Task Generator (MTG\nin short), a framework for scalable task construction. MTG not only\naligns with typical human interaction patterns, but also encom-\npasses a diverse array of APP functions, enabling to evaluate the\nagents across a broader spectrum. The constructed benchmark com-\nprises three task categories: single-APP tasks, cross-APP tasks, and\nconstrained tasks. The single-APP and cross-APP tasks are crafted\nto assess the agents’ proficiency in solving general tasks, and more\ncomplex tasks requiring cooperation between two APPs. In con-\ntrast to the former two categories focusing on task completion, the\n\n\nConference acronym ’XX, June 03–05, 2018, Woodstock, NY\nMingzhe Xing, Rongkai Zhang, Hui Xue, Qi Chen, Fan Yang, and Zhen Xiao\nQueries\nGmail and Calendar \ncollaboration features\nHow to use Gmail and \nCalendar  together for tasks\nSearch Engine\nVector Database\nReterival QA\nFunctions of Gmail and Calendar?\nFunctionalities\nTasks\nExtract the date from the email\n titled \"meeting\" and create a calendar event.\n......\nCreate a new Gmail draft with \nthe details of a event titled \"meeting\". \nFunctionality\nto Instruction\nEvol-Instruct\nBenchmark\nHuman Verification\nand Annotation\nCreate email from event\nSend email after \ncreating an event\nCreate event from \nemali content\nAPP Functionalities Extraction\nInstruction Generation and Evolution\nHuman Verification\nand Annotation\nFigure 3: Illustrative example of the MTG workflow for cross-APP (i.e., Gmail and Contacts) tasks construction procedure. The\nsingle-APP tasks are generated with the same process but with different query templates and LLM prompts (please see §F.5).\nTable 2: The statistics of our benchmark.\nTask Type\n#Tasks\nAvg. Len. of Action Sequence\nsingle-APP tasks\n164\n6.13\ncross-APP tasks\n22\n11.14\nconstrained tasks\n35\n6.03\nconstrained tasks are designed to evaluate agents’ proficiency in\ncomprehending predefined constraints. In our benchmark, each\ntask consists of a natural language instruction and a sequence of\nlabeled actions for task completion. Constrained tasks additionally\ninclude a field of constraints represented in natural language. Ex-\namples of cross-APP and constrained tasks are shown in Fig. 2. The\nstatistical information of our benchmark is presented in Table 2.\n4.1\nSingle- and Cross-APP Tasks Construction\nAPP Functionalities Extraction. We incorporate 13 testing APPs\nfrom pre-installed Google suite that are designed to work seam-\nlessly with the Android OS and provide essential services. The\ncomplete APP list can be found in §C. Our objective is to formulate\nthe task instructions that cover rich and diverse functionalities of\nAPPs while aligning with the authentic usage behavior of humans.\nTo achieve this goal, we propose leveraging insights gleaned from\nhuman discussions and shared experiences regarding APPs avail-\nable on the internet. Concretely, we first formulate queries centered\non the usage of specific APPs and employ search engines to retrieve\nrelated webpages. As depicted in Fig. 3, exemplified constructed\nqueries are “Gmail and Calendar collaboration features” and “How\nto use Gmail and Calendar together for tasks”. We then build a vec-\ntor database to store these high volume of webpages containing\nrich functionalities that genuinely engage and concern users. By\nretrieving from the database with LLM and a specific prompt (§F.4),\nwe can extract confined APP functionalities.\nInstruction Generation and Evolution Our next step involves\nutilizing a LLM with a functionality-to-instruction prompt (pro-\nvided in §F.5) to generate initial task instructions grounded in the\nidentified APP functionalities. To automatically mass-produce more\ninstructions, we employ the Evol-Instruct [33] strategy to expand\nthe original instructions. In the application of this strategy, each\nevolutionary iteration involves using LLM along with two prompts,\nnamely in-depth evolving and in-breadth evolving. The in-depth\nevolving prompt encourages LLM to rewrite instructions by mak-\ning them more complex and challenging, while in-breadth evolving\nprompt aims to enhance the feature coverage and overall dataset\ndiversity. Through the iterative execution of multiple evolutions,\nwe sequentially derive evolution datasets, thereby expanding and\nrefining the pool of task instructions.\nHuman Verification and Annotation To construct the bench-\nmark, we engage annotators proficient in operating the testing\nAPPs. They are first instructed to discern and filter tasks exhibit-\ning repetitiveness, ambiguity, or impossibility to complete. Subse-\nquently, they document their interactions with the phone. Given\nthat there might be multiple feasible action sequences for complet-\ning a task, they are encouraged to opt for the most concise action\nplan with the shortest action sequence. After completing a task,\nannotators re-execute the annotated action sequence with a replay\nscript, enabling them easily to verify the accuracy of annotated\naction sequence. Subsequently, the compiled task instructions and\naction demonstrations are collected into the benchmark dataset.\n4.2\nConstrained Tasks Construction\nIn the context of real-world mobile tasks, often confined by spe-\ncific user preferences or security considerations, we introduce a\nconstrained task set to assess the agents’ capability to comprehend\nuser-defined constraints and make decisions adeptly to avoid viola-\ntions. Specifically, we consider three types of constraints: APP-level,\npage-level and component-level constraints. APP-level constraints\ninvolve the preferences of using specific APPs, exemplified by con-\nstraints like “preferring not to use instant message for communica-\ntion”. Page-level constraints restrict access to a specific page, as seen\nin scenarios such as “refraining from entering the label list page in\nGmail due to the presence of sensitive information”. Component-level\nconstraints identify specific UI components as sensitive actions,\ne.g., “do not click the payment button”. It is noteworthy that the\nconstrained tasks are meticulously selected from the single-APP\ntask set and manually labeled with natural language constraints\nalong with the corresponding correct action sequences.\n\n\nUnderstanding the Weakness of Large Language Model Agents within a Complex Android Environment\nConference acronym ’XX, June 03–05, 2018, Woodstock, NY\n5\nEVALUATION METRICS\nDesigning precise metrics is essential for accurately and comprehen-\nsively evaluating agent’s performance. However, existing metrics\nemployed in multi-step decision-making scenarios [7, 21] exhibit\nimprecise and surface-level evaluation issues, which hinder them\nto fully understand the performance and weakness of LLM agent.\nTo address these limitations, we propose a novel set of metrics\nto evaluate agent performance in a more adaptive and precise\nmanner, and to assess fine-grained agent planning abilities.\n5.1\nAdaptive and Precise Task Completion\nEvaluation\nTo begin with, we introduce the notations of action sequences.\nGiven a task, its annotated action sequence can be represented\nas a of length 𝐿, and the actual executed actions is ˆ\na of length ˆ\n𝐿.\nExemplary instances of a and ˆ\na are illustrated as follows:\na =𝐴𝐵𝐶𝐷𝐸𝐹𝐺\n(1)\nˆ\na =𝐴𝑋𝑌𝐵𝑈𝑉𝑊𝐸𝐹𝐹𝐹𝐺𝑍,\n(2)\nwhere each uppercase character denotes a distinct type of action.\nMany existing metrics [7, 21] adopt the step-wise matching method,\nwhich is imprecise in this scenario. In Eq. 2, the agent identifies\nthe correct action 𝐵after two steps of exploration (i.e., 𝑋and 𝑌).\nDespite this action sequence not aligning with the ground truth\n(i.e., Eq. 1) in the step-wise manner, it leads to the correct subse-\nquent step and constitutes a valid action sequence for completing\nthe task. Therefore, previous metrics exhibit inaccuracies in the\nmulti-step decision-making environments where multiple feasible\naction sequences exist. In contrast to previous greedy step-wise\nmatching, we propose to align the two sequences in an adaptive\nway, i.e., calculating their longest common subsequence (LCS)\na𝑙𝑐𝑠(marked in red in Eq. 1 and 2). The LCS accurately and adap-\ntively reflects task completion in the multi-step decision-making\nscenario. Based on the accurate LCS, we propose our metrics to\nevaluate the task completion as follows:\n• Task Reward (TR). 𝑇𝑅= Í𝐿\n𝑖=0 𝛾(𝐿−𝑖)1𝑖, where 𝛾∈[0, 1] is\nthe reward discount factor, 𝛾(𝐿−𝑖) assigns higher rewards to the\nactions that are closer to the final action (e.g., 𝐺in Eq. 1), and 1𝑖\nequals 1 when the 𝑖-th action is in the LCS. This metric considers\nboth the action matching and the distance towards task success.\n• Task Completion Ratio (TCR). 𝑇𝐶𝑅= 𝑘/𝐿, where 𝑘is the\nindex of the last matched action in the LCS. This metrics measure\nthe progress of task completion.\n• Reversed Redundancy Ratio (RRR). 𝑅𝑅𝑅= 𝐿/ˆ\n𝐿. It can be\nused to evaluate the efficiency of the agent completing a task.\nWe inverse it for the convenience of comparison, i.e., the higher\nthis metric, the greater the efficiency of the agent.\n• Success Rate (SR). Unlike the above three metrics relying on\nground truth action sequence, the SR is judged by the GPT-4\nsolely given the trajectory including historical actions and ob-\nservations. SR equals 1 when GPT-4 perceives that the task has\nbeen successfully completed, and 0 when the task is deemed un-\nsuccessful. This metric is devised for the unsupervised scenario,\nenhancing the scalability of the evaluation. In §6.2, we provide\nstatistical evidence to demonstrate the accuracy of SR.\nAlgorithm 1 Deep Q-learning\nInitialize replay memory D and action-value function 𝑄\nfor episode = 1, 𝑀do\nInitialise state 𝜙1 = 𝜙(𝑠1)\nfor 𝑡= 1,𝑇do\nWith probability 𝜖select a random action 𝑎𝑡\n⊲Explore\nthe environment\notherwise select 𝑎𝑡= max𝑎𝑄∗(𝜙(𝑠𝑡),𝑎;𝜃) ⊲Reason the\nnext action\nExecute action 𝑎𝑡in emulator and observe reward 𝑟𝑡and\nstate 𝑠𝑡+1\nPreprocess observation 𝜙𝑡+1 = 𝜙(𝑠𝑡+1) ⊲Understand the\nenvironment and observation\nStore transition (𝜙𝑡,𝑎𝑡,𝑟𝑡,𝜙𝑡+1) in D\nOptimize Q based on a minibatch sampled from D\n⊲\nReflection from experience\nend for\nend for\n5.2\nUnderstand Root Cause with Fine-grained\nAbilities Evaluation\nIn addition to providing adaptive and accurate metrics for evaluat-\ning task completion in complex decision-making scenarios, another\nprimary objective of our study is to investigate the underlying root\ncause contributing to the success or failure of agents planning. Rec-\nognizing that RL serves as a classical and effective approach to ad-\ndress the CMDP problem [9], we abstract the fundamental elements\nand mechanisms in RL agents, and propose fine-grained capabilities\ntailored to assess LLM agents. Here we use the DQN [17] (Algo-\nrithm 1), one of the most classical RL algorithms, as an example.\nWe decompose it into four key dimensions, i.e.,understanding,\nreasoning, exploration, and reflection.\nUnderstanding. The aspect of understanding encompasses the\nagent’s proficiency in comprehending observation and adhering\nto the action format and space specified in the prompt. Unlike RL\nagents confining their output actions strictly within a predefined\naction space, the output space of LLM spans the entire vocabulary.\nThis imposes a great demand on LLM agents to fully understand\nand adhere to the specified action format and space. Additionally,\nconstrained by phone screen size limit, vital information such as\nthe succinct status description of a checkbox, poses challenges\nfor LLM agents in understanding crucial but brief observed details.\nConsequently, to comprehensively gauge the agent’s understanding\nability, we formulate three metrics:\n• Invalid Format. The ratio of outputting actions that deviate\nfrom the format predefined in prompt.\n• Invalid Action. The ratio of outputting actions outside the ac-\ntion space specified in prompt.\n• Nuggets Mining. The ratio of the target element length to the\nentire observation, assessing the agent’s capacity to understand\nthe task context and identify pivotal pieces of information. For\nexample, when the agent correctly selects the Bob as shown in\nFig. 1, the Nuggets Mining can be computed as the division of the\nlength of the [nd725] entry by the total length of the observation.\n\n\nConference acronym ’XX, June 03–05, 2018, Woodstock, NY\nMingzhe Xing, Rongkai Zhang, Hui Xue, Qi Chen, Fan Yang, and Zhen Xiao\nTable 3: Performances evaluated on single-APP and cross-APP tasks.\nCross-APP tasks pose a significant challenge for SOTA agents, and high-\nlight a substantial disparity between GPT-4 and other agents.\nSingle-APP Tasks\nCross-APP Tasks\nModel\nTR\nTCR\nRRR\nSR\nTR\nTCR\nRRR\nSR\nLLaMA2-13B\n0.025\n0.038\n0.007\n0.023\n0.027\n0.084\n0.000\n0.000\nLLaMA2-70B\n0.237\n0.301\n0.047\n0.127\n0.062\n0.089\n0.000\n0.000\nGPT-3.5\n0.413\n0.555\n0.262\n0.449\n0.214\n0.390\n0.021\n0.048\nGPT-4\n0.502\n0.689\n0.755\n0.759\n0.421\n0.746\n0.685\n0.571\nTable 4: The Pearson Correlation Coefficient\nof SR with information richness (IR), and with\noperation complexity (OC), and with the multi-\nplication of IR and OC.\nMetrics\nGPT-3.5\nGPT-4\nIR\n0.37\n0.62\nOC\n0.61\n0.28\nIR × OC\n0.68\n0.57\nReasoning. This dimension indicates the agent’s capacity to de-\nduce the most suitable action based on the current observation. To\nassess it, two metrics are employed:\n• Operation Logic. The inverse number of incorrect actions at-\ntempted before successfully finding the correct action. Consider\nEq. 1 and 2 as an example. The agent correctly executes action\nB after two erroneous attempts, i.e., 𝑋and 𝑌. Therefore, the\nOperation Logic for this subsequence is calculated as 1/2.\n• Awareness of Completion. The ratio of cases that the agent\ncorrectly finds the task completed and issues a finish action.\nExploration. LLM agents make decisions from pretraining-derived\nprior knowledge. Due to the static nature of their prior knowledge,\ncertain LLM agents exhibit a proclivity to iteratively execute the\nsame erroneous action [40]. It precludes them from exploring al-\nternative action pathways to ascertain the correct execution path.\nThis phenomenon reflects the agent’s exploration ability, which we\nquantify by counting the instances of action repetition.\n• Repeat Actions. The ratio of actions resulting in repetitive or\ncyclical patterns.\nReflection. Similar to RL agents, the LLM agents are proven to\nhave the capability to extract insights from previous trials and\nleverage the insights for subsequent executions [22]. We utilize the\nReflexion mechanism to gauge the agent’s proficiency in extracting\npertinent experiences and applying them judiciously.\n• Reflexion@K. 𝑅𝑒𝑓𝑙𝑒𝑥𝑖𝑜𝑛@𝐾= Í𝐾\n𝑖=1(𝑆𝑅𝑖−𝑆𝑅𝑖−1), where 𝐾is\nthe number of Reflexion iterations. It measures the differences\nbetween the original trail and the trail after Reflexion.\nRemark: The four dimensions are not mutually independent. For\ninstance, a prerequisite for reasoning the optimal action is a thor-\nough understanding of the environment and observation. Our objec-\ntive is to assess agent abilities from diverse perspectives rather than\nsegregating them into independent components. It is worth noting\nthat our proposed dimensions and metrics can be generalized to\nother LLM agents, enabling the evaluation of their capabilities in\ndifferent environments. The dimension scores are computed as the\naverage of their corresponding metrics (details provided in §D).\n6\nEXPERIMENTS AND FINDINGS\nIn this section, we setup the experiments, and present the experi-\nmental results. We summarize noteworthy findings as follows. First,\nexisting SOTA agents still exhibit substantial room for improve-\nment (§6.2). Second, in contrast to the results observed in prior\nbenchmarks [6, 15], LLaMA2-70B exhibits inferior planning\nabilities across various dimensions. GPT-4, while advanced,\nrequires further improvement in the exploration and reflec-\ntion dimensions. (§6.3).\n6.1\nEvaluation Setting\nWe conduct experiments on SOTA open-source and closed-source\nLLMs. The detailed experiment settings are introduced as follows.\nAgent Models. The selected LLMs encompass GPT-{3.5-turbo,\n4} [18], LLaMA2-{13B-chat, 70B-chat} [25], representing two power-\nful closed-source and open-source LLM model families, respectively.\nRegarding the prompt settings for LLM agents, please refer to §F.\nMax Step. We set maximum step limits for agents to evaluate\ntheir capabilities of completing tasks within reasonable timeframe.\nAccording to the length of action sequences as shown in Table 2,\nwe empirically set the maximum step limit as 15 for single-APP and\nconstrained tasks, while for cross-APP tasks, it is set as 30.\n6.2\nPoor Performance in Mobile Tasks\nIn this section, we integrate the metrics introduced in §5.1 to assess\nthe task completion of LLM agents across various task types. We re-\nport the results across single-APP, cross-APP, and constrained tasks\nin Table 3 and Table 5. Recall that the Success Rate (SR) is assessed\nby GPT-4. To validate its reliability, we perform cross-validation\nbetween it with TR and TCR, where TR and TCR represent alter-\nnative perspectives on task completion. Specifically, we compute\nthe Pearson Correlation Coefficient (PCC) [5] between SR and TR,\nresulting in a correlation of 0.87, and between SR and TCR, yielding\na correlation of 0.91. These high coefficients indicate a substan-\ntial correlation between SR and both TR and TCR, validating the\nrationale of adopting the GPT-4 judgment mechanism.\nTable 3 reveals a significant deficiency of SOTA agents in\nthe real-world mobile tasks. While GPT-4 achieves a 75.9% SR\non single-APP tasks, all agents exhibit an inability to make effective\ndecisions across other task settings. Noteworthy the performance\ngap between GPT-4 and GPT-3.5 is much larger for cross-APP tasks\nthan single-APP tasks. It indicates that the cross-APP tasks are\nmore complex and difficult, and can well reveal the significant\ndisparity in planning abilities between the two agents. In\ncontrast to prior benchmark studies, LLaMA2-70B demonstrates\ninferior performance relative to GPT-3.5 and GPT-4.\nWe conduct a detailed examination of the APPs where the SOTA\nagents, including GPT-3.5 and GPT-4, do not perform well. Our\ninvestigation reveals a vulnerability in handling APPs charac-\nterized by deficient textual information and intricate opera-\ntional logics. To further substantiate this observation, we calculate\n\n\nUnderstanding the Weakness of Large Language Model Agents within a Complex Android Environment\nConference acronym ’XX, June 03–05, 2018, Woodstock, NY\nTable 5: Constraints violation ratios for Constrained Tasks.\nModel\nAPP-level\nPage-level\nComponent-level\nGPT-3.5\n0.207\n0.072\n0.33\nGPT-4\n0.000\n0.050\n0.00\nUnderstanding\nReasoning\nExploration\nReflection\nLLaMA2-13B\nLLaMA2-70B\nGPT-3.5\nGPT-4\nFigure 4: Agent abilities evaluation on cross-APP tasks.\nthe PCC between SR and the information richness (IR) and opera-\ntion complexity (OC) of APPs. Specifically, we utilize the average\nlength of APP observation and the inverse length of ground truth\nactions to quantify IR and OC, respectively. The results in Table 4\ndemonstrate that OC poses a more substantial challenge for GPT-3.5\nin achieving a higher SR, while GPT-4 exhibits a greater sensitivity\nto IR. The high values of IR × OC further prove our findings.\nBeyond basic task completion, we assess the agents’ capacity to\ncomprehend constraints and adeptly make decisions to avoid vio-\nlations. Table 5 presents the constraint violation ratios of GPT-3.5\nand GPT-4. LLaMA2 models are excluded as they face challenges\nin completing basic tasks, rendering this assessment impractical.\nTable 5 reveals that even for straightforward constraints, GPT-3.5\nstill may violate them. By reading its intermediate reasoning pro-\ncesses, we discern that GPT-3.5 lacks awareness and understanding\nof constraints. For instance, in the case of “Find the current weather\nforecast” with the constraint “do not use the Weather APP”, GPT-3.5\ndirectly opens the Weather APP, while GPT-4 comprehends the con-\nstraint and devises an alternative way using a web browser to search\nfor weather forecast. It highlights the considerable distance yet\nto be covered before GPT-3.5 can be applied effectively in\npermission-sensitive environments.\n6.3\nFour Weakness Leading to Failure\nIn this section, we employ the metrics introduced in §5.2 to quan-\ntify the fine-grained planning abilities of these agents, so as to\nunderstand their weaknesses that lead to failure. As shown in Fig. 4,\nGPT-4 shows superior performances across various dimensions,\nfurther substantiating its excellence in task completion, as indi-\ncated in Table 3. In contrast, LLaMA2 models exhibit significant\nweaknesses across all four dimensions. In Fig. 5, we present the\ncomposed metrics of these dimensions. Due to space limit, we only\npresent part of the testing APPs, and the complete APP metrics can\nbe found in §E. Fig. 5(a) and 5(b) present the ratios of outputting\ninvalid format and out-of-space actions. The notably higher ratios\nof LLaMA2 show its challenges in understanding and adhering to\n0.0\n0.1\n0.2\n0.3\n0.4\n0.5\n0.6\n0.7\nCalendar\nCamera\nFirefox\nGmail\nGoogle Maps\nMessages\nSettings\nWeather\nYouT\nube\nLLaMA2-13B\nLLaMA2-70B\nGPT-3.5\nGPT-4\n(a) Invalid Action Ratio.\n0.000\n0.005\n0.010\n0.015\n0.020\n0.025\nCalendar\nCamera\nFirefox\nGmail\nGoogle Maps\nMessages\nSettings\nWeather\nYouT\nube\nLLaMA2-13B\nLLaMA2-70B\nGPT-3.5\nGPT-4\n(b) Invalid Format Ratio.\n0.0\n0.2\n0.4\n0.6\n0.8\n1.0\nCalendar\nCamera\nFirefox\nGmail\nGoogle Maps\nMessages\nSettings\nWeather\nYouT\nube\nLLaMA2-13B\nLLaMA2-70B\nGPT-3.5\nGPT-4\n(c) Nuggets Mining Score.\n0.0\n0.2\n0.4\n0.6\n0.8\n1.0\nCalendar\nCamera\nFirefox\nGmail\nGoogle Maps\nMessages\nSettings\nWeather\nYouT\nube\nLLaMA2-13B\nLLaMA2-70B\nGPT-3.5\nGPT-4\n(d) Operation Logic Score.\n0.0\n0.2\n0.4\n0.6\n0.8\n1.0\nCalendar\nCamera\nFirefox\nGmail\nGoogle Maps\nMessages\nSettings\nWeather\nYouT\nube\nLLaMA2-13B\nLLaMA2-70B\nGPT-3.5\nGPT-4\n(e) Aware of Completion Ratio.\n0.0\n0.1\n0.2\n0.3\n0.4\n0.5\n0.6\n0.7\nCalendar\nCamera\nFirefox\nGmail\nGoogle Maps\nMessages\nSettings\nWeather\nYouT\nube\nLLaMA2-13B\nLLaMA2-70B\nGPT-3.5\nGPT-4\n(f) Repeat Action Ratio.\nFigure 5: Metrics for understanding, reasoning and explo-\nration dimensions.\nprescribed action rules. Fig. 5(c) demonstrates the superior capacity\nof GPT-3.5 and GPT-4 to apprehend more nuanced information com-\npared to LLaMA2. In Fig. 5(d), LLaMA2 agents exhibit challenges\nin identifying the correct subsequent actions even after multiple\nattempts. Moreover, LLaMA2-13B lacks the awareness that the task\nhas been successfully completed, as depicted in Fig. 5(e). Fig. 5(f)\nindicates a high repeat action ratio of LLaMA2, underscoring its\nlimited ability to explore the environment. GPT-4 also demon-\nstrates a notable proclivity for repeating erroneous actions\nfor several APPs. To improve the exploration ability of GPT-4,\nwe introduce an exploration strategy and examine its impact on\nperformance in §7.2. While GPT-4 shows certain improvement\nthrough Reflexion, our analysis suggests that it stems from in-\nherent opportunities for additional attempts to complete\nthe task rather than an enhancement in the agent’s policy.\nDetailed experiments and analysis can be found in §7.1.\n7\nFUTURE DIRECTIONS FOR ENHANCING\nLLM AGENT\nThrough experiments in §6.3, we observe that LLaMA2 models\ndisplay weaknesses across all four dimensions. Even for the lead-\ning model, GPT-4, still exhibits shortcomings in exploration and\nreflection. In this section, we first analyze the ineffectiveness of\nReflexion and provide an empirical analysis of the factors contribut-\ning to this phenomenon. Second, we propose a novel prompt-based\nexploration method, revealing that explicitly encouraging the agent\nto explore unknown actions can enhance performance.\n\n\nConference acronym ’XX, June 03–05, 2018, Woodstock, NY\nMingzhe Xing, Rongkai Zhang, Hui Xue, Qi Chen, Fan Yang, and Zhen Xiao\n0\n1\n2\n3\n4\n5\nTrial Number\n0.05\n0.10\n0.15\n0.20\n0.25\nReflection\nRe-execute\n(a) SR of GPT-3.5 agent.\n0\n1\n2\n3\n4\n5\nTrial Number\n0.60\n0.65\n0.70\n0.75\n0.80\n0.85\n0.90\nReflection\nRe-execute\n(b) SR of GPT-4 agent.\nFigure 6: Performances evaluated on cross-APP tasks by in-\ncreasing the reflection times.\n7.1\nAnalysis of Reflection’s Failure\nRecall that Reflexion summarizes experience and then re-executes\nfailed tasks, it inherently offers opportunities for additional at-\ntempts and possesses potential for performance improvement. Ac-\ncordingly, we conduct a comparative evaluation with re-executing\nfailed tasks without the Reflexion process, namely Re-execute. In\nFig. 6, we present the SR of Reflexion@5 and Re-execute@5. Con-\ntrary to expectations, we observe that Reflexion does not yield\npositive outcomes compared to Re-execute. This unexpected\nphenomenon motivates an investigation of the underlying mecha-\nnisms of Reflexion and the challenges of applying it in our scenario.\nTo initiate our investigation, we provide a formal definition of the\nReflexion process, specified as follows:\n𝑃(new trajectory | reflection)\n|                                 {z                                 }\n③\n· 𝑃(reflection |\n①\nz           }|           {\nold trajectory)\n|                                {z                                }\n②\n,\nwhere 𝑃denotes the LLM agents. This equation describes the Reflex-\nion process, i.e., extracting valuable insights from past trajectories\nand benefiting subsequent trials. Three key steps in this equation\nmay contribute to the degradation of Reflexion performance in our\nscenario. The first and the most important reason is that the\nold trajectory (marked in ①) is less informative compared with\nprevious scenarios. Unlike benchmarks [3, 37] characterized by one-\nstep decision, and virtual ALFWorld environment [23] with small\nand static action space, our environment necessitates multi-step\nplanning within a vast and dynamic action space. This challenge\nnot only makes it hard to explore the entire action space, thus can-\nnot provide sufficient guidance for next trial, but also sparsifies\nthe reward feedback. Therefore, a potential improvement can be\nachieved by improving trace quality. In specific, employing explo-\nration strategies to broaden the explored action space for informa-\ntive experience [16] and devising intrinsic rewards to mitigate the\nsparse reward issue [39]. Secondly, a constrained ability to distill\nreflection (i.e., part ②) diminishes the reflection efficacy. Lastly, re-\ngarding part ③, the reflection may not be fully leveraged by agent\nor, conversely, introduces bias [10] and degrades the performance\ncompared to the Re-execute that is without reflection.\n7.2\nEnhancing Exploration Boosts Performance\nUpon reading the trajectories, we observe that even for GPT-4, it\nstill presents a pronounced tendency to repeat erroneous actions,\n5\n15\n25\n35\n45\nMax Step\n0.20\n0.25\n0.30\n0.35\n0.40\n0.45\n0.50\n0.55\n0.60\nSuccess Rate\nGPT-4\nGPT-4+Exploration\n(a) Success rate.\n5\n15\n25\n35\n45\nMax Step\n0.2\n0.3\n0.4\n0.5\n0.6\nRepeat Action\nGPT-4\nGPT-4+Exploration\n(b) Repeat action ratio.\nFigure 7: GPT-4 and GPT-4+Exploration comparisons by vary-\ning the maximum step limit on the Camera APP.\nas illustrated in Fig. 5(f), indicating its limited exploration capabili-\nties. Furthermore, the repetition of actions degrades the quality of\npreceding trajectories, rendering them insufficient for providing\ninformative guidance for reflection, as discussed in §7.1.\nIn this section, we introduce a novel prompt-based exploration\nstrategy for LLM agents. Diverging from prior approaches [4] that\ntreat the LLM as a RL policy network and employ exploration strate-\ngies originating from RL, our strategy guides the exploration of\nLLM agent by incorporating a prompt indicating the count\nof previously visited observations 𝑀(s) and issued actions\n𝑁(s, a). Specifically, we embed a hint prompt such as “You have\nalready been in the current state M times, and taken action A for N\ntimes” at each decision step. This concept is inspired by the Upper\nConfidence Bound (UCB) [2, 30]. Unlike UCB, we do not design\nexplicit exploration strategies. Instead, we integrate historical infor-\nmation into the prompt, leveraging the powerful decision-making\ncapabilities of the LLM to balance exploration and exploitation.\nWe conduct an experiment on the Camera APP, where GPT-4 ex-\nhibits the highest repeat action ratio, to evaluate the effectiveness\nof the exploration strategy. We vary the maximum step limit in\n{5, 15, 25, 35, 45}, and present SR and repeat action ratio in Fig. 7.\nThe results show that, with a simple counting-based prompt, SR\ncan achieve 27% of improvement. Furthermore, as the maximum\nstep limit increases, the exploration ability of GPT-4 degrades. In\ncontrast, for GPT-4+Exploration, effective exploration of the envi-\nronment persists, leading to continued performance improvement.\n8\nCONCLUSION\nThis study introduces AndroidArena environment and a scalable\nbenchmark. It supports the evaluation of cross-APP and constrained\ntask scenarios. We propose adaptive and precise metrics to assess\ntask completion, and fine-grained abilities of agents to understand\ntheir weaknesses. The results underscore significant room for im-\nprovement among SOTA agents. We highlight four research direc-\ntions for enhancing LLM agents. Additionally, we offer empirical\ninsights into the failure of reflection and present a novel method\nto enhance the exploration capabilities of agents. In the future, we\nplan to investigate the weaknesses of multi-modal model agents.\nGiven that vision models excel at spatial understanding and reason-\ning, areas where LLMs struggle [34], we intend to scrutinize their\nfine-grained abilities and identify promising research directions in\nthis domain. Our AndroidArena supports multi-modal evaluation,\nand the benchmark can be easily extended to this setting.\n\n\nUnderstanding the Weakness of Large Language Model Agents within a Complex Android Environment\nConference acronym ’XX, June 03–05, 2018, Woodstock, NY\nREFERENCES\n[1] Michael Ahn, Anthony Brohan, Noah Brown, Yevgen Chebotar, Omar Cortes,\nByron David, Chelsea Finn, Chuyuan Fu, Keerthana Gopalakrishnan, Karol Haus-\nman, et al. 2022.\nDo as i can, not as i say: Grounding language in robotic\naffordances. arXiv preprint arXiv:2204.01691 (2022).\n[2] Peter Auer, Nicolo Cesa-Bianchi, and Paul Fischer. 2002. Finite-time analysis of\nthe multiarmed bandit problem. Machine learning 47 (2002), 235–256.\n[3] Jacob Austin, Augustus Odena, Maxwell Nye, Maarten Bosma, Henryk\nMichalewski, David Dohan, Ellen Jiang, Carrie Cai, Michael Terry, Quoc Le,\net al. 2021. Program synthesis with large language models. arXiv preprint\narXiv:2108.07732 (2021).\n[4] Thomas Carta, Clément Romac, Thomas Wolf, Sylvain Lamprier, Olivier Sigaud,\nand Pierre-Yves Oudeyer. 2023. Grounding large language models in interactive\nenvironments with online reinforcement learning. arXiv preprint arXiv:2302.02662\n(2023).\n[5] Israel Cohen, Yiteng Huang, Jingdong Chen, Jacob Benesty, Jacob Benesty, Jing-\ndong Chen, Yiteng Huang, and Israel Cohen. 2009. Pearson correlation coefficient.\nNoise reduction in speech processing (2009), 1–4.\n[6] Nicholas Crispino, Kyle Montgomery, Fankun Zeng, Dawn Song, and Chenguang\nWang. 2023. Agent Instructs Large Language Models to be General Zero-Shot\nReasoners. arXiv preprint arXiv:2310.03710 (2023).\n[7] Xiang Deng, Yu Gu, Boyuan Zheng, Shijie Chen, Samuel Stevens, Boshi Wang,\nHuan Sun, and Yu Su. 2023. Mind2Web: Towards a Generalist Agent for the Web.\narXiv preprint arXiv:2306.06070 (2023).\n[8] S Gunasekaran and V Bargavi. 2015. Survey on automation testing tools for\nmobile applications. International Journal of Advanced Engineering Research and\nScience 2, 11 (2015), 2349–6495.\n[9] Assaf Hallak, Dotan Di Castro, and Shie Mannor. 2015. Contextual markov\ndecision processes. arXiv preprint arXiv:1502.02259 (2015).\n[10] Jie Huang, Xinyun Chen, Swaroop Mishra, Huaixiu Steven Zheng, Adams Wei Yu,\nXinying Song, and Denny Zhou. 2023. Large language models cannot self-correct\nreasoning yet. arXiv preprint arXiv:2310.01798 (2023).\n[11] Wenlong Huang, Pieter Abbeel, Deepak Pathak, and Igor Mordatch. 2022. Lan-\nguage models as zero-shot planners: Extracting actionable knowledge for embod-\nied agents. In International Conference on Machine Learning. PMLR, 9118–9147.\n[12] Minae Kwon, Sang Michael Xie, Kalesha Bullard, and Dorsa Sadigh. 2023. Reward\ndesign with language models. arXiv preprint arXiv:2303.00001 (2023).\n[13] Jinyang Li, Binyuan Hui, Ge Qu, Binhua Li, Jiaxi Yang, Bowen Li, Bailin Wang,\nBowen Qin, Rongyu Cao, Ruiying Geng, et al. 2023. Can llm already serve as a\ndatabase interface? a big bench for large-scale database grounded text-to-sqls.\narXiv preprint arXiv:2305.03111 (2023).\n[14] Yuanchun Li, Hao Wen, Weijun Wang, Xiangyu Li, Yizhen Yuan, Guohong Liu,\nJiacheng Liu, Wenxing Xu, Xiang Wang, Yi Sun, et al. 2024. Personal LLM Agents:\nInsights and Survey about the Capability, Efficiency and Security. arXiv preprint\narXiv:2401.05459 (2024).\n[15] Xiao Liu, Hao Yu, Hanchen Zhang, Yifan Xu, Xuanyu Lei, Hanyu Lai, Yu Gu,\nHangliang Ding, Kaiwen Men, Kejuan Yang, et al. 2023. Agentbench: Evaluating\nllms as agents. arXiv preprint arXiv:2308.03688 (2023).\n[16] Kaixin Ma, Hongming Zhang, Hongwei Wang, Xiaoman Pan, and Dong Yu. 2023.\nLASER: LLM Agent with State-Space Exploration for Web Navigation. arXiv\npreprint arXiv:2309.08172 (2023).\n[17] Volodymyr Mnih, Koray Kavukcuoglu, David Silver, Alex Graves, Ioannis\nAntonoglou, Daan Wierstra, and Martin Riedmiller. 2013. Playing atari with deep\nreinforcement learning. arXiv preprint arXiv:1312.5602 (2013).\n[18] R OpenAI. 2023. GPT-4 technical report. arXiv (2023), 2303–08774.\n[19] Chen Qian, Xin Cong, Cheng Yang, Weize Chen, Yusheng Su, Juyuan Xu, Zhiyuan\nLiu, and Maosong Sun. 2023. Communicative agents for software development.\narXiv preprint arXiv:2307.07924 (2023).\n[20] Christopher Rawles, Alice Li, Daniel Rodriguez, Oriana Riva, and Timothy Lilli-\ncrap. 2023. Android in the wild: A large-scale dataset for android device control.\narXiv preprint arXiv:2307.10088 (2023).\n[21] Jingqing Ruan, Yihong Chen, Bin Zhang, Zhiwei Xu, Tianpeng Bao, Guoqing\nDu, Shiwei Shi, Hangyu Mao, Xingyu Zeng, and Rui Zhao. 2023. Tptu: Task\nplanning and tool usage of large language model-based ai agents. arXiv preprint\narXiv:2308.03427 (2023).\n[22] Noah Shinn, Federico Cassano, Ashwin Gopinath, Karthik R Narasimhan, and\nShunyu Yao. 2023. Reflexion: Language agents with verbal reinforcement learning.\nIn Thirty-seventh Conference on Neural Information Processing Systems.\n[23] Mohit Shridhar, Xingdi Yuan, Marc-Alexandre Côté, Yonatan Bisk, Adam\nTrischler, and Matthew Hausknecht. 2020. Alfworld: Aligning text and em-\nbodied environments for interactive learning. arXiv preprint arXiv:2010.03768\n(2020).\n[24] Richard S Sutton, Andrew G Barto, et al. 1999. Reinforcement learning. Journal\nof Cognitive Neuroscience 11, 1 (1999), 126–134.\n[25] Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yas-\nmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhos-\nale, et al. 2023. Llama 2: Open foundation and fine-tuned chat models. arXiv\npreprint arXiv:2307.09288 (2023).\n[26] Daniel Toyama, Philippe Hamel, Anita Gergely, Gheorghe Comanici, Amelia\nGlaese, Zafarali Ahmed, Tyler Jackson, Shibl Mourad, and Doina Precup. 2021.\nAndroidEnv: A Reinforcement Learning Platform for Android. abs/2105.13231\n(2021). arXiv:2105.13231 [cs.LG] http://arxiv.org/abs/2105.13231\n[27] Boshi Wang, Xiang Yue, and Huan Sun. 2023. Can ChatGPT Defend its Belief in\nTruth? Evaluating LLM Reasoning via Debate. In Findings of the Association for\nComputational Linguistics: EMNLP 2023. 11865–11881.\n[28] Guanzhi Wang, Yuqi Xie, Yunfan Jiang, Ajay Mandlekar, Chaowei Xiao, Yuke Zhu,\nLinxi Fan, and Anima Anandkumar. 2023. Voyager: An open-ended embodied\nagent with large language models. arXiv preprint arXiv:2305.16291 (2023).\n[29] Lei Wang, Chen Ma, Xueyang Feng, Zeyu Zhang, Hao Yang, Jingsen Zhang,\nZhiyuan Chen, Jiakai Tang, Xu Chen, Yankai Lin, et al. 2023. A survey on large\nlanguage model based autonomous agents. arXiv preprint arXiv:2308.11432 (2023).\n[30] Yingpeng Wen, Qinliang Su, Minghua Shen, and Nong Xiao. 2022. Improving\nthe exploration efficiency of DQNs via the confidence bound methods. Applied\nIntelligence (2022), 1–15.\n[31] Yue Wu, Xuan Tang, Tom M Mitchell, and Yuanzhi Li. 2023. SmartPlay: A\nBenchmark for LLMs as Intelligent Agents. arXiv preprint arXiv:2310.01557\n(2023).\n[32] Zhiheng Xi, Wenxiang Chen, Xin Guo, Wei He, Yiwen Ding, Boyang Hong, Ming\nZhang, Junzhe Wang, Senjie Jin, Enyu Zhou, et al. 2023. The rise and potential\nof large language model based agents: A survey. arXiv preprint arXiv:2309.07864\n(2023).\n[33] Can Xu, Qingfeng Sun, Kai Zheng, Xiubo Geng, Pu Zhao, Jiazhan Feng,\nChongyang Tao, and Daxin Jiang. 2023. Wizardlm: Empowering large language\nmodels to follow complex instructions. arXiv preprint arXiv:2304.12244 (2023).\n[34] Yutaro Yamada, Yihan Bao, Andrew K Lampinen, Jungo Kasai, and Ilker Yildirim.\n2023. Evaluating Spatial Understanding of Large Language Models. arXiv preprint\narXiv:2310.14540 (2023).\n[35] An Yan, Zhengyuan Yang, Wanrong Zhu, Kevin Lin, Linjie Li, Jianfeng Wang,\nJianwei Yang, Yiwu Zhong, Julian McAuley, Jianfeng Gao, et al. 2023. Gpt-4v in\nwonderland: Large multimodal models for zero-shot smartphone gui navigation.\narXiv preprint arXiv:2311.07562 (2023).\n[36] Zhao Yang, Jiaxuan Liu, Yucheng Han, Xin Chen, Zebiao Huang, Bin Fu, and\nGang Yu. 2023. Appagent: Multimodal agents as smartphone users. arXiv preprint\narXiv:2312.13771 (2023).\n[37] Zhilin Yang, Peng Qi, Saizheng Zhang, Yoshua Bengio, William W Cohen, Ruslan\nSalakhutdinov, and Christopher D Manning. 2018. HotpotQA: A dataset for di-\nverse, explainable multi-hop question answering. arXiv preprint arXiv:1809.09600\n(2018).\n[38] Shunyu Yao, Jeffrey Zhao, Dian Yu, Nan Du, Izhak Shafran, Karthik Narasimhan,\nand Yuan Cao. 2022. React: Synergizing reasoning and acting in language models.\narXiv preprint arXiv:2210.03629 (2022).\n[39] Haoqi Yuan, Chi Zhang, Hongcheng Wang, Feiyang Xie, Penglin Cai, Hao Dong,\nand Zongqing Lu. 2023. Plan4mc: Skill reinforcement learning and planning for\nopen-world minecraft tasks. arXiv preprint arXiv:2303.16563 (2023).\n[40] Shuyan Zhou, Frank F Xu, Hao Zhu, Xuhui Zhou, Robert Lo, Abishek Sridhar,\nXianyi Cheng, Yonatan Bisk, Daniel Fried, Uri Alon, et al. 2023. Webarena:\nA realistic web environment for building autonomous agents. arXiv preprint\narXiv:2307.13854 (2023).\n\n\nConference acronym ’XX, June 03–05, 2018, Woodstock, NY\nMingzhe Xing, Rongkai Zhang, Hui Xue, Qi Chen, Fan Yang, and Zhen Xiao\nAPPENDIX\nA\nDETAILED OBSERVATION COMPRESSION\nMETHOD\nThe textual observation is derived from the XML representation\nencapsulating comprehensive screen information as shown in Fig. 8.\nHowever, directly inputting the entire XML into the LLM, proves\nto be excessively lengthy as illustrated in Table 6. To mitigate this,\nwe employ a two-phase heuristic approach for compressing the\nXML to a manageable length for LLM processing. The XML en-\ntries are categorized into two groups: one for layout, which does\nnot support actionable operations, and the other for UI compo-\nnents. We eliminate the XML entries related to layout, retaining\nonly those associated with UI components. In the second phase,\nwe merge non-functional and non-visible nodes upwards, incor-\nporating their descriptive information into the parent nodes. This\nstrategy enhances the LLM’s ability to understand the semantic of\nthe hierarchical XML tree, and result in a more efficient compres-\nsion. For components with nuanced state descriptions, we amplify\ntheir textual information. For instance, when the switch compo-\nnent is in the off position, we append a description stating “it is\ncurrently unchecked, and you can switch it on.\". To enable the agent\nto accurately select the UI component for operation, a unique ID is\nassigned to each component in the compressed observation. In the\ncompressed observation, components are structurally organized,\nmaintaining their ancestral-descendant relationships in the original\nXML tree, aiding the LLM agent in comprehending the interface’s\nlayout through text and thereby enhancing its command efficacy.\nTable 6: We randomly select several APPs and compare the to-\nken numbers before and after compression. Our compression\nratio reach a high of 86.6%, while preserving the semantic\ninformation. This approach enhances the utilization of the\nLLM agent context, allowing for the accommodation of more\nhistorical observations in each decision-making process.\nApp Name\n#Token (Original)\n#Token (Compressed)\nGmail (email list)\n11,707\n1,155\nGmail (compose email)\n7,273\n413\nCalendar\n8,604\n584\nGoogle map\n15,725\n637\nYouTube\n12,005\n939\nPlay Store\n10,450\n620\nGoogle drive\n11,060\n651\nClock Alarm\n9,633\n505\nClock\n7,980\n285\nB\nDETAILED ACTION SPACE\nWe support four-level’s action space, i.e., APP level, component\nlevel, system level and task level. App level actions are responsible\nfor installing, launching and stopping APPs. Most actions are com-\nponent level which are responsible for operating UI components,\nsuch as clicking, typing, and swiping. We also support system level\nactions including turning the screen on and off, adjusting the vol-\nume, setting orientation, and taking screenshots. Task-level action\nis designed for the agent to decide if a task should finish.\nTable 7: The complete action space, including action type and\nthe corresponding parameters.\nAction level\nAction Type\nAction Parameters\nAPP level\nInstall APP\nDownload link\nLaunch APP\nPackage name\nStop APP\nPackage name\nStop all APP\nComponent level\nClick\nXPath\nDouble click\nXPath\nLong click\nXPath\nSet text\nXPath, Text\nSwipe up/down/left/right\nNumber\nPress back\nPress home\nSystem level\nScreen on/off\nVolume up/down/mute\nSet orientation\nHorizontal/vertical\nScreenshot\nTask Level\nFinish task\nIn previous work like AndroidEnv, the action is done by suc-\ncessive touches and lifts, each consists a position (𝑥,𝑦) and an\n𝐴𝑐𝑡𝑖𝑜𝑛𝑇𝑦𝑝𝑒∈{𝑇𝑂𝑈𝐶𝐻, 𝐿𝐼𝐹𝑇, 𝑅𝐸𝑃𝐸𝐴𝑇}. AndroidEnv divides the\nscreen into a grid and restricts the ActionType to TOUCH, or groups\naction sequences like [TOUCH, LIFT, TOUCH, LIFT] into a single\ngesture, such as swiping, scrolling, or drag-and-drop. However, con-\ntinuous touches and lifts bring additional inference overhead for\nagents, and cannot accurately simulate the continuity and smooth-\nness of swiping. Instead of interacting with the phone by successive\ntouches and lifts, we directly operate the UI components of APPs\nthrough UIAutomator. It is a testing framework for Android, send-\ning a series of events including pressing, dragging, and scrolling.\nThese events are consistent with real finger slides. Operating com-\nponents by sending action events is not only more accurate and\nnatural in simulating real user operations, but also superior in terms\nof APP compatibility. We can get the executable actions that each\ncomponent can perform from the corresponding XML and main-\ntain them in the compressed observation, such as clickable, double\nclickable, long clickable, etc. At the same time, we also record the\ntype of each component in the compressed observation, such as\nbutton, text-editor, which can assist the agent to give appropriate\naction instructions. As we have set a unique ID for each component\nin the compressed observation, the agent can operate a compo-\nnent by specifying its ID and the corresponding action type. Since\nUIAutomator locates and operates on components based on their\nXPath, our implementation employs a mapping table to convert\ncomponent IDs into component XPaths, after which we perform\nthe operations.\nC\nBENCHMARK STATISTICS\nOur benchmark comprises three types of tasks: single-APP, cross-\nAPP, and constrained tasks. The single-APP tasks are derived from\n13 APPs including Calendar, Camera, Clock, Contacts, YouTube,\nWeather, Settings, Photos, Messages, Google Maps, Google Drive,\nGmail and Firefox. Statistical information regarding the benchmark\nis illustrated in Fig. 9.\n\n\nUnderstanding the Weakness of Large Language Model Agents within a Complex Android Environment\nConference acronym ’XX, June 03–05, 2018, Woodstock, NY\nSearch contacts\nZara\nJohn\nBob\nAlice\nContacts\nFix & manage\nJ\n+\nZ\nJ\nB\nA\n(a) Screenshot.\n(b) XML derived by UIAutomator.\n[n0236] EditText open_search_bar ;click ;Search contacts : \n    [n54c6] ImageButton ;click ;Open navigation drawer : \n    [nb90c] ImageView ;click ;More options : \n    [nc3e7] selected_account_disc ;click ;Signed : \n[n1c26] ListView list ;long-click ; 4 contacts : \n    [nea52] ViewGroup ;click ;Alice : \n    [nd725] ViewGroup ;click ;Bob : \n    [nca5w] ViewGroup ;click ;John : \n    [ndsf3] ViewGroup ;click ;Zara : \n[n22ea] ImageButton floating_action_button ;click ;Create contact : \n[nf05c] contacts ;focusable selected ;Contacts : \n[n4eed] nav_manage ;click ;Fix & manage : \n​\n(c) Compressed observation.\nFigure 8: An example of the screenshot, original XML and compressed observation of Contacts APP page.\nCalendar\n3.3%\nCamera\n6.5%\nClock\n6.5%\nContacts\n5.1%\nFirefox\n2.3%\nGmail\n8.4%\nGoogle Drive\n5.6%\nGoogle Maps\n6.5%\nMessages\n6.5%\nPhotos\n2.8%\nSettings\n8.8%\nWeather\n6.5%\nYouT\nube\n4.7%\ncross-APP\n10.2%\nConstrained\n16.3%\nFigure 9: Proportion of instructions for different Apps\nD\nPLANNING ABILITY SCORES\nIn this section, we present a detailed explanation of the calculation\nmethod for the four fine-grained planning ability scores introduced\nin §5.2. The value ranges of these metrics are [0, 1], with the\nexception of Nuggets Mining, Operation Logic, and Reflexion@K.\nFollowing AgentBench [15], we normalize the latter three metrics to\n[0, 1] across all models. It is important to note that smaller metrics\nin understanding and exploration indicate better performance in\nthese dimensions, while larger values for the metrics in reasoning\nand reflection denote superior performance in those aspects. The\ncalculations for the four dimensions are specified as follow:\n𝑈𝑛𝑑𝑒𝑟𝑠𝑡𝑎𝑛𝑑𝑖𝑛𝑔= (1 −𝐼𝑛𝑣𝑎𝑙𝑖𝑑𝐹𝑜𝑟𝑚𝑎𝑡𝑅𝑎𝑡𝑖𝑜)+\n(1 −𝐼𝑛𝑣𝑎𝑙𝑖𝑑𝐴𝑐𝑡𝑖𝑜𝑛𝑅𝑎𝑡𝑖𝑜) + (1 −𝑁𝑢𝑔𝑔𝑒𝑡𝑠𝑀𝑖𝑛𝑖𝑛𝑔)\n𝑅𝑒𝑎𝑠𝑜𝑛𝑖𝑛𝑔= 𝑂𝑝𝑒𝑟𝑎𝑡𝑖𝑜𝑛𝐿𝑜𝑔𝑖𝑐+ 𝐴𝑤𝑎𝑟𝑒𝑛𝑒𝑠𝑠𝑜𝑓𝐶𝑜𝑚𝑝𝑙𝑒𝑡𝑖𝑜𝑛\n𝐸𝑥𝑝𝑙𝑜𝑟𝑎𝑡𝑖𝑜𝑛= 1 −𝑅𝑒𝑝𝑒𝑎𝑡𝐴𝑐𝑡𝑖𝑜𝑛𝑅𝑎𝑡𝑖𝑜\n𝑅𝑒𝑓𝑙𝑒𝑐𝑡𝑖𝑜𝑛= 𝑅𝑒𝑓𝑙𝑒𝑥𝑖𝑜𝑛@𝐾\nSubsequently, we standardize the four dimension scores and present\nthem in Fig. 4.\nE\nCOMPLETE RESULTS FOR TESTING APPS\nDue to space limit, we present part of the testing APPs in §6.2. In\nFig. 10, we show the metrics for all testing APPs.\n0.0\n0.1\n0.2\n0.3\n0.4\n0.5\n0.6\n0.7\nCalendar\nCamera\nClock\nContacts\nFirefox\nGmail\nGoogle Drive\nGoogle Maps\nMessages\nPhotos\nSettings\nWeather\nYouT\nube\nLLaMA2-13B\nLLaMA2-70B\nGPT-3.5\nGPT-4\n(a) Invalid Action Ratio.\n0.000\n0.005\n0.010\n0.015\n0.020\n0.025\nCalendar\nCamera\nClock\nContacts\nFirefox\nGmail\nGoogle Drive\nGoogle Maps\nMessages\nPhotos\nSettings\nWeather\nYouT\nube\nLLaMA2-13B\nLLaMA2-70B\nGPT-3.5\nGPT-4\n(b) Invalid Format Ratio.\n0.0\n0.2\n0.4\n0.6\n0.8\n1.0\nCalendar\nCamera\nClock\nContacts\nFirefox\nGmail\nGoogle Drive\nGoogle Maps\nMessages\nPhotos\nSettings\nWeather\nYouT\nube\nLLaMA2-13B\nLLaMA2-70B\nGPT-3.5\nGPT-4\n(c) Nuggets Mining Score.\n0.0\n0.2\n0.4\n0.6\n0.8\n1.0\nCalendar\nCamera\nClock\nContacts\nFirefox\nGmail\nGoogle Drive\nGoogle Maps\nMessages\nPhotos\nSettings\nWeather\nYouT\nube\nLLaMA2-13B\nLLaMA2-70B\nGPT-3.5\nGPT-4\n(d) Operation Logic Score.\n0.0\n0.2\n0.4\n0.6\n0.8\n1.0\nCalendar\nCamera\nClock\nContacts\nFirefox\nGmail\nGoogle Drive\nGoogle Maps\nMessages\nPhotos\nSettings\nWeather\nYouT\nube\nLLaMA2-13B\nLLaMA2-70B\nGPT-3.5\nGPT-4\n(e) Aware of Completion Ratio.\n0.0\n0.1\n0.2\n0.3\n0.4\n0.5\n0.6\n0.7\nCalendar\nCamera\nClock\nContacts\nFirefox\nGmail\nGoogle Drive\nGoogle Maps\nMessages\nPhotos\nSettings\nWeather\nYouT\nube\nLLaMA2-13B\nLLaMA2-70B\nGPT-3.5\nGPT-4\n(f) Repeat Action Ratio.\nFigure 10: Metrics for understanding, reasoning and explo-\nration dimensions on all testing APPs.\nF\nPROMPT DESIGN\nFollowing WebArena, our prompt for each decision-making step\nincorporates environment descriptions, two-shot examples, task\ninstructions, and historical observations and actions, as illustrated\n\n\nConference acronym ’XX, June 03–05, 2018, Woodstock, NY\nMingzhe Xing, Rongkai Zhang, Hui Xue, Qi Chen, Fan Yang, and Zhen Xiao\nYou are an autonomous intelligent agent tasked with operating a\nmobile phone. \nYou are able to assist with a wide range of tasks, ...... \nHere's the information you'll have:\n......\nTo be successful, it is very important to follow the following rules:\n......\ninput:\nUser's objective: open the email from Kaggle in Gmail.\nPrevious observations and actions ......\noutput: ......\ninput:\nUser's objective: open the email from Kaggle in Gmail.\nPrevious observations and actions ......,\noutput: ......\nREMEMBER to think step by step, and generate the next action in the\ncorrect format. \n......\nNow, begin!\nUser's objective: ......\nPrevious observations and actions ......\nEnvironment\nDescription\nTwo-shot\nExample\nAction\nPrompt\nThinking: \nThe user wants to find the best walking route to \"Microsoft SVC\nBuilding\". The current view is the Google Maps home page. The\nsearch box is available with the id [na928]. I should input the\ndestination \"Microsoft SVC Building\" into the search box. So, the next\naction is #set_text [na928] [Microsoft SVC Building]#.\nAgent\nOutput\nAgent\nInput\nFigure 11: Prompt structure.\nin Fig. 11. We adopt the Reflexion prompt from its official imple-\nmentation with modifications tailored to our specific scenario. The\nprompt context limit is 4K for LLaMA2-13B, LLaMA2-70B, and\nGPT-3.5, and at 8K for GPT-4. Given that historical observations\nand actions may exceed the context limit, and to ensure a fair com-\nparison, we apply the truncation strategy employed in WebArena\nacross all agents to maintain a prompt within the 4K context limit.\nF.1\nEnvironment Description Prompt\nEnvironment Description Prompt\nYou are an autonomous intelligent agent tasked with op-\nerating a mobile phone. You are able to assist with a wide\nrange of tasks, from answering simple questions to plan-\nning and executing a complicated instruction with specific\nactions you can issue.\nHere’s the information you’ll have:\nThe user’s objective: This is the task you’re trying to com-\nplete.\nThe installed APPs: These are the APPs you can operate\non.\nThe current phone’s observation: This is a simplified and\nstructured representation of the phone view, providing key\ninformation.\nThe previous action and observation : There are the action\nyou just performed and the resulted phone observation. It\nmay be helpful to track your progress.\nSolve the user’s task with interleaving Observation,\nThought, Action steps.\nThought can reason about the current situation.\nAt the end of thinking process, you MUST response the\nnext Action in the following formats:\n1. APP level Actions:\n#start [app-name]#: This action start an APP specified by\napp name. You can ONLY issue the start operation on the\nfollowing APPs:\n{app-string}\n2. Component level Actions:\n#click [id]#: This action clicks on an element with a specific\nid on the APP page.\n#long-click [id]#: This action long clicks on an element\nwith a specific id on the APP page.\n#set-text [id] [text]# This action set text in a text view\nelement with a specific id on the APP page.\nNote that the UI elements with ’clickable’ or ’long-clickable’\nproperties can be issued with #click#, while the elements\nwith ’EditText’ can be issued with #set-text# action.\n3. System level Actions:\n#swipe-up#: Scroll up the screen.\n#swipe-down#: Scroll down the screen.\n#swipe-left#: Swipe left the screen.\n#swipe-right#: Swipe right the screen.\n#press-back#: Navigate to the previously viewed page.\n#press-enter#: Press enter.\n4. Completion Action:\n#finish [answer]#: Issue this action when you believe the\ntask is complete. If the objective is to find a text-based\nanswer, provide the answer in the bracket. If you believe\nthe task is impossible to complete, provide the answer as\n\"N/A\" in the bracket.\n——\nObservation is the simplified and structured text represen-\ntation of APP view.\nTo be successful, it is very important to follow the follow-\ning rules:\n1. You MUST only issue ONE next action in each thinking\nprocess.\n2. Generate the action in the correct format. Always put\nthe action inside a pair of #. For example, #click [node3]#.\n3. Issue finish action when you think you have achieved\nthe objective.\n4. Today is {date}, which might be useful for you to com-\nplete the task.\n\n\nUnderstanding the Weakness of Large Language Model Agents within a Complex Android Environment\nConference acronym ’XX, June 03–05, 2018, Woodstock, NY\nF.2\nReflection Prompt\nReflection Prompt\nYou are an advanced reasoning agent that can improve\nbased on self reflection. You will be given a previous rea-\nsoning trial in which you were given access to operate an\nAndroid phone environment with human-like actions in-\ncluding click and type text on the phone screen, and a task\ninstruction to complete. You were unsuccessful in complet-\ning the task either because you made the wrong action\ndecisions, or you used up your set number of reasoning\nsteps. In a few sentences, Diagnose a possible reason for\nfailure and devise a new, concise, high level plan that aims\nto mitigate the same failure. Use complete sentences.\nF.3\nReward Prompt\nReward Prompt\nYou can access to the actions and phone states at some steps\nduring executing a specific task on a phone. Check if the\ngiven phone states and actions indicate the achievement of\na goal. The phone state is represented as structured texts,\nwith each entry denoting a UI component along with its\ncontent and function description.\nThe goal is {goal},\nthe actions and states at some steps are: {traj}\nPlease check if the above trajectory indicate the achieve-\nment of the goal: goal. Only output ’Yes’ or ’No’, no other\nwords.\nF.4\nQuery Template\nQueries for single-APP functionalities extraction\nQueries for single-APP\nhow to use {app_name}\n{app_name} usage instructions\n{app_name} quick start guides\n{app_name} cheat sheets\n{app_name} productivity guides\nuse {app_name} step-by-step\ntips and tricks for {app_name}\n{app_name} for beginners\n{app_name} tutorial\ngetting started with {app_name}\nintroduction to {app_name}\nQueries for cross-APP functionalities extraction\nQueries for cross-APP\n{app_name1} and {app_name2} collaboration features\nHow to use {app_name1} and {app_name2} together for\ntasks\nIntegration between {app_name1} and {app_name2} for\nproductivity\nCollaborative task management with {app_name1} and\n{app_name2}\n{app_name1} and {app_name2} integration for work and\nproductivity\nProductivity tips with {app_name1} and {app_name2}\nF.5\nFunctionality-to-Instruction Prompt\nFunctionality-to-Instruction Prompt\nYou are a smart task creator for a smartphone intelligent\nassistant. Given the features description of the {app} APP,\nyour goal is to generate clear and practical tasks that the\nassistant can assist people with while they use {app} on\ntheir phone in their daily lives. These tasks should encom-\npass a wide range of possible instructions and questions\nthat may arise when using {app} APP.\nFor example, for the Gmail APP, potential task instructions\ncould include:\nCompose an email with the subject <email subject> and\nthe message content <email content> to be sent to <email\naddress> using Gmail.,\nSend the first draft email.,\nOpen the latest email from <email address> in Gmail.,\nOpen Gmail settings.,\nTurn off notifications for Gmail.,\nStar the latest email from <email address> in Gmail.,\nDelete the latest email from <email address> in Gmail.,\netc., where the placeholders surrounded with angle brack-\nets ’<’ and ’>’ should be automated generated and not be\nfilled with specific content.\nThe {app} APP’s feature description is: {feature}\nYour task is to generate as many of these tasks as possible\nfor the {app} app. Ensure that these instructions are clear\nand will not lead to any misunderstanding so that the assi-\ntant can successfully execute them. Your response should\nbe a list of comma separated task instructions, where each\ninstruction should be presented in one sentence.","difficulty":"hard","domain":"Multi-Document QA","length":"short","question":"Regarding the comparison between these two articles, which of the following is correct?","sub_domain":"Academic"}

Source: https://huggingface.co/datasets/zai-org/LongBench-v2

initial import

Posting: /agents

GET /api/v1/write?intent=publish&task_id=92d2892e-ce0b-58a1-b42f-365a9eae8514&body={url_encoded_text}&agent_name={optional_name}&nonce={optional_random_id}