# LongBench v2 / 66eae4de5a08c7b9b35dd12d

task_id: 7ee065cb-121a-599b-80af-a7b3ddeb1f9a
task_key: train--66eae4de5a08c7b9b35dd12d
task_revision_id: 3

{"choice_A":"Segment and label objects within the scene using SoM and represent the functional parts by vectors.","choice_B":"Understand and execute higher level and multi-step instruction.","choice_C":"Process accurate coordinates of object parts generated by VLMs to form a 3D grounding.","choice_D":"Generate the appropriate 6-DoF grasp pose for the specified objects of interest and show a higher success rate than Voxposer.","context":"CoPa: General Robotic Manipulation through\nSpatial Constraints of Parts with Foundation Models\nInitial Observation\nHammer the nail.\nTask-Oriented\nGrasping\nModule\nObservation after Grasping\nPost-Grasp Poses\n{𝑃\n1, 𝑃2, … , 𝑃𝑁}\nPose 𝑃\n1\nPose 𝑃2\n“Find scissors”\n“Press button”\n“Open drawer”\n“Pour water”\n“Put eraser into drawer”\n“Insert flower into vase”\n“Put glasses onto shelf”\n“Put spoon into cup”\n“Sweep nuts”\nTask-Aware\nMotion Planning\nModule\nFig. 1: Overview. We present CoPa, a novel framework that utilizes common sense knowledge embedded within VLMs\nfor robotic low-level control. Left. Our pipeline. Given an instruction and scene observation, CoPa first generates a grasp\npose through Task-Oriented Grasping Module (detailed in Fig. 3). Subsequently, a Task-Aware Motion Planning Module\n(detailed in Fig. 4) is utilized to obtain post-grasp poses. Right. Examples of real-world experiments. Boasting a fine-grained\nphysical understanding of scenes, CoPa can generalize to open-world scenarios, handling open-set instructions and objects\nwith minimal prompt engineering and without the need for additional training.\nAbstract— Foundation models pre-trained on web-scale data\nare shown to encapsulate extensive world knowledge beneficial\nfor robotic manipulation in the form of task planning. However,\nthe actual physical implementation of these plans often relies on\ntask-specific learning methods, which require significant data\ncollection and struggle with generalizability. In this work, we\nintroduce Robotic Manipulation through Spatial Constraints\nof Parts (CoPa), a novel framework that leverages the com-\nmon sense knowledge embedded within foundation models to\ngenerate a sequence of 6-DoF end-effector poses for open-\nworld robotic manipulation. Specifically, we decompose the\nmanipulation process into two phases: task-oriented grasping\nand task-aware motion planning. In the task-oriented grasping\nphase, we employ foundation vision-language models (VLMs) to\nselect the object’s grasping part through a novel coarse-to-fine\ngrounding mechanism. During the task-aware motion planning\nphase, VLMs are utilized again to identify the spatial geometry\nconstraints of task-relevant object parts, which are then used to\nderive post-grasp poses. We also demonstrate how CoPa can be\nseamlessly integrated with existing robotic planning algorithms\nto accomplish complex, long-horizon tasks. Our comprehensive\nreal-world experiments show that CoPa possesses a fine-grained\nphysical understanding of scenes, capable of handling open-set\ninstructions and objects with minimal prompt engineering and\nwithout additional training. Project page: copa-2024.github.io\n∗The first two authors contributed equally.\n1 Institute of Interdisciplinary Information Sciences, Tsinghua University.\n2 Shanghai Qi Zhi Institute.\n3 Shanghai Jiao Tong University.\n4 Shanghai Artificial Intelligence Laboratory.\nI. INTRODUCTION\nDeveloping a general-purpose robot necessitates effective\napproaches in two critical areas: (i) high-level task planning,\nwhich determines what to do next, and (ii) low-level robotic\ncontrol, focusing on the precise actuation of joints [1], [2].\nThe emergence of high-capacity foundation models [3], [4],\npre-trained on extensive web-scale datasets, has inspired a\nsurge of recent research efforts aimed at integrating these\nmodels into robotics [5], [6]. Nonetheless, these methods\ngenerally address only the “higher level” aspects of task\nplanning [7]–[10]. In contrast, the prevailing approach for\nlow-level control continues to revolve around crafting task-\nspecific policies via diverse learning methods [11], [12].\nSuch policies, however, are brittle and prone to failure when\nencountering unseen scenarios [13]. Even the largest robotics\nmodels struggle outside environments they have previously\nencountered [14], [15].\nThe question then arises: what makes generalizable low-\nlevel robotic control so hard? We attempt to answer this\nquestion through the lens of human object manipulation. For\ninstance, when an individual is tasked with hammering a\nnail, regardless of their familiarity with the specific hammer,\nthey intuitively grasp it by the handle (instead of the head),\nadjust its orientation so the striking surface aligns with the\nnail, and then execute the strike. This process underscores\narXiv:2403.08248v1  [cs.RO]  13 Mar 2024\n\n\nthe importance of a fine-grained understanding of the phys-\nical properties of task-related objects, or more broadly, the\nextensive common sense knowledge of the world that fa-\ncilitates generalizable object manipulation. Some pioneering\nworks [16], [17] have sought to leverage the rich semantic\nknowledge of Internet-scale foundation models to enhance\nlow-level robotic control. Yet, these approaches are heavily\ndependent on intricate prompt engineering and suffer from a\nfundamental limitation: a coarse understanding of the scene,\nleading to failures in tasks requiring fine-grained physical\nunderstanding. Such a detailed understanding is essential for\nnearly all real-world robotic tasks of interest.\nTo endow robots with fine-grained physical understand-\ning, we propose Robotic Manipulation through Spatial\nConstraints of Parts (CoPa), a novel framework that incor-\nporates common sense knowledge embedded within foun-\ndation vision-language models (VLMs), such as GPT-4V,\ninto the robotic manipulation tasks. We observe that most\nmanipulation tasks require a part-level, fine-grained physical\nunderstanding of objects within the scene. Hence, we design\na coarse-to-fine grounding module to identify task-relevant\nparts. Then, to leverage VLMs for aiding the robotic low-\nlevel control, it is necessary to design an interface that not\nonly allows VLMs to reason in the form of language but\nalso facilitates robot’s object manipulation. Therefore, we\npropose utilizing spatial constraints as a bridge between\nVLMs and robots. Specifically, we utilize VLMs to generate\nthe spatial constraints that task-relevant parts must meet to\naccomplish the task, and then employ a solver to determine\nthe robot’s poses based on these constraints. Finally, to\nensure the precise execution of the robot’s actions, transitions\nbetween adjacent poses are achieved through traditional\nmotion planning methods.\nWe demonstrate that CoPa is capable of completing every-\nday manipulation tasks with a high success rate through ex-\ntensive real-world experiments. Attributed to the innovative\ndesign of coarse-to-fine grounding and constraint generation\nmodule, CoPa possesses a profound physical understanding\nof the environment and can generate precise 6-Dof poses to\ncomplete complex manipulation tasks, significantly surpass-\ning a strong baseline VoxPoser [16].\nOur contributions are summarized as follows:\n• We propose CoPa, a novel framework that utilizes\nthe common sense knowledge of VLMs for low-level\nrobotic control, which can handle open-set instructions\nand objects with minimal prompt engineering and with-\nout additional training.\n• Through extensive real-world experiments, CoPa is\ndemonstrated to possess the capability to complete\nmanipulation tasks that require a fine-grained under-\nstanding of physical properties of task-relevant objects,\nsignificantly surpassing baselines.\n• We show that CoPa can be seamlessly integrated with\nhigh-level planning methods to accomplish complex,\nlong-horizon tasks (e.g. make pour-over coffee and set\nup romantic table).\nII. RELATED WORK\nLearning for Robotic Manipulation. Manipulation is a\ncritical and challenging aspect in the robotic field. Nu-\nmerous studies harness imitation learning (IL) from expert\ndemonstrations to acquire manipulation skills [14], [15],\n[18]–[22]. Despite IL’s conceptual simplicity and its no-\ntable success across a broad spectrum of real-world tasks,\nit struggles with out-of-distribution samples and demands\nconsiderable effort in collecting expert data. Reinforcement\nlearning (RL) [12] emerges as another principal approach [7],\n[23]–[26], enabling robots to develop manipulation skills via\ntrial-and-error interactions with their environment. However,\nRL’s sample inefficiency limits its applicability in real-world\nsettings, leading most robotic systems to rely on sim-to-real\ntransfers [27]–[29]. Nonetheless, sim-to-real approaches ne-\ncessitate the construction of specific simulators and confront\nthe sim-to-real gap. Furthermore, policies learned via these\nend-to-end learning methods often lack generalization to new\ntasks. In contrast, by leveraging foundation models’ common\nsense knowledge, our CoPa can generalize to open-world\nscenarios without additional training.\nFoundation Models For Robotics. In recent years, foun-\ndation models have dramatically transformed the landscape\nof robotics [5]. Many works employ vision models, pre-\ntrained on large-scale image datasets, to generate visual\nrepresentations for visuomotor control tasks [20], [30]–[34].\nSome other studies utilize foundation models for reward\nspecification in reinforcement learning [35]–[40]. Further-\nmore, numerous studies have leveraged foundation models\nfor robotic high-level planning, achieving remarkable suc-\ncess [7], [8], [10], [41]–[49]. There is also a body of works\nthat employs foundation models for low-level control [14],\n[15], [21], [22]. Some works fine-tune vision-language mod-\nels (VLMs) to directly output robot actions. However, such\nfine-tuning approaches require extensive amounts of expert\ndata. To address this issue, Code as Policies [17] applies large\nlanguage models (LLMs) to write code to control robots,\nand VoxPoser [16] generates robot trajectories by producing\nvalue maps based on foundation models. Nevertheless, these\nmethods rely on complex prompt engineering and possess\nonly a coarse understanding of the scene. In stark contrast,\nbenefiting from the rational use of common sense knowledge\nwithin VLMs, our method exhibits a fine-grained understand-\ning of scenarios and generalizes to open-world scenarios\nwithout additional training, requiring only minimal prompt\nengineering.\nIII. METHOD\nIn this section, we first introduce the formulation of\nmanipulation tasks in Section III-A. Then, we describe\ntwo critical components within our framework — the task-\noriented grasping in Section III-B and the task-aware motion\nplanning in Section III-C.\nA. Promblem Formulation\nMost manipulation tasks can be decomposed into two\nphases: the initial grasp of the object and the subsequent\nmotion required to complete the task. For example, opening\n\n\nFind grasping objects.\nHammer the nail.\nCurrent Image\nUser\nSoM\nUser\nFind grasping parts.\nHammer the nail.\nGrasping Objects\nFine-Grained\nPart Grounding\nGrasping Parts\nGPT-4V\nCoarse-Grained\nObject Grounding\nFig. 2: Grounding Module. This module is utilized to identify the grasping part for task-oriented grasping or task-relevant\nparts for task-aware motion planning. The grounding process is divided into two stages: coarse-grained object grounding\nand fine-grained part grounding. Specifically, we first segment and label objects within the scene using SoM. Then, in\nconjunction with the instruction, we employ GPT-4V to select the grasping/task-relevant objects. Finally, similar fine-grained\npart grounding is applied to locate the specific grasping/task-relevant parts.\na drawer involves grasping the handle and pulling it in\na straight line, while picking up a water glass requires\nfirst seizing the glass and then lifting it. Motivated by this\nobservation, we structure our approach into two modules:\ntask-oriented grasping and task-aware motion planning.\nAdditionally, we posit that the execution of robotic tasks\nessentially entails generating a series of target poses for the\nrobot’s end-effector. The transition between adjacent target\nposes can be achieved through motion planning.\nGiven a language instruction l and the initial scene obser-\nvation O0 (RGB-D images), our objective in the task-oriented\ngrasping module is to generate the appropriate grasp pose for\nthe specified objects of interest. This process is represented\nas P0 = f(l, O0). We denote the observation after the\nrobot reaches P0 as O1. For the task-aware motion planning\nmodule, our goal is to derive a sequence of post-grasp poses,\nexpressed as g(l, O1) −\n→{P1, P2, ..., PN}, where N is the\ntotal number of poses required to complete the task. After\nacquiring the target poses, the robot’s end-effector can reach\nthese poses utilizing motion planning algorithms such as\nRRT* [50] and PRM* [51].\nB. Task-Oriented Grasping\nTo generate the task-oriented grasp pose, our approach\ninitially employs a grasping model to produce grasp pose\nproposals, and filter out the most feasible one through our\nnovel grasping part grounding module. The entire process is\ndepicted in Fig. 3.\nGrasp Pose Proposals. We leverage a pre-trained grasping\nmodel for generating grasp pose proposals. To achieve this,\nwe first convert RGB-D images into point clouds by back-\nprojecting them into 3D space. These point clouds are then\ninput into GraspNet [52], a model trained on a vast dataset\ncomprising over one billion grasp poses. GraspNet outputs 6-\nDOF grasp candidates, including information on grasp point,\nwidth, height, depth, and a “graspness score,” which indicates\nthe likelihood of a successful grasp. However, given that\nGraspNet yields all potential grasps within a scene, it is\nnecessary for us to employ a filtering mechanism that selects\nPose\nFiltering\nCurrent Observation\nUser\nGrasp Pose 𝑃\n0\nGrasp Pose Candidates\nFind the grasping part.\nHammer the nail.\nGrounding\nModule\nFig. 3: Task-Oriented Grasping Module. This module is\nemployed to generate grasp poses. Grasp pose candidates\nare generated from the scene point cloud using GraspNet.\nConcurrently, given the instruction and the scene image, the\ngrasping part is identified by a grounding module (detailed in\nFig. 2). Ultimately, the final grasp pose is selected by filtering\ncandidates based on the grasping part mask and GraspNet\nscores.\nthe optimal grasp based on the specific task outlined by the\nlanguage instruction.\nGrasping Part Grounding. Humans grasp specific parts of\nan object corresponding to the intended use. For instance,\nwhen grasping a knife for cutting, we hold onto the handle\nrather than the blade; similarly, when picking up glasses, we\ngrasp the frame instead of the lenses. This process essentially\nrepresents the application of common sense knowledge by\nhumans. To mimic this ability, we utilize vision-language\nmodels (VLMs), such as GPT-4V [53], which incorporate\nvast amounts of common sense knowledge [10], [54], to\nidentify the appropriate part of an object to grasp.\nWe employ a two-stage process to ground language in-\nstructions to the specific parts of objects intended for grasp-\ning: coarse-grained object grounding and fine-grained part\ngrounding. The entire grounding process is shown in Fig.\n2. At both stages, we incorporate a recent visual prompting\nmechanism known as Set-of-Mark (SoM) [55]. SoM lever-\nages segmentation models to partition an image into distinct\nregions, assigning a numeric marker to each, significantly\n\n\nUser\nGrounding\nModule\n3D Processing\nTask-Relevant\n3D Components\nCurrent Observation\nSpatial Constraints:\n1. Vector 2 and Vector 3 are colinear.\n2. Point 2 is 5 cm above Point 3.\n3. Vector 2 points downward.\n4. Vector 1 is parallel to the table \nsurface.\nSubsequent Actions:\n1. Move vertically down 7 cm.\nPost-Grasp Poses\n{𝑃\n1, 𝑃2, … , 𝑃𝑁}\nConstraint\nGeneration\nSolver\nFind task-relevant \ngeometry parts.\nHammer the nail.\nFig. 4: Task-Aware Motion Planning Module. This module is used to obtain a series of post-grasp poses. Given the instruction\nand the current observation, we first employ a grounding module (detailed in Fig. 2) to identify task-relevant parts within the\nscene. Subsequently, these parts are modeled in 3D, and are then projected and annotated onto the scene image. Following\nthis, VLMs are utilized to generate spatial constraints for these parts. Finally, a solver is applied to calculate the post-grasp\nposes based on these constraints.\nboosting the visual grounding capabilities of VLMs. During\nthe coarse-grained object grounding phase, SoM is utilized\nat the object level to detect and label all objects within the\nscene. Following this, VLMs are tasked with pinpointing the\ntarget object for grasping (e.g., a hammer), guided by the\nuser’s instructions. The selected object is then cropped from\nthe image, upon which fine-grained part grounding is applied\nto determine the specific part of the object to be grasped (e.g.,\nthe handle of the hammer). This coarse-to-fine design endows\nour method with fine-grained physical understanding ability,\nenabling generalization across complex scenarios. Finally,\nwe filter the grasp pose candidates, projecting all the grasp\npoints onto the image and retaining only those within the\ngrasping part mask. From these, the pose with the highest\nconfidence scored by GraspNet is selected as the ultimate\ngrasp pose P0 for execution.\nC. Task-Aware Motion Planning\nAfter successfully executing task-oriented grasping, now\nwe aim to obtain a series of post-grasp poses. We divide\nthis step into three modules: task-relevant part grounding,\nmanipulation constrains generation and target pose planning.\nThe entire process is shown in Fig. 4.\nTask-Relevant Part Grounding. Similar to the previous\ngrasp part grounding module, we use coarse-grained object\ngrounding and fine-grained part grounding to locate task-\nrelevant parts. Here we need to identify multiple task-relevant\nparts (e.g. the hammer’s striking surface, handle and the\nnail’s surface). Additionally, we observe that numeric marks\non the robotic arm may affect VLMs’ selection, so we filter\nout the masks on the robotic arm (detailed in the Appendix).\nManipulation Constraints Generation. During the execu-\ntion of tasks, task-relevant objects are often subject to various\nspatial geometric constraints. For instance, when charging\na phone, the charger’s connector must be aligned with the\ncharging port; similarly, when capping a bottle, the lid must\nbe positioned directly above the mouth of the bottle. These\nconstraints inherently necessitate common sense knowledge,\nwhich includes a profound comprehension of the physical\nproperties of objects. We aim to leverage VLMs to generate\nspatial geometric constraints for the object manipulated by\nthe robot.\nWe first model identified task-relevant parts as simple\ngeometric elements. Specifically, we represent slender parts\n(e.g. hammer handle) as vectors, while other parts are\nmodeled as surfaces. For the parts modeled as vectors, we\ndirectly draw them on the scene image; for those modeled\nas surfaces, we ascertain their center points and normal\nvectors, which are then projected and marked on the 2D\nscene image. The annotated image is used as input for\nVLMs, which are prompted to generate spatial constraints for\nthese geometric elements. We craft a set of descriptions for\nspatial constraints, such as collinearity between two vectors,\nperpendicularity between a vector and a surface, and so\nforth. We instruct the VLMs to first generate the constraints\nnecessary for the first target pose, followed by the subsequent\nactions required after reaching that pose. Fig. 4 provides an\nillustrative example of this process. Implementation details\nof this process are provided in the Appendix.\nTarget Pose Planning. Upon obtaining manipulation con-\nstraints, we proceed to derive the sequence of post-grasp\nposes. This is equivalent to computing a sequence of SE(3)\nmatrices such that, when applied to the parts of the object\nmanipulated by the robotic arm, these parts satisfy the spatial\ngeometric constraints. We operate under the assumption that\nthe object part under manipulation and the robotic end-\neffector together constitute a rigid body. Consequently, these\ncalculated SE(3) transformations can be directly applied to\nthe robotic end-effector. We formalize the computation of\nthe SE(3) matrix as a constrained optimization problem.\nSpecifically, we compute a loss for each constraint, and\nthen a nonlinear constraint solver is used to find the SE(3)\nmatrix that minimizes the sum of these losses. Taking the\nconstraint “Vector 2 points downward” from Fig. 4 as an\nexample, the loss can be defined as the negative dot product\nof the normalized Vector 2 after SE(3) transformation and\nthe vector (0, 0, −1). After obtaining the first target pose,\nwe solve for subsequent poses in alignment with the actions\nspecified by VLMs. Concretely, we sequentially compute\na new pose corresponding to each subsequent action. For\nexample, for the action “Move vertically down 7 cm,” we\nsimply subtract 7 cm from the current pose on the z-\n\n\naxis. This process results in a complete set of post-grasp\nposes {P1, P2, ..., PN}, with the transitions between adjacent\nposes facilitated by motion planning algorithms. The detailed\nprocess for solving the SE(3) matrix and a comprehensive\ndescription of the subsequent actions can be found in the\nAppendix.\nIV. EXPERIMENTS\nWe first introduce the experimental setup in Section IV-\nA. Subsequently, we evaluate the performance of CoPa in\nreal-world manipulation tasks in Section IV-B. Then we\nhighlight CoPa’s intriguing properties by comparing it with\nthe baseline VoxPoser [16] in Section IV-C. We further\npresent an ablation study to analyze the contribution of key\nmodules within our framework in Section IV-D. Finally, we\ndemonstrate that CoPa can be seamlessly integrated with\nhigh-level task planning methods to accomplish complex\nlong-horizon tasks in Section IV-E.\nA. Experimental Setup\nHardware. We set up a real-world tabletop environment. We\nuse a Franka Emika Panda robot (a 7-DoF arm) and a 1-DoF\nparallel jaw gripper. For perception, we mount two RGB-D\ncameras (Intel RealSense D435) at two opposite ends of the\ntable and calibrate them.\nTasks and Evaluations. We design 10 real-world manipu-\nlation tasks, each demanding a comprehensive understand-\ning of the physical properties of objects. See Fig. 1 for\nillustrations of the tasks. For each task, we evaluate all\nmethods across 10 different variations of the environment,\nwhich encompass alterations in object types and their ar-\nrangements. Detailed descriptions of the tasks are provided\nin the Appendix.\nVLMs and Prompting. We employ GPT-4V from OpenAI\nAPI as the VLM. CoPa involves minimal few-shot prompts\nto aid VLMs in comprehending their roles. Additionally,\nthe chain-of-thought technique [56] is utilized to facilitate a\ndeeper understanding of the scene by VLMs. The full prompt\nis provided in the Appendix.\nBaselines. We compare with Voxposer [16], a method ca-\npable of synthesizing closed-loop robot trajectories without\nnecessitating additional training through the utilization of a\nseries of foundational models. Following Huang et al [16],\nwe employ GPT-4 from OpenAI API as the LLM, and utilize\nthe open-vocabulary detector Owl-ViT [57] and Segment\nAnything [58] for perception.\nB. CoPa for Real-World Manipulation\nWe study whether CoPa can generate robot trajectories\nto perform real-world manipulation tasks. The quantitative\nresults are detailed in Table I, while the Appendix showcases\nadditional qualitative outcomes, including visualizations of\npart grounding results and manipulation constraints. We\nfind that CoPa achieves a remarkable success rate of 63%\nacross ten different tasks, significantly outperforming the\nVoxPoser baseline and various ablation variants (detailed\nin the following sections). A key factor in CoPa’s superior\nperformance is its leverage of common sense knowledge em-\nbedded in VLMs, which enables a fine-grained understanding\nGrasp stem of flower\nGrasp flower\nMove hammer to top of nail\nAlign striking surface with nail Insert spoon vertically down\nRotate to face the cup\nOurs\nVoxPoser\nFig. 5: Comparison with VoxPoser. We illustrate the exe-\ncution of CoPa (top) and VoxPoser (bottom), demonstrating\nthat CoPa possesses a fine-grained physical understanding\nof scenes and can effectively handle rotation DoF. The tasks\nfrom left to right are sequentially Insert flower into\nvase, Hammer nail, Put spoon into cup.\nof objects’ physical properties during both part grounding\nand constraint generation phases. For example, in the part\ngrounding phase, CoPa accurately identifies the need to grasp\nthe protective cover of an eraser in the Put eraser on\nshelf task, and recognizes the stem of the flower and the\nrim of the vase as critical parts in the Insert flower\ninto vase task. During the constraint generation phase,\nCoPa comprehends that the spoon can be inserted vertically\ndown into the cup in the Put spoon into cup task, and\nthat the wooden stick needs to be aligned directly with the\nbutton in the Press button task.\nC. Understanding Properties of CoPa\nIn this section, we delve deeper into CoPa, shedding light\non its intriguing properties through a comparative analysis\nwith Voxposer, another method that utilizes the common\nsense knowledge embedded in foundation models to synthe-\nsize robot trajectories. CoPa exhibits significant advantages\nin the following three aspects:\nFine-Grained Physical Understanding. Many manipula-\ntion tasks require a nuanced physical understanding of the\nscene, which necessitates not only identifying object parts\nwith fine granularity but also comprehending their intricate\nattributes. CoPa excels in this aspect, employing a coarse-to-\nfine part grounding module to select grasping/task-relevant\nobject parts, and then utilizing VLMs to provide their spatial\ngeometry constraints. In contrast, Voxposer only perceives\nobjects in the scene as a whole. This coarse-grained level\nof comprehension often leads to failure in tasks that require\nprecise operations. For instance, in the Insert flower\ninto vase task (shown in Fig. 5 left), CoPa grasps the\nstem of the flower, whereas Voxposer seizes the petals. In the\nHammer nail task (shown in Fig. 5 middle), CoPa orients\nthe hammer to align precisely with the nail, while Voxposer\noverlooks this fine-grained physical constraint, treating the\nhammer as a single rigid body.\nSimple Prompt Engineering. CoPa demonstrates remark-\nable generalizability across a wide range of scenarios with\n\n\nTasks\nCoPa\n(Ours)\nVoxposer\nCoPa\nw/o foundation\nCoPa\nw/o coarse-to-fine\nCoPa\nw/o constraint\nHammer nail\n30%\n0%\n0%\n0%\n10%\nFind scissors\n70%\n50%\n10%\n70%\n70%\nPress button\n80%\n10%\n10%\n60%\n20%\nOpen drawer\n80%\n40%\n10%\n70%\n30%\nPour water\n30%\n0%\n0%\n10%\n0%\nPut eraser into drawer\n80%\n30%\n30%\n60%\n80%\nInsert flower into vase\n70%\n0%\n0%\n60%\n0%\nPut glasses onto shelf\n60%\n20%\n30%\n50%\n60%\nPut spoon into cup\n60%\n10%\n0%\n30%\n30%\nSweep nuts\n70%\n20%\n20%\n50%\n70%\nTotal\n63%\n18%\n11%\n46%\n37%\nTABLE I: Quantitative results in real-world experiments. CoPa successfully complete everyday manipulation tasks with a\nhigh success rate, demonstrating a profound physical understanding of scenes, significantly surpassing the baseline VoxPoser.\nFurthermore, we conduct ablation study to validate the importance of foundation models in our algorithm, as well as the\ndesign of coarse-to-fine grounding and constraint generation.\nminimal prompt engineering. In our CoPa experiments, we\nemploy just three examples to aid the VLMs in compre-\nhending their roles. In contrast, Voxposer relies on highly\ncomplex prompts containing 85 hand-crafted examples. Its\ncapability for reasoning predominantly stems from the pro-\nvided prompts, thereby limiting its generalizability to new\nscenarios. When we attempt to simplify Voxposer’s prompts,\nreducing the example count to three for each module, the\nsystem’s performance drastically declines, resulting in almost\ncomplete failure across all evaluated tasks.\nHandling Rotation DoF. Robotic manipulation requires not\njust the movement of the end-effector to a specified location\nbut also the precise control of its rotation. For example, in\nthe Pour water task, it is essential to rotate the kettle\nto a certain angle to enable the water to flow out through\nthe spout. CoPa calculates the end-effector’s 6-DoF pose by\nconsidering the spatial geometric constraints of key object\nparts within the scene, allowing for accurate and continuous\ncontrol over rotation DoF. Conversely, Voxposer attempts\nto have LLMs directly specify the end-effector’s rotation\nDoF based on simple examples in prompts, causing the\noutput rotation values to be selected from a limited set of\ndiscrete options. This approach often overlooks the dynamic\ninteractions and constraints between objects. For example,\nin the Put spoon into cup (shown in Fig. 5 right),\nCoPa rotates the spoon to a vertical orientation, whereas\nVoxposer positions the robot’s end-effector to face the cup,\nresulting in a collision between the spoon and the cup.\nD. Ablation Study\nWe next conduct a series of ablation studies to demonstrate\nthe significance of the foundation model within our frame-\nwork, as well as the design of coarse-to-fine grounding and\nconstraint generation. The results are shown in Table I.\n1) CoPa w/o foundation: We eliminate the use of founda-\ntion vision-language models (GPT-4V). Specifically, we sub-\nstitute grasping/task-relevant parts grounding module with an\nopen-vocabulary detector, Owl-ViT. Additionally, we remove\nthe constraint generation phase and instead compute post-\ngrasp poses in a predefined rule-based manner (detailed in\nthe Appendix). The results, as presented in Table I, reveal\nthat this approach encounters significant challenges, with\na success rate of merely 11% across all the tasks. This\nunderscores the crucial role of the common sense knowledge\nembedded within VLMs. For example, in the Sweep nuts\ntask, it becomes challenging to determine which tool in the\nscene is suitable for sweeping without the aid of VLMs.\n2) CoPa w/o coarse-to-fine: We eliminate the coarse-\nto-fine design in the grounding module, opting instead for\ndirect utilization of fine-grained SoM and GPT-4V to select\nobject parts within scenes. Experimental results indicate\nthat removing coarse-to-fine design leads to a performance\ndecline, especially in tasks where identifying important parts\naccurately is challenging. For example, in the Hammer\nnail tasks, the absence of the coarse-to-fine design makes\nthis variant impossible to accurately identify the hammer’s\nstriking surface, leading to zero success rate for this task.\n3) CoPa w/o constraint: In this ablation study, we have\nthe VLMs directly output numerical values for the post-\ngrasp poses of the end-effector, instead of the constraints\nthat need to be satisfied by the object being manipulated.\nExperiments demonstrate that, for most manipulation tasks,\ndirectly deriving precise pose values from scene images is\nextremely challenging. For instance, in the Pour water\ntask, it’s almost impossible for this variant to generate precise\npose values to tilt the kettle to the correct pose. In contrast,\nutilizing constraints given by VLMs to solve for post-grasp\nposes presents a more viable option.\nE. Integration with High-Level Planning\nHigh-level planning and low-level control are two critical\nand decoupled aspects of robotic task execution. Our low-\nlevel control framework can be seamlessly integrated with\nhigh-level planning methods to accomplish complex long-\nhorizon tasks. We design two long-horizon tasks, Make\npour-over coffee and Set up romantic table,\nto validate the effectiveness of this combination. Not only\ndo these two tasks need to be accurately decomposed into\nreasonable and actionable steps, but the execution of each\nstep requires a profound understanding of the physical prop-\nerties of the task-relevant objects. Specifically, we employ\n\n\n“Put flowers into vase”\n“Right fallen bottle”\n“Place fork and knife”\n“Pour red wine”\n“Scoop beans into container”\n“Put funnel onto carafe”\n“Pour powder into funnel”\n“Pour water to funnel”\nMake pour-over coffee\nSet up romantic table\nLong-Horizon Tasks\nSequential Steps\nFig. 6: Intergration with High-Level Planning. We show the execution process of two long-horizon tasks: Make\npour-over coffee and Set up romantic table. We demonstrate that CoPa can be seamlessly integrated with\nhigh-level planning methods to accomplish complex long-horizon tasks.\nVILA [10] as the high-level planning method to decom-\npose the high-level instruction into a sequence of low-level\ncontrol tasks. Subsequently, these low-level control tasks\nare executed sequentially using CoPa. Fig. 6 shows some\nenvironment rollouts. Experiments demonstrate that CoPa,\ncombined with high-level planning methods, can effectively\ncomplete long-horizon tasks, showcasing the potential of this\ncombination for real-world applications.\nV. DISCUSSION & LIMITATIONS\nIn this work, we present CoPa, a novel framework that\nleverages the common sense knowledge of foundation vision-\nlanguage models to generate pose sequences for robotic\nmanipulation tasks. CoPa operates effectively with simple\nprompt engineering without requiring any training. Boasting\na fine-grained physical understanding of scenes, CoPa can\ngeneralize to open-world scenarios, handling open-set in-\nstructions and objects. Moreover, CoPa can be naturally\ncombined with high-level planning algorithms to accomplish\ncomplex, long-horizon tasks.\nCoPa has a few limitations that future work can improve.\nFirst, CoPa’s capability to process complex objects is con-\nstrained by its reliance on simplistic geometric elements\nsuch as surfaces and vectors. This can be improved by\nincorporating more geometric elements into our modeling\nprocess. Second, the VLMs currently in use are pre-trained\non large-scale 2D images and lack a genuine grounding in\nthe 3D physical world. This limitation hampers their ability\nto perform accurate spatial reasoning. Integrating 3D inputs,\nlike point clouds, into the training phase of VLMs may alle-\nviate this challenge. Lastly, the existing VLMs produce only\ndiscrete textual outputs, whereas our framework essentially\nnecessitates continuous output values, like the coordinates\nof object parts. The development of foundation models that\nincorporate these capabilities remains a highly anticipated\nadvancement.\nREFERENCES\n[1] S. Cambon, R. Alami, and F. Gravot, “A hybrid approach to intricate\nmotion, manipulation and task planning,” The International Journal\nof Robotics Research, vol. 28, no. 1, pp. 104–126, 2009.\n[2] L. P. Kaelbling and T. Lozano-P´\nerez, “Hierarchical task and motion\nplanning in the now,” in 2011 IEEE International Conference on\nRobotics and Automation.\nIEEE, 2011, pp. 1470–1477.\n[3] R. Bommasani, D. A. Hudson, E. Adeli, R. Altman, S. Arora, S. von\nArx, M. S. Bernstein, J. Bohg, A. Bosselut, E. Brunskill, et al.,\n“On the opportunities and risks of foundation models,” arXiv preprint\narXiv:2108.07258, 2021.\n[4] J. Achiam, S. Adler, S. Agarwal, L. Ahmad, I. Akkaya, F. L. Aleman,\nD. Almeida, J. Altenschmidt, S. Altman, S. Anadkat, et al., “Gpt-4\ntechnical report,” arXiv preprint arXiv:2303.08774, 2023.\n[5] Y. Hu, Q. Xie, V. Jain, J. Francis, J. Patrikar, N. Keetha, S. Kim,\nY. Xie, T. Zhang, Z. Zhao, et al., “Toward general-purpose robots\nvia foundation models: A survey and meta-analysis,” arXiv preprint\narXiv:2312.08782, 2023.\n[6] R. Firoozi, J. Tucker, S. Tian, A. Majumdar, J. Sun, W. Liu, Y. Zhu,\nS. Song, A. Kapoor, K. Hausman, et al., “Foundation models in\nrobotics: Applications, challenges, and the future,” arXiv preprint\narXiv:2312.07843, 2023.\n[7] M. Ahn, A. Brohan, N. Brown, Y. Chebotar, O. Cortes, B. David,\nC. Finn, C. Fu, K. Gopalakrishnan, K. Hausman, et al., “Do as i\ncan, not as i say: Grounding language in robotic affordances,” arXiv\npreprint arXiv:2204.01691, 2022.\n[8] W. Huang, F. Xia, D. Shah, D. Driess, A. Zeng, Y. Lu, P. Florence,\nI. Mordatch, S. Levine, K. Hausman, et al., “Grounded decoding:\nGuiding text generation with grounded models for robot control,”\narXiv preprint arXiv:2303.00855, 2023.\n[9] W. Huang, F. Xia, T. Xiao, H. Chan, J. Liang, P. Florence, A. Zeng,\nJ. Tompson, I. Mordatch, Y. Chebotar, et al., “Inner monologue:\nEmbodied reasoning through planning with language models,” arXiv\npreprint arXiv:2207.05608, 2022.\n[10] Y. Hu, F. Lin, T. Zhang, L. Yi, and Y. Gao, “Look before you leap:\nUnveiling the power of gpt-4v in robotic vision-language planning,”\narXiv preprint arXiv:2311.17842, 2023.\n[11] A. Hussein, M. M. Gaber, E. Elyan, and C. Jayne, “Imitation learning:\nA survey of learning methods,” ACM Computing Surveys (CSUR),\nvol. 50, no. 2, pp. 1–35, 2017.\n[12] R. S. Sutton and A. G. Barto, Reinforcement learning: An introduction.\nMIT press, 2018.\n[13] A. Xie, L. Lee, T. Xiao, and C. Finn, “Decomposing the generalization\ngap in imitation learning for visual robotic manipulation,” arXiv\npreprint arXiv:2307.03659, 2023.\n[14] A. Brohan, N. Brown, J. Carbajal, Y. Chebotar, J. Dabis, C. Finn,\nK. Gopalakrishnan, K. Hausman, A. Herzog, J. Hsu, et al., “Rt-1:\n\n\nRobotics transformer for real-world control at scale,” arXiv preprint\narXiv:2212.06817, 2022.\n[15] A. Brohan, N. Brown, J. Carbajal, Y. Chebotar, X. Chen, K. Choro-\nmanski, T. Ding, D. Driess, A. Dubey, C. Finn, et al., “Rt-2: Vision-\nlanguage-action models transfer web knowledge to robotic control,”\narXiv preprint arXiv:2307.15818, 2023.\n[16] W. Huang, C. Wang, R. Zhang, Y. Li, J. Wu, and L. Fei-Fei, “Voxposer:\nComposable 3d value maps for robotic manipulation with language\nmodels,” arXiv preprint arXiv:2307.05973, 2023.\n[17] J. Liang, W. Huang, F. Xia, P. Xu, K. Hausman, B. Ichter, P. Florence,\nand A. Zeng, “Code as policies: Language model programs for em-\nbodied control,” in 2023 IEEE International Conference on Robotics\nand Automation (ICRA).\nIEEE, 2023, pp. 9493–9500.\n[18] M. Shridhar, L. Manuelli, and D. Fox, “Cliport: What and where\npathways for robotic manipulation,” in Conference on Robot Learning.\nPMLR, 2022, pp. 894–906.\n[19] ——, “Perceiver-actor: A multi-task transformer for robotic manipula-\ntion,” in Conference on Robot Learning.\nPMLR, 2023, pp. 785–799.\n[20] T. Zhang, Y. Hu, H. Cui, H. Zhao, and Y. Gao, “A universal semantic-\ngeometric representation for robotic manipulation,” arXiv preprint\narXiv:2306.10474, 2023.\n[21] A. Padalkar, A. Pooley, A. Jain, A. Bewley, A. Herzog, A. Ir-\npan, A. Khazatsky, A. Rai, A. Singh, A. Brohan, et al., “Open\nx-embodiment: Robotic learning datasets and rt-x models,” arXiv\npreprint arXiv:2310.08864, 2023.\n[22] O. M. Team, D. Ghosh, H. Walke, K. Pertsch, K. Black, O. Mees,\nS. Dasari, J. Hejna, C. Xu, J. Luo, et al., “Octo: An open-source\ngeneralist robot policy,” 2023.\n[23] Y. Liu, W. Dong, Y. Hu, C. Wen, Z.-H. Yin, C. Zhang, and Y. Gao,\n“Imitation learning from observation with automatic discount schedul-\ning,” arXiv preprint arXiv:2310.07433, 2023.\n[24] W. Ye, Y. Zhang, M. Wang, S. Wang, X. Gu, P. Abbeel, and\nY. Gao, “Foundation reinforcement learning: towards embodied\ngeneralist agents with foundation prior assistance,” arXiv preprint\narXiv:2310.02635, 2023.\n[25] I. Akkaya, M. Andrychowicz, M. Chociej, M. Litwin, B. McGrew,\nA. Petron, A. Paino, M. Plappert, G. Powell, R. Ribas, et al., “Solving\nrubik’s cube with a robot hand,” arXiv preprint arXiv:1910.07113,\n2019.\n[26] S. Levine, C. Finn, T. Darrell, and P. Abbeel, “End-to-end training\nof deep visuomotor policies,” The Journal of Machine Learning\nResearch, vol. 17, no. 1, pp. 1334–1373, 2016.\n[27] Z. Xu, Z. Xian, X. Lin, C. Chi, Z. Huang, C. Gan, and S. Song,\n“Roboninja: Learning an adaptive cutting policy for multi-material\nobjects,” arXiv preprint arXiv:2302.11553, 2023.\n[28] J. Matas, S. James, and A. J. Davison, “Sim-to-real reinforcement\nlearning for deformable object manipulation,” in Conference on Robot\nLearning.\nPMLR, 2018, pp. 734–743.\n[29] R. Jeong, Y. Aytar, D. Khosid, Y. Zhou, J. Kay, T. Lampe, K. Bous-\nmalis, and F. Nori, “Self-supervised sim-to-real adaptation for visual\nrobotic manipulation,” in 2020 IEEE international conference on\nrobotics and automation (ICRA).\nIEEE, 2020, pp. 2718–2724.\n[30] S. Parisi, A. Rajeswaran, S. Purushwalkam, and A. Gupta, “The\nunsurprising effectiveness of pre-trained vision models for control,”\nin International Conference on Machine Learning.\nPMLR, 2022, pp.\n17 359–17 371.\n[31] S. Nair, A. Rajeswaran, V. Kumar, C. Finn, and A. Gupta, “R3m: A\nuniversal visual representation for robot manipulation,” arXiv preprint\narXiv:2203.12601, 2022.\n[32] I. Radosavovic, T. Xiao, S. James, P. Abbeel, J. Malik, and T. Dar-\nrell, “Real-world robot learning with masked visual pre-training,” in\nConference on Robot Learning.\nPMLR, 2023, pp. 416–426.\n[33] T. Xiao, I. Radosavovic, T. Darrell, and J. Malik, “Masked visual pre-\ntraining for motor control,” arXiv preprint arXiv:2203.06173, 2022.\n[34] A. Majumdar, K. Yadav, S. Arnaud, Y. J. Ma, C. Chen, S. Silwal,\nA. Jain, V.-P. Berges, P. Abbeel, J. Malik, et al., “Where are we in\nthe search for an artificial visual cortex for embodied intelligence?”\narXiv preprint arXiv:2303.18240, 2023.\n[35] Y. J. Ma, S. Sodhani, D. Jayaraman, O. Bastani, V. Kumar, and\nA. Zhang, “Vip: Towards universal visual reward and representa-\ntion via value-implicit pre-training,” arXiv preprint arXiv:2210.00030,\n2022.\n[36] Y. J. Ma, W. Liang, G. Wang, D.-A. Huang, O. Bastani, D. Ja-\nyaraman, Y. Zhu, L. Fan, and A. Anandkumar, “Eureka: Human-\nlevel reward design via coding large language models,” arXiv preprint\narXiv:2310.12931, 2023.\n[37] M. Alakuijala, G. Dulac-Arnold, J. Mairal, J. Ponce, and C. Schmid,\n“Learning reward functions for robotic manipulation by observing\nhumans,” in 2023 IEEE International Conference on Robotics and\nAutomation (ICRA).\nIEEE, 2023, pp. 5006–5012.\n[38] P. Mahmoudieh, D. Pathak, and T. Darrell, “Zero-shot reward speci-\nfication via grounded natural language,” in International Conference\non Machine Learning.\nPMLR, 2022, pp. 14 743–14 752.\n[39] Y. Cui, S. Niekum, A. Gupta, V. Kumar, and A. Rajeswaran, “Can\nfoundation models perform zero-shot task specification for robot\nmanipulation?” in Learning for Dynamics and Control Conference.\nPMLR, 2022, pp. 893–905.\n[40] Y. J. Ma, W. Liang, V. Som, V. Kumar, A. Zhang, O. Bastani, and\nD. Jayaraman, “Liv: Language-image representations and rewards for\nrobotic control,” arXiv preprint arXiv:2306.00958, 2023.\n[41] I. Singh, V. Blukis, A. Mousavian, A. Goyal, D. Xu, J. Tremblay,\nD. Fox, J. Thomason, and A. Garg, “Progprompt: Generating situated\nrobot task plans using large language models,” in 2023 IEEE Interna-\ntional Conference on Robotics and Automation (ICRA).\nIEEE, 2023,\npp. 11 523–11 530.\n[42] J. Gao, B. Sarkar, F. Xia, T. Xiao, J. Wu, B. Ichter, A. Majumdar, and\nD. Sadigh, “Physically grounded vision-language models for robotic\nmanipulation,” arXiv preprint arXiv:2309.02561, 2023.\n[43] Y. Ding, X. Zhang, C. Paxton, and S. Zhang, “Task and motion\nplanning with large language models for object rearrangement,” arXiv\npreprint arXiv:2303.06247, 2023.\n[44] W. Huang, P. Abbeel, D. Pathak, and I. Mordatch, “Language models\nas zero-shot planners: Extracting actionable knowledge for embodied\nagents,” in International Conference on Machine Learning.\nPMLR,\n2022, pp. 9118–9147.\n[45] K. Lin, C. Agia, T. Migimatsu, M. Pavone, and J. Bohg, “Text2motion:\nFrom natural language instructions to feasible plans,” arXiv preprint\narXiv:2303.12153, 2023.\n[46] B. Liu, Y. Jiang, X. Zhang, Q. Liu, S. Zhang, J. Biswas, and P. Stone,\n“Llm+ p: Empowering large language models with optimal planning\nproficiency,” arXiv preprint arXiv:2304.11477, 2023.\n[47] A. Z. Ren, A. Dixit, A. Bodrova, S. Singh, S. Tu, N. Brown, P. Xu,\nL. Takayama, F. Xia, J. Varley, et al., “Robots that ask for help:\nUncertainty alignment for large language model planners,” arXiv\npreprint arXiv:2307.01928, 2023.\n[48] C. H. Song, J. Wu, C. Washington, B. M. Sadler, W.-L. Chao,\nand Y. Su, “Llm-planner: Few-shot grounded planning for embodied\nagents with large language models,” in Proceedings of the IEEE/CVF\nInternational Conference on Computer Vision, 2023, pp. 2998–3009.\n[49] J. Wu, R. Antonova, A. Kan, M. Lepert, A. Zeng, S. Song,\nJ. Bohg, S. Rusinkiewicz, and T. Funkhouser, “Tidybot: Personal-\nized robot assistance with large language models,” arXiv preprint\narXiv:2305.05658, 2023.\n[50] S. Karaman and E. Frazzoli, “Sampling-based algorithms for optimal\nmotion planning,” The international journal of robotics research,\nvol. 30, no. 7, pp. 846–894, 2011.\n[51] L. Gang and J. Wang, “Prm path planning optimization algorithm\nresearch,” Wseas Transactions on Systems and control, vol. 11, pp.\n81–86, 2016.\n[52] H.-S. Fang, C. Wang, M. Gou, and C. Lu, “Graspnet-1billion: A large-\nscale benchmark for general object grasping,” in Proceedings of the\nIEEE/CVF conference on computer vision and pattern recognition,\n2020, pp. 11 444–11 453.\n[53] OpenAI, “Gpt-4v(ision) system card,” https://cdn.openai.com/papers/\nGPTV System Card.pdf, 2023.\n[54] H. Geng, S. Wei, C. Deng, B. Shen, H. Wang, and L. Guibas,\n“Sage: Bridging semantic and actionable parts for generalizable\narticulated-object manipulation under language instructions,” arXiv\npreprint arXiv:2312.01307, 2023.\n[55] J. Yang, H. Zhang, F. Li, X. Zou, C. Li, and J. Gao, “Set-of-mark\nprompting unleashes extraordinary visual grounding in gpt-4v,” arXiv\npreprint arXiv:2310.11441, 2023.\n[56] J. Wei, X. Wang, D. Schuurmans, M. Bosma, F. Xia, E. Chi, Q. V.\nLe, D. Zhou, et al., “Chain-of-thought prompting elicits reasoning in\nlarge language models,” Advances in Neural Information Processing\nSystems, vol. 35, pp. 24 824–24 837, 2022.\n[57] M. Minderer, A. Gritsenko, A. Stone, M. Neumann, D. Weissenborn,\nA. Dosovitskiy, A. Mahendran, A. Arnab, M. Dehghani, Z. Shen, et al.,\n“Simple open-vocabulary object detection with vision transformers.\narxiv 2022,” arXiv preprint arXiv:2205.06230.\n[58] A. Kirillov, E. Mintun, N. Ravi, H. Mao, C. Rolland, L. Gustafson,\nT. Xiao, S. Whitehead, A. C. Berg, W.-Y. Lo, et al., “Segment\nanything,” arXiv preprint arXiv:2304.02643, 2023.\n\n\nAPPENDIX\nA. Hardware Setup\nWe set up a real-world tabletop environment. We use a Franka Emika Panda robot (a 7-DoF arm) and a 1-DoF parallel jaw\ngripper. We use Franka ROS and MoveIt1 to control the robot, which by default uses an RRT-Connect planner for motion\nplanning. For perception, we mount two RGB-D cameras (Intel RealSense D435) at two opposite ends (left and right from\nthe top-down view) of the table and calibrate them.\nB. Tasks and Evaluations.\nWe design 10 real-world manipulation tasks, each demanding a comprehensive understanding of the physical properties\nof objects. We provide a detailed description of these tasks in Table II. For each task, we evaluate all methods across 10\ndifferent variations of the environment, which encompass alterations in object types and their arrangements.\nHammer nail\nInstruction: “Hammer the nail.”\nDescription: This task requires the robot to first grasp the handle of the hammer, then rotate it\nuntil its striking surface aligns with the surface of the nail, and finally hammer downwards. To\naccomplish this task, it is essential to accurately identify and model the hammer’s striking surface,\nhandle and the nail’s surface.\nFind scissors\nInstruction: “Find scissors for me.”\nDescription: In this task, the scissors may be partially obscured by other objects, such as books.\nThe robot is required to first locate the scissors and then grasp its handle.\nPress button\nInstruction: “Press the button with the stick.”\nDescription: This task necessitates initially grasping the stick, then rotating it until its axis is\ndirectly aligned with the button, and finally pressing it. To accomplish this task, it is imperative\nto accurately identify and model the stick and the button.\nOpen drawer\nInstruction: “Open the drawer.”\nDescription: This task requires the initial grasping of the drawer handle, followed by a linear pull\nalong the handle’s normal vector.\nPour water\nInstruction: “Pour water from kettle to funnel/cup.”\nDescription: This task requires that the spout needs to be moved directly above the funnel, and\nthe kettle needs to be rotated at a certain angle so that the water can flow out. This task imposes\nstringent demands on the robot’s control over its rotation DoF.\nPut eraser into drawer\nInstruction: “Put eraser into the drawer.”\nDescription: In this task, a portion of the eraser is encapsulated by a protective cover, necessitating\nthat the robot exclusively grasps this protective cover.\nInsert flower into vase\nInstruction: “Put flowers into the vase.”\nDescription: This task requires first grasping the flower by its stem (not the petals), then moving\nthe flower directly above the vase while rotating the flower to an upright position, and finally\ninserting it straight down into the vase.\nPut glasses onto shelf\nInstruction: “Put glasses onto the shelf.”\nDescription: In this task, We need to utilize common sense knowledge to determine that, when\npicking up glasses, one should grasp the frame rather than the lenses.\nPut spoon into cup\nInstruction: “Put spoon into the cup.”\nDescription: This task requires first grasping the spoon’s handle, then rotating it to the vertical\ndirection, moving it directly above the cup, and finally inserting it vertically down into the cup.\nSweep nuts\nInstruction: “Select a tool to sweep nuts aside.”\nDescription: This task requires the robot to first identify a tool (e.g. rasp) suitable for sweeping\nnuts through common sense knowledge, and then to grasp the handle of the selected tool.\nTABLE II: A List of 10 Real-World Manipulation Tasks. These tasks require a profound physical understanding of the\nscene. We provide the instructions used in our experiments and detailed descriptions for each task.\nC. VLMs and Prompting.\nWe employ GPT-4V from OpenAI API as the VLM. CoPa involves minimal few-shot prompts to aid VLMs in\ncomprehending their roles. Additionally, the chain-of-thought technique [56] is utilized to facilitate a deeper understanding\nof the scene by VLMs. Prompts used in Section III-B and Section III-C can be found as follows:\nCoarse-Grained Grasping Object Grounding: copa-2024.github.io/prompts/coarse grained grasping object grounding.pdf\n1http://docs.ros.org/en/kinetic/api/moveit tutorials/html/\n\n\nFine-Grained Grasping Part Grounding: copa-2024.github.io/prompts/fine grained grasping part grounding.pdf\nCoarse-Grained Task-Relevant Object Grounding: copa-2024.github.io/prompts/coarse grained relevant object grounding.pdf\nFine-Grained Task-Relevant Part Grounding: copa-2024.github.io/prompts/fine grained relevant part grounding.pdf\nConstraint Generation: copa-2024.github.io/prompts/constraint generation.pdf\nD. Baselines.\nWe compare with Voxposer [16], a method capable of synthesizing closed-loop robot trajectories without necessitating\nadditional training through the utilization of a series of foundational models. Following Huang et al [16], we employ GPT-4\nfrom OpenAI API as the LLM, and utilize the open-vocabulary detector Owl-ViT [57] and Segment Anything [58] for\nperception. Additionally, we adopt their real-world prompt as the prompt for Voxposer in our experiments.\nE. Robotic Arm Filtering.\nBased on the camera’s extrinsic parameters, we render the robot’s URDF model onto the camera plane, thereby obtaining\nthe robot’s mask.\nF. Part Modeling and Annotation.\nIn the task-aware motion planning phase (Section III-C), we need to model task-relevant parts selected by the grounding\nmodule and annotate them in the scene image. We complete this through the following stages:\nPart Modeling. We model identified task-relevant parts as vectors or surfaces. Specifically, we commence by obtaining the\nminimum bounding rectangle for each part. Parts with an aspect ratio exceeding a predetermined threshold are considered\nslender and are modeled as vectors. The remaining parts are modeled as surfaces.\nPart Formulation. We need to obtain the mathematical representation of each part on the 2D image in this stage. For parts\nmodeled as vectors, we first perform linear regression to fit a line that best corresponds to their 2D masks, then identify the\nintersection points of the line with the boundaries of the parts, which serve as the endpoints of the vector. For parts modeled\nas surfaces, we first employ the RANSAC algorithm to determine their 3D center points and normal vectors, which are then\nprojected onto the 2D image.\nPart Annotation. Now we need to annotate the parts according to their formulation on the scene image. First, each part is\nmasked with color and translucency on the image. Then for the parts modeled as vectors, we connect the two endpoints and\nput a numerical label adjacent to the endpoint farther from the robot arm. For parts modeled as faces, we mark the center\npoint and normal vector, and label adjacent to the center point.\nG. Details of Constraints.\nFor each task, we obtain a set of constraints through vision-language models. We then utilize optimization algorithms\n(e.g., the BFGS algorithm or Trust-Region Constrained Optimization) to solve for an SE(3) matrix that minimizes the\ncumulative loss associated with these constraints. In Table III, we provide a detailed description of the constraints used in\nour experiments, along with their corresponding loss calculation methods. In the descriptions of these constraints, Point\nA and Vector A are located on the object being manipulated, and thus require an SE(3) transformation when calculating\nloss. Other points and vectors are considered static and do not require SE(3) transformation.\nWe define the SE(3) matrix we need to solve as follows:\nT =\n\u0014 R\nt\n0T\n1\n\u0015\n∈SE(3),\n(1)\nwhere Euclidean group SE(3) := {R, t | R ∈SO(3), t ∈R3}. We denote the points as p ∈R3, and the normalized vectors\nas V ∈R3. Furthermore, we denote T as the SE(3) transformation, which can be applied to both points and vectors:\nT (p) = Rp + t,\nT (V ) = RV,\n(2)\nH. Details of Subsequent Actions.\nIn Table IV, we provide a detailed description of the subsequent actions utilized in our experiments, along with their\ncorresponding methodologies for calculating new poses.\nI. Predefined Rule-Based Post-Grasp Pose Generation.\nWe design a rule-based method to replace the constraint generation module within our framework to generate post-grasp\nposes. This method entails a prescribed pose calculation protocol specific to each format of instruction. The formats of the\ninstructions along with their corresponding post-grasp poses calculation methodologies are detailed in Table V.\nJ. More Visualization.\nAdditional visualization for grounding module are presented in Fig. 7, for task-oriented grasping in Fig. 8, and for\ntask-relevant motion planning in Fig. 9.\n\n\nDescriptions of Constraints\nLoss Calculation\nVector A and Vector B are on the same line,\nwith the opposite direction.\nloss = ∥T (VA) × VB∥+ ∥(T (pA) −pB) × VB∥+ ∥T (VA) + VB∥\nThe target position of Point A is x cm along\nVector B from Point C’s current position.\nloss = |(T (pA) −pC) · VB −x| + ∥(T (pA) −pC) × VB∥\nVector A is parallel to the table surface.\nloss = |T (VA) · Vtable|\nPoint A is x cm above the table surface.\nloss = |(T (pA) −ptable) · Vtable −x|\nVector A is perpendicular to the table surface.\nloss = ∥T (VA) × Vtable∥\nTABLE III: Desciptions of Constraints and Their Corresponding Loss Calculation Methods. ∥·∥represents l2-norm.\nThe variables ptable and Vtable respectively denote the coordinate and the normal vector of the table, with their values being\n(0.5, 0, 0.07) and (0, 0, 1) respectively. Each Vector corresponds to a Point. The normal vector of the part modeled as\nthe surface corresponds to the center point of the surface, while the point corresponding to the part modeled as the vector\nis the point farther away from the robotic arm among its two endpoints.\nDescriptions of Subsequent Actions\nNew Pose Calculation\nMove vertically down x cm.\nSubtract x cm from the current pose on the z-axis.\nMove forward x cm.\nMove x cm along the current orientation of the end-effector.\nOpen the gripper.\nOpen the gripper.\nEnd-effector rotates 180 degrees.\nRotate the joint corresponding to the end-effector (the 7th\njoint for Franka Emika Panda robot) by 180 degrees.\nTABLE IV: Desciptions of subsequent actions and their corresponding new pose calculation methods.\nInstruction Format\nPost-Grasp Pose Calculation\nHammer A.\n1. Move hammer to 5 cm above A.\n2. Move vertically down 6 cm.\nPress A with B.\n1. Move B to 5 cm above A.\n2. Move vertically down 6 cm.\nOpen A.\n1. Move backward 10 cm.\nPour water from A to B.\n1. Move A to 5 cm above B.\n2. End-effector rotates 180 degrees.\nPut A into B.\n1. Move A to 5 cm above B.\n2. Open the gripper.\nTABLE V: Predefined rule-based pose calculation methods. A and B in the instruction can refer to any object.\n\n\nEnvironment Image\nCoarse-Grained SoM\nFine-Grained SoM\nGrasping Part\n“Put eraser into drawer”\n“Open drawer”\n“Insert flower into vase”\nFig. 7: Visualization for Grounding Module.\nEnvironment Image\nPose Candidates\nGrasp Filtering\nFinal Grasp\n“Put glasses onto shelf”\n“Find scissors for me”\n“Sweep nuts”\nFig. 8: Visualization for Task-Oriented Grasping.\n\n\nEnvironment Image\nTask-Relevant 3D Components\nConstraints\nPost-Grasp Poses Execution\nSpatial Constraints:\n1. Vector 1 and Vector 2 are \ncolinear with opposite direction.\n2. Point 1 is 5 cm along Vector \n2 from Point 2.\nSubsequent Actions:\n1. Move forward 6 cm.\nSpatial Constraints:\n1. Vector 2 and Vector 1 are \ncolinear with opposite direction.\n2. Vector 2 is 3 cm Vector 1 \nalong from Point 1.\nSubsequent Actions:\nNone\nSpatial Constraints:\n1. Vector 2 and Vector 1 are \ncolinear with opposite direction.\n2. Point 2 is 2 cm along Vector 1\nfrom Point 1.\nSubsequent Actions:\n1. Move vertically down 6 cm.\n2. Open the gripper.\n“Pour water”\n“Press button”\n“Put spoon into cup”\nFig. 9: Visualization for Task-Aware Motion Planning.","difficulty":"hard","domain":"Single-Document QA","length":"short","question":"Which kind of ability is not mentioned in the essay?","sub_domain":"Academic"}

Source: https://huggingface.co/datasets/zai-org/LongBench-v2

initial import

Posting: /agents

GET /api/v1/write?intent=publish&task_id=7ee065cb-121a-599b-80af-a7b3ddeb1f9a&body={url_encoded_text}&agent_name={optional_name}&nonce={optional_random_id}