Robotic dense packing

Abstract

A dynamic robotic packaging approach places a set of objects in a containment or shipping container based on dynamic, or real time evaluation and imaging of the set of objects for placement. Predefined object dimensions need not be provided, as an RGB-D or similar imaging provides a representation of each object in a set of objects for packaging. A sensed point cloud of each object is employed to compute a sphere tree, which is ordered according to sequencing logic for placing the object in the containment. Placement order is based on a prioritized score for tightly packing the most unwieldy or largest objects first. A no-fit volume is computed around each object, including accommodations for object orientation and deformation against adjacent objects. A final placement disposes the irregularly shaped objects in a close adjacency for optimal utilization of containment volume.

Description

[0001] PATENT APPLICATION

[0002] CJL Attorney Docket No.: WPI24-32((W25-008-02)PCT

[0003] ROBOTIC DENSE PACKING

[0004] Inventors: Shichen Cao, Jing Xiao

[0005] Attorney Docket No.: WPI24-32((W25-008-02)PCT

[0006] BACKGROUND

[0007] Continued investment in Internet driven commerce, coupled with increased automation of delivery mechanisms, drives incentives for efficient packaging of goods for delivery to consumers. As the demand for faster, more cost-effective shipping escalates in e-commerce, there is a growing need to have robots capable of packing objects with unknown shapes and sizes into containers automatically.

[0008] SUMMARY

[0009] A dynamic robotic packaging approach places a set of objects in a containment or shipping container based on dynamic, or real time evaluation and imaging of the set of objects for placement. Predefined object dimensions need not be provided, as an RGB-D or similar imaging provides a representation of each object in a set of objects for packaging. The camera provides a point cloud of each object, which is then employed to compute a sphere tree representation of the object.

[0010] The detected objects are ordered according to sequencing logic for placing the objects in the containment. Placement order is based on a prioritized score for tightly packing the most unwieldy or largest objects first. A no-fit volume is computed around each object, including accommodations for object orientation and deformation against adjacent objects. A final placement disposes the irregularly shaped objects in a close adjacency for optimal utilization of containment volume.

[0011] Configurations herein are based, in part, on the observation that packaging or placement of a plurality of arbitrarily sized objects in a containment or shipping container is often a manual task performed on a trial-and-error basis, often leading to oversized package sizes and substantial use of packing materials for filling voids or Attorney Docket No.: WPI24-32((W25-008-02)PCT empty space. Unfortunately, conventional approaches to automated or robotic packaging approaches suffer from the shortcoming that packed items must conform to a predetermined shape and size, and / or handle only a single item at a time, without considering all objects intended for placement in a single package or box.

[0012] Accordingly, configurations herein substantially overcome the shortcomings of conventional approaches by providing an object recognition and placement approach that considers all objects in a set of objects for packaging, represents each irregular object based on an individual scan of the object without reference to predetermined sizes, dimensions, or mandatory shape constraints, orders each object in the set of objects, and executes a placement of each object according to the planned order. Placement also allows for a subtle deformation or compressibility of each object based on a resilient nature or surrounding compressible packing material or cushioning wrap.

[0013] The disclosed approach provides for densely packing unknown objects that arrive continually at a packing site, such as via a conveyor belt in a warehouse or fulfillment center. The configuration performs simultaneous online object perception, rapid object modeling, dense packing planning, and object picking and packing in the presence of multi-modal sensing uncertainties, approximations in rapid object modeling, and robot motion inaccuracies. The method can accommodate packing both rigid and non-rigid objects aggregated in staging areas where a set of items may arrive piled together. Validation results demonstrate the effectiveness and efficiency of the approach in achieving high-density packing of different kinds of arbitrarily shaped objects.

[0014] In further detail, a method for arranging objects in a fixed container includes perceiving a set of objects for packing in a containment, and representing each object of the set of objects in a world frame of reference relative to the others of the objects in the set. An application having planning logic plans a sequence of each object in the set of objects for placement in the containment, and iteratively executes a placement of each object of the set of objects into the containment according to the sequence.

[0015] In a particular example usage in a packaging environment having a plurality of arbitrary shaped objects for packing into a fixed size containment, the disclosed Attorney Docket No.: WPI24-32((W25-008-02)PCT method for organizing and disposing the objects in the containment identifies a set of objects in a field of view of an imaging device, and represents, based on an imaging signal from the imaging device such as an RGB-D camera, each of the objects in a point cloud. The application establishes, for each of the objects of the set of objects, a local frame of reference, and locates the local frame of reference of each respective object in a world frame of reference including a location of each object in the set of objects, represented by simultaneously considering objects in the field of view of the RGB-D camera. The application generates a sphere tree of each object from the respective point cloud of each object, and orders each of the objects in a sequence based on the sphere tree and a volume of the containment. Execution involves iteratively placing each object of the set of objects in the containment based on a computed no-fit volume, such that the no-fit volume is determined from the respective object for placement and an aggregation of the objects previously placed in the containment. A robotic gripper fits each object in the containment adjacent the objects previously placed in the containment based on a radial shrinkage of the respective sphere tree of the object for placement.

[0016] BRIEF DESCRIPTION OF THE DRAWINGS

[0017] The foregoing and other objects, features and advantages of the invention will be apparent from the following description of particular embodiments of the invention, as illustrated in the accompanying drawings in which like reference characters refer to the same parts throughout the different views. The drawings are not necessarily to scale, emphasis instead being placed upon illustrating the principles of the invention.

[0018] Fig. 1 is a context diagram of robotic insertion of a grasped object from a set of objects for packing;

[0019] Fig. 2 shows a process flow of steps for recognizing, ordering and placing objects in a containment;

[0020] Fig. 3 is a schematic concurrency diagram of the steps of Fig. 2;

[0021] Figs. 4A-4D show representation of objects in a perceived or recognized set of objects for placement; Attorney Docket No.: WPI24-32((W25-008-02)PCT Figs. 5A-5C show computation of a no-fit volume around each object for determining placement against adjacent objects;

[0022] Figs. 6A-6D show an example placement of a first object in a sequence; and Figs. 7A-7J show a full placement sequence of a set of perceived objects in an object set.

[0023] DETAILED DESCRIPTION

[0024] Autonomous packing of 3D objects densely into containers presents a challenging and largely underexplored variant of the classic bin packing problem, particularly for online operations in real time. Enabling a robot manipulator to perform dense packing is a beneficial enhancement for robotics applications in warehouse automation, logistics, and domestic help. Conventional approaches to dense packing predominantly address packing rigid items of simple shapes using classical combinatorial optimization methods. However, real-world packing tasks in a warehouse frequently involve irregularly shaped and even non-rigid objects (such as fabric-wrapped items or bubble-wrapped packages), and objects often arrive continually in a dynamic setting. Such challenges are not addressed sufficiently in conventional approaches.

[0025] Conventional approaches generally assume rigid, simple-shaped items, and approaches integrating real-world applicability with online adaptability remain limited. Conventional methods dealing with complex geometries often face challenges such as computational inefficiency, dependence on predefined object models, and an absence of real-time adaptability. These limitations restrict practical deployment in dynamic packing scenarios. The result is a deficiency for real-time adaptability and generalization. Conventional approaches operate offline and depend on predefined object models and are thus unsuitable for dynamic environments where unknown objects arrive continually and are unable to generalize to previously unseen objects. In other words, packed items must conform to a predetermined set of known shapes, often simple polygons. These conventional approaches that address non-rigid or complex objects often rely on accurate object models, and physically realistic simulation systems, which are computationally intensive and impractical for real-time execution. Attorney Docket No.: WPI24-32((W25-008-02)PCT In contrast, the disclosed approach presents a perception-driven, concurrent framework for autonomous grasping and dense packing of unknown objects arriving continually. The proposed approach perceives incoming unknown objects, builds object models for picking and dense packing, conducts packing planning, and performs grasping and dense packing operations robustly. Particular features include real-time concurrent perception, planning, and execution. The disclosed system simultaneously conducts object detection and model building, grasp planning, packing sequence and pose optimization, and manipulation motions for grasping and packing, adapting dynamically to incoming new objects. An efficient rapid object modeling capability builds a sphere tree representation for each object, based on detected prism-like object bounding volumes, which enables rapid and effective modeling of unknown objects directly from perception and efficient computation of object interactions in dense packing. A 4-DoF (Degree -of-Freedom) dense packing planning of incoming 3-D objects prioritizes incoming objects for packing and determines the placement poses of objects in a container to achieve high packing density.

[0026] The description below presents an example of a robotic placement and object packing apparatus and system for a robotically controlled member such as an actuated claw or end effector achieving an orientation and pose for placing or depositing an object into fixed size container along with a plurality of other objects.

[0027] Fig. 1 is a context diagram for robotic insertion of a grasped object from a set of objects for packing. Referring to Fig. 1, in a warehouse or packing environment 100. an imaging sensor or camera 110 perceives a plurality of objects 120-1..120-3 (120 generally) as a set 125 of objects for packing. The camera 110 identifies each object 120 as a collection of colored pixelations and a depth in the environment 100. A server 130 includes a processor 132 and a memory 134 executing a packaging application 136 including instructions for performing the approach herein. The server 130 is in communication with the camera 110 and the placement robot 112 for simultaneously representing each of the objects 120 in the set 125 until executing the placement using a robotic actuator 114 for precise manipulation. A conveyor 124 and / or staging area aggregates the set 125 of objects until placement in a containment 122, typically a shipping container or box. Attorney Docket No.: WPI24-32((W25-008-02)PCT Upon transport into the packaging environment 100 by the conveyor 124, motorized transport or wheeled vehicle, the camera 110 captures a field of view 111 for locating each object 120 in a local frame of reference 121. The application 136 locates each local frame of reference 121 in a world frame of reference 127 based on the robot 112, such that the world frame of reference includes each of the plurality of objects 120 and the containment 122.

[0028] In an example configuration, the application 136 receives the colored pixelations and depth from an RGB-D camera, which is a 3-D vision device that records both color and distance. It gives Al and robotic systems a way to “see” not only what objects look like but also how far away they are and what shape they have. Other imaging approaches may be employed for providing a concurrent location of the individual objects 120 within the world frame of reference 127.

[0029] An example workspace environment 100 as depicted includes a general -purpose robotic am 116 equipped with a fixed base 118 and a wrist force / torque sensor 115 attached to a gripper actuator 114, and one or more fixed external RGB-D cameras 110 positioned to perceive the workbench (with incoming objects), and the containment 122 for object packing.

[0030] Objects unknown to the robot 112 are assumed to arrive continually from a conveyor and randomly placed on a flat workbench. The objects may be piled together so that some of them may not be entirely visible due to occlusions or limited camera viewpoints. The objects 120 may also be non-rigid and can undergo small deformations. Grasping one object may cause neighboring objects to move slightly.

[0031] Objects are packed one by one into a single rectangular container. The container is generally in the shape of a rectangular cuboid. Once a container is completely packed, a new container is assumed to arrive. Hence, only a single container need be considered at any time for packing.

[0032] The robot's base frame is defined as the world frame, denoted as w. Once an object is detected, its local frame O is established at the centroid of its top surface, with the positive z-axis along the outward surface normal. From visual perception, an object is detected with its configuration, which is defined as a 6-dimensional vector: Attorney Docket No.: WPI24-32((W25-008-02)PCT §

[0033]

[0034] A 0,

[0035] where [x, y, z] describes the object's position, and

[0036]

[0037] * * ' " ' ' * are the roll, pitch, and yaw angles describing its orientation, with respect to the world frame. The object's pose in the world frame, represented by a homogeneous transformation matrix

[0038]

[0039] is computed from q using the Product of Exponentials (PoE) formulation.

[0040] Fig. 2 shows a process flow of steps for recognizing, ordering and placing the objects 120 in the containment 122 of Fig. 1. Referring to Figs. 1 and 2, the method for arranging the objects 120 in a fixed container (containment) 122 includes, at step 210, perceiving the set 125 of objects 120 for packing in a containment 122, and representing each object 120 of the set of objects in a world frame of reference 127 relative to the others of the objects 120 in the set 125. Based on the aggregate set 125 of objects in the world frame 127, the application 136 plans a sequence for placing each object 120 in the set 125 of objects into the containment 122. as depicted at step 220. The application 125 directs the robot 112 for iteratively executing placement of each object 120 of the set 125 of objects into the containment 122 according to the sequence, as shown at step 230. Fig. 2 shows the data and workflow during an operation cycle to complete packing of one object. It demonstrates how objects are concurrently processed across the perception, planning, and execution modules. Each module performs a distinct function to provide input to the next module 210, 220, 230.

[0041] Fig. 3 is a schematic concurrency diagram of the steps of Fig. 2. The autonomous robotic online packing system consists of concurrent processes of perception 210", planning 220’, and execution 230’, as illustrated in Fig. 3. These different processes run at different frequencies. In the first (initial) cycle, the perception module 210 works first to produce object information for the planning module 220, and the planning module then starts its first cycle to conduct packing planning; finally, the execution module 230 starts its first cycle to conduct the physical packing operations based on the planning results. Note that the frequency of each process 210, 220, 230 need not be constant, even though Fig. 2 depicts constant frequency for each process for simplicity of illustration. The frequency of Attorney Docket No.: WPI24-32((W25-008-02)PCT each process can change depending on the nature and number of incoming objects. Generally, the perception process 210’ has a higher frequency than the planning process 220’, both of which are computational, and the physical execution process 230’ has the lowest frequency. In a steady state, each new cycle of the perception process starts when new objects arrive, and its corresponding new results trigger a new cycle for planning. Each execution cycle is determined by the time needed to grasp and pack a single object 120. The timing of Fig. 2 depicts concurrent system of perception, planning, and execution modules. The two dotted vertical lines indicate a single operation cycle in steady state: the first 205-1 marks the beginning of one perception module, and the second 205-2 marks the end of the corresponding execution module.

[0042] In the description that follows, the perception module 210 constantly captures RGB-D data of the scene, segments the data to extract all the point clouds of visible objects, and constructs a sphere-tree model for each visible object. When multiple views are available, they may be integrated into a unified point cloud for an object. It also computes feasible grasp configurations of modeled objects based on the gripper's physical constraints.

[0043] The planning module 220 determines the packing sequence and object placement poses in the container, and continually updates the packing sequence with new object output from the perception module. The object poses in the container are optimized through a volumetric packing planning approach, discussed further below.

[0044] The execution module 230 enables the robot to grasp and pack an object from the packing sequence. It extends to consider deformation effects in contact detection and optimization criteria and overcomes the effects of inaccuracies and uncertainties from sensing, object modeling, and robot motion to ensure successful dense packing of objects in a container with tight clearance.

[0045] For timing and concurrency, each module operates independently and communicates through shared data structures and / or inter-process communication (IPC) mechanisms to enable simultaneous executions on multi -threaded computing platforms to be both flexible and efficient.

[0046] The concurrent modules / processes 210, 220, 230 operate in an RGB-D sensing pipeline to acquire synchronized color and depth data from cameras and Attorney Docket No.: WPI24-32((W25-008-02)PCT generate point clouds for object detection and model construction. For each camera, intrinsic parameters such as principal point, distortion coefficients, and depth scales are extracted, and alignment modules are configured to register depth and color streams.

[0047] The RGB-D sensing stream is employed to form a point cloud representation of each object 120 in the set 125. The application 136 denotes the point cloud of object O[ as Pi, where i = 1... N and N is the total number of visible objects. Multiple cameras 110 may also be employed, and segmentation and other image processing applied.

[0048] After perceiving the point cloud set P of an object O at 210, the application 136 estimates the object pose and identifies a suitable grasp pose for each object 120. First, the object's top surface plane is determined as the topmost region of the object by selecting points near its upper extremity in the world frame 127. The application 136 computes a top surface normal Iltop from this plane, providing an initial estimate To refine the plane's position, the empirical cumulative distribution function (CDF) of point projections along Iltop is used to select the height at which the slope of the CDF's numerical derivative is maximized, which indicates the dominant surface layer, denoted as the highest point Pmax with position Pmax in the world coordinate system. Each point p with position p in the world coordinate system within a threshold distance of this plane is then projected onto it:

[0049] V

[0050]

[0051] p Ppmji P HP ~~ Ptaaa) '

[0052] where Pproj is the projection of p onto the plane that passes through Pmax with normal Iltop. Next, Principal Component Analysis (PCA) is applied to {Pproj } to compute the covariance matrix. The eigendecomposition of £ provides two orthogonal eigenvectors { Vi, V }, ordered by descending eigenvalues, which span the local surface plane. The first eigenvector vl indicates the direction of maximum in-plane variance, while v2 is the orthogonal direction and indicates the grasp width Attorney Docket No.: WPI24-32((W25-008-02)PCT direction. The centroid of the plane and the three axes vl, v2, and Htop form the object coordinate system. A triangle mesh top is constructed from the projected points using 2D Delaunay triangulation. The object height h of the point cloud P along Htop is then computed as:

[0053] h ™ max(p -. mi»(p -

[0054]

[0055] For grasping, the gripper's approach vector is aligned with and opposite to Htop Different grasping strategies are used depending on the gripper type. For a vacuum gripper, the grasping pose is at the centroid of the identified top surface (i.e., the origin of the object frame). For a parallel-jaw gripper, the grasping direction is chosen along v2, the shorter axis determined from Principal Component Analysis (PCA). Contact points for grasping are identified as the extremal points along v2, and the required jaw opening width is decided accordingly.

[0056] The resulting grasp pose, along with object height and top- surface normal, provides the necessary geometric context for downstream processing. In the following section, we describe how such information is used to construct object models for

[0057] dense packing.

[0058] Given an object O with the perceived top surface normal, object height, and top surface mesh

[0059]

[0060] A prism-like bounding volume representation can be formed by extending top with the positions of the three vertices {ul, u2, u3} in the object's local frame, we compute the positions of the corresponding vertices at the bottom of the prism as:

[0061]

[0062] OSK u* — h - f-hajj, % 3

[0063] Repeating this process for all triangles in tOp yields a closed prism mesh as the bounding volume of the object, from which the sphere tree can be generated, discussed below.

[0064] Figs. 4A-4D show representation of objects in a perceived or recognized set of objects for placement. Referring to Figs. 4A-4D, an evolution of the object image Attorney Docket No.: WPI24-32((W25-008-02)PCT is shown in Fig. 4A, the captured point cloud in Fig. 4B, the leaf nodes of the sphere-tree in Fig. 4C, and Fig. 4D shows a sphere-tree model bounding the original object. This progression for representing the objects 120 further includes identifying each object 120 from the others of the set 120 of objects in the field of view 111 containing all the objects 120-1..120-N in the set 125 of objects. The application 136 forms, for each object 120, a point cloud 420 indicative of the object 120, and computes, from the point cloud 420, a sphere tree 422.

[0065] Forming the point cloud further includes identifying a top surface plane of the object as defined by M_top and n_top, and generates a triangular mesh prism from the top surface plane and the point cloud, such that the triangle mesh prism defining an outer surface formed from triangles and surface points. The application 136 then converts each surface point into a sphere tree leaf as in Fig. 4C, iteratively generating parent spheres from the sphere tree leaf that encloses its child spheres for forming the sphere tree 422.

[0066] In further detail, once the prism representation of an object 120 is obtained, a sphere-tree model 422 is constructed using a bottom-up sphere merging method. In this approach, the shape of the prism is approximated by converting evenly sampled points on the object's surface into leaf spheres. Specifically, each sampled surface point becomes the center of a leaf sphere whose radius is scaled such that the sphere exactly touches, i.e., is tangent to, the original object's surface. Sphere merging proceeds from the small spheres at the leaf level upwards, progressively reducing the total number of spheres at higher levels to form a tree. Consequently, each parent sphere fully encloses its child spheres. The sphere -tree model of an object enables efficient detection and localization of potential contacts between objects by rapidly eliminating non-intersecting spheres and accurately identifying intersecting regions of spheres involved in contact. Figs. 4A-4D depict the construction results of a bottle- shaped object, starting from the camera-captured point cloud and its derived prism.

[0067] In one example, generation of the sphere tree proceeds to, for each detected object

[0068]

[0069] c, the application 136 denotes the resulting sphere-tree model as Si. Each kth leaf node of the sphere- tree is a sphere Sk defined by the tuple (Pk, rk)i, with k = 1... n, where n is the total number of leaf nodes in Si. Here, Pk = [xk,yk, Attorney Docket No.: WPI24-32((W25-008-02)PCT zk]i specifies the center coordinates of Sk with respect to the world frame, and rk represents its radius.

[0070] To capture a non-rigid object that allows small deformation, we further introduce a compression ratio parameter β, defining the maximum allowable radial shrinkage relative to the original size of each sphere. The local deformation is modeled by permitting each sphere's radius to vary within a defined range

[0071] K

[0072]

[0073] i - Moreover, as deformation is often not symmetric about the original center of a sphere, we also introduce a positional shift parameter δp for each sphere's center, allowing a more flexible and realistic deformation approximation. Specifically, each sphere center Pk can slightly shift within a bounded region defined by:

[0074]

[0075] where δp_k ∈ ℝ³ is the 3D

[0076] positional shift vector and dmaxbounds its Euclidean norm.

[0077] Typically, dmaxis set as a small fraction of the sphere's original radius

[0078] (

[0079]

[0080] e.g. 0,01 < 7 < 0,1)

[0081] This positional shift significantly enhances the realism of deformation modeling by capturing asymmetric and localized shape changes.

[0082] Effective dense packing of irregular 3D objects in a container or containment 122 requires careful consideration of geometric constraints due to complex interactions and minimizing void spaces in the container. In 2D packing, no-fit polygons and inner- fit polygons capture overlapping regions between planar shapes. Configurations herein extend this concept by introducing no-fit volumes (NFV), which encode all forbidden placements of one object relative to another. Identifying NFVs transforms the packing task into a boundary -constrained optimization: place each object outside NFVs but on the boundary of NFVs to achieve dense packing.

[0083] Consider two objects: Oa (the object to be packed) and Ob (already packed), with their corresponding sphere-tree models Sa and Sb, where Sb can be a set of sphere-tree models if Ob consists of multiple packed objects. Let a denote the relative orientation of Oa with respect to Ob. Let Va( a) and Vb, denote the sets of Attorney Docket No.: WPI24-32((W25-008-02)PCT positions of all points in the union of leaf spheres of Sa and Sb respectively, under orientation a. The NFV of Oa with respect to Ob with relative orientation a is defined as:

[0084] the set of positions of all points

[0085]

[0086] such that placing Oa with its reference point at x results in an interpenetration with Ob:

[0087] 3 G G < fi 4- r.j |

[0088]

[0089] where

[0090]

[0091] denotes the Minkowski sum, G

[0092]

[0093] and j }, are leaf spheres in S, and

[0094]

[0095] .respectively, and

[0096]

[0097] d(s_i, s_j) is the Euclidean distance between the two sphere centers. Note that with this formulation, the NFV does not include contact placements, which are defined as

[0098]

[0099] ∀s_i ∈ S_a, s_j ∈ S_b, d(s_i,s_j) ≥ r_i + r_j. The boundary of an NFV is defined as including all contact placements, Directly computing the NFV for arbitrary orientations in continuous 3D space is impractical due to high memory consumption and computational complexity. Instead, we approximate the NFV as the union of collision-inducing spheres computed from pairs of leaf nodes. We discretize the continuous orientation range α ∈ [0, 2π) into a finite set of discrete orientations {α_ℓ}^K_{ℓ=1}. For each pair of spheres s_i = (p_i, r_i)_a ∈

[0100]

[0101] S_a and s_j = (p_j, r_j)_b ∈ S_b, we construct a forbidden region B_ij centered at p_j −

[0102]

[0103] with radius r_i + r_j, where R_α is the rotation matrix associated with orientation α. This region represents all relative poses of

[0104]

[0105] with respect to that would lead to collision between s_i and s_j

[0106]

[0107] under orientation o, The NFV is thus approximated as:

[0108] NFV_α(S_a, S_b) ≈ ⋃_{i,j} B_ij

[0109]

[0110] M

[0111] This approach captures complex spatial contact constraints between irregular Attorney Docket No.: WPI24-32((W25-008-02)PCT objects. Figs. 5A-5C show computation of a no-fit volume around each object for determining placement against adjacent objects. In the example Oa (120-a) and Ob (120-bl -b3) shown in Figs. 5A-5C, the red dashed boundary 501-1..501-3 (501 generally) around Ob is the NFV boundary of Oa with respect to Ob. Placing Oa's origin at any position within this boundary yields an interpenetration 503 with Ob 120-bl. Figs. 5B and 5C show two feasible Oa's placement in different relative orientations with respect to Ob for 120-b2 and 120-b3. Fig. 5B shows an alternative, feasible (non-collision) orientation for

[0112]

[0113] = 1.5, and Fig. 5C shows an alternative, feasible orientation for

[0114]

[0115] = 5.8. Fig. 5D shows the irregularly placed object 120 in a comer of the containment 122.

[0116] The no-fit volume attributable to an aggregation of previously placed objects and an object currently being placed includes generating a sequence of the object by computing a no-fit volume around each object 120, and iteratively placing each object 120 in the containment 122 based on the no-fit volume of each previously placed object and the containment. The actuator 114 locates each object adjacent the no-fit boundary of the previously placed object. This involves computing the no-fit volume based on an orientation of the sphere tree 422 of an object 120, and collision inducing spheres of a sphere tree of an adjacent object.

[0117] The NFV can be employed in both the sequence planning and in placement execution. Figs. 6A-6D show an example placement of a first object in a sequence. Referring to Figs. 1-6D, planning the placement further includes computing a score for each object 120 to determine the placement order. The application 136 computes, for each object 120, a complexity score for placement based on the other object in the perceived set of objects, generally referring to an eccentric evaluation of the shape, with simple polygon type shapes having a lower score. The application 136 also computes, for each object, a footprint score and a height. The application 136 ranks the objects based on the complexity score, footprint score and the height for determining the order of placement in the containment.

[0118] The ranking identifies an ordering of the objects 120 for placement into the containment 122. The score generally will place larger objects and those with a well defined corner or edge first, to abut comers and side walls of the containment, while Attorney Docket No.: WPI24-32((W25-008-02)PCT smaller objects 120 and irregular shapes are deferred to “fill in” around previously placed objects.

[0119] To achieve a real-time 4-DoF dynamic dense packing of continually incoming objects into a container, an optimization approach is employed that combines two optimization criteria and leverages NFVs to efficiently guide packing sequence

[0120] decisions and local pose optimizations.

[0121] To ensure consistent and compact packing, the application first assigns a fixed comer of the container as a reference corner oc, along with associated container axes:

[0122] x,, z]

[0123] originating from this comer. Subsequent object placements align with these reference axes to achieve dense and organized packing. Let Q be the set containing already-packed objects and the container itself, and let SQ represent their collective occupied volume using the combined sphere -tree model. Initially, SQ corresponds to the outer boundary of the container's sphere- tree, and it is dynamically updated with each object successfully packed into Q.

[0124] Given a set of detected objects:

[0125] {Oih l.

[0126] let Si denote the combined volume of spheres in the sphere-tree model of Oi. A complexity-based score Ci is first computed to quantify each object's difficulty of placement. The complexity score aggregates NFV volumes computed across all other objects and orientations:

[0127] , V £

[0128] - 22 52Vo1

[0129]

[0130] where Vol(») calculates the approximate NFV volume using the union of pairwise spheres. To further enhance packing stability and compactness, a partial ranking of objects based on complexity Ci, footprint area A; (computed from A / top), and object Attorney Docket No.: WPI24-32((W25-008-02)PCT height hi, Each parameter is normalized. This results in the following normalized priority metrica:

[0131]

[0132] — AQCJ A Aj Agh,

[0133] where A_c, A_A, A_h > 0 are normalization factors. The term A A? promotes stable packing by prioritizing objects with larger base surfaces, while the −A_h·h_i term encourages lower-height items to be placed before taller ones, reducing the risk of collision between the robot manipulator and previously packed objects. Finally, all objects are sorted in descending order of Tli, yielding a prioritized packing sequence II. The concurrent nature depicted in Fig. 3 ensures that the prioritized sequence is updated dynamically with any newly detected objects.

[0134] Figs. 7A-7J show a full placement sequence of a set of perceived objects 120-1..120-4 in an object set, shown in Fig. 7A. Referring to Figs. 1-7J, in general terms, execution of the placement phase includes grasping each object 120, by the robotic gripper 114, shown in Fig. 7B, for placement in the containment according to the generated sequence, and disposing or moving (Fig. 7C) each object 120 into the containment (Fig. 7D) with a placement outside the no-fit volume 501 of the previously placed objects 120 in the containment 122. Final placement involves applying a force-torque sensing based on a deformation indicative of allowable shrinkage of each sphere in the sphere tree corresponding to the object, allowing subtle deformations and shifting of the sphere tree to accommodate for flexible objects, padding or wrapping, and elastic or sloping sides of the containment, for example a cardboard box or plastic container amenable to a modest flexible deformation without rupture.

[0135] The application formulates a pose optimization task as a constrained optimization problem. The goal is to find a feasible pose Xi that minimizes a cost function F(Xi), ensuring tight packing, proximity to previously packed objects, and stable alignment with container walls and with existing objects. Formally, this optimization can be described as: Attorney Docket No.: WPI24-32((W25-008-02)PCT

[0136] min F(X_i) = (1 / n)∑d(·,·) + λ·w(X_i)

[0137] Xi n I I

[0138] where

[0139] w(X_i) = 1 − max_{u∈{x,y,z}} |v_2 · u|,

[0140] u∈{x,y,z} s.t. X_i ∉ NFV

[0141]

[0142] 0 ≤ (X_i − o_c) · u ≤ L_u, ∀u ∈ {x, y, z}

[0143] Where

[0144] n = |S_i(X_i)|

[0145] is the number of leaf spheres in Si(Xi), and X > 0 is a scaling factor. In the objective function F(X), the first term is the average of the Euclidean distance / between each sphere center

[0146]

[0147] s_k = (p_k, r_k) ∈

[0148] and its nearest neighboring sphere center

[0149]

[0150] Minimizing this term reduces gaps, achieving tighter packing arrangements between Si and SQ. Minimizing the alignment term w(Xj) penalizes orientations where the object's minor principal axis v2 deviates from being aligned with one of the container's principal axes and prefers the minor axis to be either parallel or perpendicular to the container walls. There are two constraints. The first constraint ensures that the placement Xi is outside the NFV of Si(Xi) and SQ, and the second constraint ensures that Si(Xj) remains entirely within the container by constraining the object's position Xi to lie inside the container boundary defined by the container reference corner Oc, and each edge length:

[0151]

[0152] The pose optimization constitutes a non-convex constrained optimization problem due to discrete orientations and complex spatial contact constraints. To address this efficiently, the application 136 first discretizes possible orientations and employs a Attorney Docket No.: WPI24-32((W25-008-02)PCT coarse-to-fine search strategy to evaluate candidate poses systematically.

[0153] Subsequently, it refines promising candidates locally using constrained gradientbased optimization, achieving efficient convergence to a locally optimal solution. If no feasible placement is found, the algorithm skips the current object and proceeds to the next in the optimized sequence II.

[0154] The sphere tree provides a robust data structure for both sequence planning and packing execution as the application 136 traverses the sphere tree for identifying interference or potential interference with another object of the plurality of objects. Upon detecting a collision with another object, the application 136 may compare a successive level of the sphere tree for achieving a more granular, finer representation of the object 120. If the successive level does not indicate a collision with another object, the application may conclude placement, and if not, traversal may continue with successive levels until a leaf node of the sphere tree is reached, indicating the maximum granularity of the object 120 boundary. Other invocations of the sphere tree for evaluating collisions between adjacent objects 120 and containment 122 walls may be performed.

[0155] Given uncertainties in perception and robotic execution, unintended contacts and placement failures may occur in packing motions with tight clearances. From the packing algorithm, an object Oi's target pose in the container is decided as Xi with respect to the container frame, which is subject to uncertainties and inaccuracies in perception, robot kinematics, and object models. During the packing motion, the effect of uncertainties can misalign Oi with the container or with other packed objects to cause undesired contacts. Such misalignment can occur:

[0156] (a) before placement: where the object makes contact with a container edge (Fig.

[0157] 7D) or already packed objects; or

[0158] (b) after placement: where the object leaves unutilized space between its edges and the container or other packed objects (Fig. 7E).

[0159] To address these uncertainties, the application 136 applies a robust insertion strategy based on force-torque (F / T) sensing, which simultaneously estimates pose uncertainties and predicts contact configurations of contacting rigid objects. This can be extended to include non-rigid objects with small deformations as modeled above. Attorney Docket No.: WPI24-32((W25-008-02)PCT

[0160] From the packing planning result Xi, a nominal goal configuration3of the object Oi can be obtained as a homogeneous transformation matrix. There is an unknown pose uncertainty XTi between

[0161]

[0162] and the actually desired goal configuration GtI

[0163]

[0164] This uncertainty can hinder successful packing by leading to undesired contacts between Oi and the container with already packed objects. Once Oi is in contact, the application 136 applies logic such as a prediction neural network to take the measured F / T as input and outputs a predicted uncertainty ΔT_i*, shown in Figs 7E-7F. We consider this prediction as one of the most likely ΔT_i candidates and use a neighborhood search starting from T7to find the most likely ΔT_i.

[0165] A constrained optimization method may be used to determine the contact configuration corresponding to each candidate for ΔT_i by minimizing the simulated elastic energy between the contact configuration q and the estimated contact configuration q*, while satisfying non-penetration constraints between two rigid parts, shown graphically in the sequence of Figs. 7G-7J. The application 136 extends this method to scenarios involving non-rigid objects that allow small deformations by incorporating deformation characteristics modeled above.

[0166] Specifically, the application introduces an additional deformation cost term and introduces further constraints on allowable sphere center shifts and radial shrinkage for each sphere of the relevant sphere tree 422:

[0167] | (T "" ’I" M

[0168] s.t. ||p_k + δp_k − p_j|| ≥ r_k(1−β) + r_j,

[0169]

[0170] ||δp_k|| ≤ γr_k

[0171] where D is a diagonal stiffness matrix, λ is a scaling factor,

[0172] and δp_k ∈ ℝ³ and β_k are the 3D positional shift vector

[0173]

[0174] ositional shift vector (defined above) and & is the radial shrinkage variable, bounded by an allowable compression ratio 0. The parameter ) ’ limits the center Attorney Docket No.: WPI24-32((W25-008-02)PCT shift to a fraction of the original sphere radius. The constrained optimization generates the most likely contact configuration corresponding to ΔT_i. Next, a force / torque is computed based on Hooke's law using a haptic simulation approach. The computed force / torque is then matched against the sensed force / torque through a force calibration model to further verify ΔT_i and the corresponding contact configuration.

[0175] Once the best estimation AT7is obtained for object Oi, it is used to update the relative goal configuration Gi, and a compliant corrective motion is then executed to reach G / . Once 0 / is packed into the container, the estimated uncertainty ΔT_i is then used to update the estimated goal location of the next object O / +7, i.e.:

[0176] G

[0177]

[0178] G_{i+1} = ΔT_i · G_{i+1}^nominal

[0179] This updates the nominal goal pose

[0180]

[0181] ; closer to the actual pose Gi+i, thereby reducing the possible uncertainty candidates for O_{i+1}, leading to a faster convergence to an accurate ΔT_{i+1}.

[0182] For a type-(a) misalignment, the application 136 generates compliant corrective motions based on the estimated pose uncertainty to ensure that the object enters the container properly. To handle misalignments of type-(b), the application 136 generates a compliant motion to move the object towards the nearest comer of the container ensuring a tighter fit (Fig. 7 J). For the first placed object (Fig. 7D), this comer of the container is designated as the reference corner and used for aligning all subsequent placements.

[0183] Those skilled in the art should readily appreciate that the programs and methods defined herein are deliverable to a user processing and rendering device in many forms, including but not limited to a) information permanently stored on non-writeable storage media such as ROM devices, b) information alterably stored on writeable non-transitory storage media such as solid state drives (SSDs) and media, flash drives, floppy disks, magnetic tapes, CDs, RAM devices, and other magnetic Attorney Docket No.: WPI24-32((W25-008-02)PCT and optical media, or c) information conveyed to a computer through communication media, as in an electronic network such as the Internet or telephone modem lines. The operations and methods may be implemented in a software executable object or as a set of encoded instructions for execution by a processor responsive to the instructions, including virtual machines and hypervisor controlled execution environments. Alternatively, the operations and methods disclosed herein may be embodied in whole or in part using hardware components, such as Application Specific Integrated Circuits (ASICs), Field Programmable Gate Arrays (FPGAs), state machines, controllers or other hardware components or devices, or a combination of hardware, software, and firmware components.

[0184] While the system and methods defined herein have been particularly shown and described with references to embodiments thereof, it will be understood by those skilled in the art that various changes in form and details may be made therein without departing from the scope of the invention encompassed by the appended claims.

Claims

Attorney Docket No.: WPI24-32((W25-008-02)PCT CLAIMSWhat is claimed is:

1. A method for arranging objects in a fixed container, comprising:perceiving a set of objects for packing in a containment;representing each object of the set of objects in a world frame of reference, the world frame of reference locating each object relative to the others of the objects in the set;planning, based on the world frame of reference, a sequence of each object in the set of objects for placement in the containment; anditeratively executing a placement of each object of the set of objects into the containment according to the sequence.

2. The method of claim 1, wherein perceiving further comprises:identifying each object as a collection of colored pixelations and a depth; and simultaneously representing the others of the object in the set until executing the placement.

3. The method of claim 1, further comprising:locating each object in a local frame of reference; andlocating each local frame of reference in a world frame of reference, the world frame of reference including each of the plurality of objects and the containment.

4. The method of claim 1, wherein representing the objects further comprises identifying each object from the others of the set of objects in a field of view containing all the objects in the set of objects;forming, for each object, a point cloud indicative of the object; and computing, from the point cloud, a sphere tree.

5. The method of claim 4, wherein forming the point cloud further comprises:Attorney Docket No.: WPI24-32((W25-008-02)PCT identifying a top surface plane of the object;generating a triangular mesh prism from the top surface plane and the point cloud, the triangle mesh prism defining an outer surface formed from triangles and surface points;converting each surface point into a sphere tree leaf; anditeratively generating parent spheres from the sphere tree leaf that encloses its child spheres.

6. The method of claim 1, wherein planning the placement further comprises:computing, for each object, a complexity score for placement based on the other object in the perceived set of objects;computing, for each object, a footprint score;computing, for each object, a height; andranking the objects based on the complexity score, footprint score and the height for determining the order of placement in the containment.

7. The method of claim 1, wherein iteratively executing further comprises: generating a sequence of the object by computing a no-fit volume around each object;placing each object in the containment based on the no-fit volume of each previously placed object and the containment; andlocating each object adjacent the no-fit boundary of the previously placed object.

8. The method of claim 7 further comprising computing the no-fit volume based on an orientation of the sphere tree of an object, and collision inducing spheres of a sphere tree of an adjacent object.

9. The method of claim 7, wherein execution further comprises:grasping each object, by the robotic gripper, for placement in the containment according to the generated sequence;Attorney Docket No.: WPI24-32((W25-008-02)PCT disposing each object in the containment with a placement outside the no-fit volume of previously placed objects in the containment: andapplying a force-torque sensing based on a deformation indicative of allowable shrinkage of each sphere in the sphere tree corresponding to the object.

10. The method of claim 4, further comprising:traversing the sphere tree for identifying interference with another object of the plurality of objectsupon detecting a collision with another object, comparing a successive level of the sphere tree; andif the successive level does not indicate a collision with another object, concluding placement, and if notcontinuing traversing the successive levels until a leaf node of the sphere tree is reached.

11. The method of claim 2, further comprising receiving the colored pixelations and depth from an RGB-D camera.

12. A robotic packing device for arranging a set of objects in a container, comprising:a robotic arm having an actuator configured for grasping object for placement in the container;a processor, the robotic arm responsive to the processor and a packaging application;a memory for storing the packaging application;an imaging sensor for perceiving a set of objects for packing in a containment;the processor responsive to the application, the application having instructions for:representing each object of the set of objects in a world frame of reference relative to the others of the objects in the set;Attorney Docket No.: WPI24-32((W25-008-02)PCT planning a sequence of each object in the set of objects for placement in the containment; anditeratively executing a placement of each object of the set of objects into the containment according to the sequence.

13. The method of claim 12, wherein representing the objects further comprises identifying each object from the others of the set of objects in a field of view containing all the objects in the set of objects;forming, for each object, a point cloud indicative of the object; and computing, from the point cloud, a sphere tree.

14. The method of claim 13, wherein forming the point cloud further comprises:identifying a top surface plane of the object;generating a triangular mesh prism from the top surface plane and the point cloud, the triangle mesh prism defining an outer surface formed from triangles and surface points;converting each surface point into a sphere tree leaf; anditeratively generating parent spheres from the sphere tree leaf that encloses its child spheres.

15. The method of claim 12, wherein planning the placement further comprises:computing, for each object, a complexity score for placement based on the other object in the perceived set of objects;computing, for each object, a footprint score;computing, for each object, a height; andranking the objects based on the complexity score, footprint score and the height for determining the order of placement in the containment.

16. In a packaging environment having a plurality of arbitrary shaped objects for packing into a fixed size containment, a computer program embodying program code on a non-transitory computer readable storage medium that, when executed byAttorney Docket No.: WPI24-32((W25-008-02)PCT a processor, performs steps for implementing a method for organizing and disposing the objects in the containment, the method comprising:identifying a set of objects in a field of view of an imaging device; representing, based on an imaging signal from the imaging device, each of the objects in a point cloud:establishing, for each of the objects of the set of objects, a local frame of reference;locating the local frame of reference of each respective object in a world frame of reference including a location of each object in the set of objects;generating a sphere tree of each object from the respective point cloud of the object;ordering each of the objects based on the sphere tree and a volume of the containment;iteratively placing each object of the set of objects in the containment based on a computed no-fit volume, the no-fit volume based on the respective object for placement and an aggregation of the objects previously placed in the containment; andfitting each object in the containment adjacent the objects previously placed in the containment based on a radial shrinkage of the respective sphere tree of the object for placement.

Images