Robotic system with object detection mechanism

The robotic system addresses object detection and grip location challenges in cluttered environments by using center-based heatmaps for improved accuracy and efficiency in object handling.

WO2026239334A1PCT designated stage Publication Date: 2026-11-19MUJIN INC
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
PCT/IB2026/054823
Authority / Receiving Office
WO · WO
Patent Type
Applications
Current Assignee / Owner
Priority Date
2025-05-16
Filing Date
2026-05-14
Publication Date
2026-11-19

AI Technical Summary

Technical Problem

Existing robotic systems face challenges in accurately identifying and gripping objects in cluttered, mixed-SKU stacks and densely packed scenes due to issues like glare, noise, incomplete depth returns, and object occlusions, leading to inefficiencies and errors such as double picks, missed objects, and crushed objects.

Method used

A robotic system generates center-based probability maps or heatmaps to represent the likelihood that points in the image or point cloud belong to the interior surface of an object, using algorithmic techniques and AI models to refine and validate object hypotheses, allowing for more accurate object detection and grip location estimation.

Benefits of technology

The system improves object detection accuracy and grip stability, reducing errors and increasing efficiency by leveraging interior surface analysis, even in challenging environments with glare, noise, and occlusions.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure IB2026054823_19112026_PF_FP_ABST
    Figure IB2026054823_19112026_PF_FP_ABST
Patent Text Reader

Abstract

According to an aspect of an embodiment, a method of operating a robot system may include obtaining image data of a scene including a set of objects. The method may further include generating, based on the image data, a detection hypothesis representing that at least one candidate region within the image data corresponds to an object of the set of objects for detecting and locating the set of objects or a subset thereof depicted in the image data, generating a center heatmap for at least a portion of the at least one candidate region. In some embodiments, the center heatmap may represent relative likelihoods of image data points being a part of the object surface of an object of the set of objects. The method may include verifying the detection hypothesis based on the center heatmap.
Need to check novelty before this filing date? Find Prior Art

Description

P335240.W0.01ROBOTIC SYSTEM WITH OBJECT DETECTION MECHANISMCROSS-REFERENCE TO RELATED APPLICATIONS

[0001] This patent application claims priority to United States Provisional Application No. 63 / 806,720, filed May 16, 2025, which is incorporated herein by reference in its entirety.FIELD

[0002] The embodiments discussed herein are related to systems, processes, and techniques for object detection mechanisms for robotic systems.BACKGROUND

[0003] According to an aspect of an embodiment, a method of detecting objects using a robot may include obtaining image data of a scene including a set of objects. The method may further include generating, based on the image data, a detection hypothesis representing that at least one candidate region within the image data corresponds to an object of the set of objects and generating a center heatmap for at least a portion of at least one candidate region corresponding to an object within the set of objects. In some embodiments, the center heatmap may represent relative likelihoods of image data points being a part of an object surface of the object of the set of objects, such as for detecting / estimating existence and / or location of one object, a center portion of the object surface, or a combination thereof.

[0004] The subject matter claimed herein is not limited to embodiments that solve any disadvantages or that operate only in environments such as those described above. Rather, this background is only provided to illustrate one example technology area where some embodiments described herein may be practiced.SUMMARY

[0005] According to an aspect of an embodiment, a method of operating a robot system may include obtaining image data of a scene including a set of objects. The method may further include generating, based on the image data, a detection hypothesis representing that at least one candidate region within the image data corresponds to an object of the set of objects for detecting and locating the set of objects or a subset thereof depicted in the image data, generating a center heatmap for at least a portion of the at least one candidate14932-7414-0331A1P335240.W0.01region. In some embodiments, the center heatmap may represent relative likelihoods of image data points being a part of the object surface of an object of the set of objects. The method may include verifying the detection hypothesis based on the center heatmap.

[0006] The object and advantages of the embodiments will be realized and achieved at least by the elements, features, and combinations particularly pointed out in the claims. It is to be understood that both the foregoing general description and the following detailed description are exemplary and explanatory and are not restrictive of the invention, as claimed.BRIEF DESCRIPTION OF THE DRAWINGS

[0007] Example embodiments will be described and explained with additional specificity and detail through the use of the accompanying drawings in which:

[0008] FIG. 1 illustrates an example environment in which a robotic system may be implemented, in accordance with at least one embodiment of the present technology in accordance with at least one embodiment of the present technology.

[0009] FIG. 2 is an illustration of an example environment in which a robotic system with an object detection mechanism may operate in accordance with at least one embodiment of the present technology.

[0010] FIG. 3 is a block diagram illustrating the robotic system in accordance with at least one embodiment of the present technology.

[0011] FIG. 4 is a perspective view of a robotic system in accordance with at least one embodiment of the present technology.

[0012] FIG. 5 illustrates a block diagram of a method of operating the robotic system in accordance with at least one embodiment of the present technology.

[0013] FIG. 6A illustrates example two-dimensional image data after preprocessing operations that remove non-object regions in accordance with at least one embodiment of the present technology.

[0014] FIG. 6B illustrates example three-dimensional image data after preprocessing, showing depth-based spatial structure of objects in the scene in accordance with at least one embodiment of the present technology.

[0015] FIG. 7A illustrates example features extracted from the preprocessed image data in accordance with at least one embodiment of the present technology.24932-7414-0331A1P335240.W0.01

[0016] FIG. 7B is image data 710 illustrating bounding regions corresponding to detection hypotheses derived from the extracted corner features in accordance with at least one embodiment of the present technology.

[0017] FIG. 8A illustrates image data for identification and removal of image regions corresponding to a robotic end effector in accordance with at least one embodiment of the present technology.

[0018] FIG. 8B illustrates image data generated based on temporal combination of three-dimensional image data obtained at different times to compensate for end-effector occlusion in accordance with at least one embodiment of the present technology.

[0019] FIG. 8C illustrates a reconstructed three-dimensional view of the scene formed by combining depth data captured at different times to compensate for temporary occlusion by the robotic end effector in accordance with at least one embodiment of the present technology.

[0020] FIG. 8D illustrates a reconstruction process applied to 2D image data in accordance with at least one embodiment of the present technology.

[0021] FIG. 9A illustrates preprocessed image data in accordance with at least one embodiment of the present technology.

[0022] FIG. 9B illustrates segmented image data including detected comers and adjacent edges in accordance with at least one embodiment of the present technology.

[0023] FIG. 9C illustrates an initial detection hypothesis defined based on the detected comers and lines in accordance with at least one embodiment of the present technology.

[0024] FIG. 9D illustrates results of line detection performed on 3D image data corresponding to the segmented image region in accordance with at least one embodiment of the present technology.

[0025] FIG. 9E illustrates a subsequent processing step in which the system expands the detected lines in accordance with at least one embodiment of the present technology.

[0026] FIG. 9F illustrates determination of continuous surface regions derived from the expanded line features shown in FIG. 9E in accordance with at least one embodiment of the present technology.

[0027] FIG. 9G illustrates refinement of the initial detection hypothesis based on overlap between the initial hypothesis region and the continuous areas identified in FIG. 9F in accordance with at least one embodiment of the present technology.

[0028] FIG. 9H illustrates expansion of retained continuous regions to approximate an object surface in accordance with at least one embodiment of the present technology.34932-7414-0331UP335240.W0.01

[0029] FIG. 91 illustrates access to segmented image information corresponding to the expanded continuous areas in accordance with at least one embodiment of the present technology.

[0030] FIG. 9J illustrates determination of a modified detection hypothesis based on evaluating the overlap between the expanded regions and the segmentation information in accordance with at least one embodiment of the present technology.

[0031] FIG. 9K illustrates determination of representative comers for the expanded and refined detection hypothesis in accordance with at least one embodiment of the present technology.

[0032] FIG. 10A illustrates determination of object edges and estimation of object width based on line analysis within segmented image data in accordance with at least one embodiment of the present technology.

[0033] FIG. 10B illustrates further refinement of candidate object edges identified based at least on line-based analysis described with respect to FIG. 10A in accordance with at least one embodiment of the present technology.

[0034] FIG. 10C illustrates determination of an updated or refined object edge within the low-confidence edge region identified in FIG. 10B using depth-based analysis in accordance with at least one embodiment of the present technology.

[0035] FIG. 11A illustrates a portion of preprocessed image data corresponding to a surface of an object for which center-based analysis is to be performed in accordance with at least one embodiment of the present technology.

[0036] FIG. 1 IB illustrates a bounding region defined based on the detected comers in accordance with at least one embodiment of the present technology.

[0037] FIG. 12 illustrates multiple bounding boxes generated based on comer detection in accordance with at least one embodiment of the present technology.

[0038] FIG. 13 illustrates center heatmaps generated for the bounding boxes of FIG. 12 in accordance with at least one embodiment of the present technology.

[0039] FIG. 14A illustrates a reflective region, in which packing tape causes reflection of the light in accordance with at least one embodiment of the present technology.

[0040] FIG. 14B illustrates an example initial detection hypothesis to be revised and / or expanded in accordance with at least one embodiment of the present technology.

[0041] FIG. 14C illustrates center heatmaps corresponding to the objects in FIG. 14B in accordance with at least one embodiment of the present technology.44932-7414-0331A1P335240.W0.01

[0042] FIG. 14D illustrates determination of centered regions derived based on the center heatmaps in accordance with at least one embodiment of the present technology.

[0043] FIG. 14E illustrates verification of the initial detection hypothesis using the centered region derived from the corresponding center heatmap in accordance with at least one embodiment of the present technology.

[0044] FIG. 14F illustrates expansion of the detection hypothesis using information derived from candidate edges and the extent of the centered region in accordance with at least one embodiment of the present technology.

[0045] FIG. 15A illustrates detected line segments in accordance with at least one embodiment of the present technology.

[0046] FIG. 15B illustrates a superimposed representation of 2D and 3D image data for a portion of the scene that includes one or more objects in accordance with at least one embodiment of the present technology.

[0047] FIG. 15C illustrates results of line detection performed on the image data depicted in FIG. 15B in accordance with at least one embodiment of the present technology.

[0048] FIG. 15D illustrates generation of center heatmaps based on the same image data and extracted features used in FIG. 15C in accordance with at least one embodiment of the present technology.

[0049] FIG. 15E illustrates estimation of an object boundary using centeredness values derived from one or more center heatmaps in accordance with at least one embodiment of the present technology.

[0050] FIG. 15F illustrates a validated detection hypothesis generated following refinement and validation of a detection hypothesis based on center heatmap-based boundary estimation in accordance with at least one embodiment of the present technology.

[0051] FIG. 16A illustrates an example of a center heatmap generated for a candidate object surface as part of a center heatmap validation subprocess in accordance with at least one embodiment of the present technology.

[0052] FIG. 16B illustrates a distribution of centeredness values derived from the center heatmap of FIG. 16A in accordance with at least one embodiment of the present technology.54932-7414-0331UP335240.W0.01

[0053] FIG. 16C illustrates an expected distribution of centeredness values corresponding to a valid center heatmap in accordance with at least one embodiment of the present technology.

[0054] FIG. 16D illustrates another example of center heatmap validation in accordance with at least one embodiment of the present technology.

[0055] FIG. 17A illustrates a detection hypothesis generated prior to center heatmap-based validation in accordance with at least one embodiment of the present technology.

[0056] FIG. 17B illustrates validation of the detection hypothesis using a corresponding center heatmap in accordance with at least one embodiment of the present technology.

[0057] FIG. 17C illustrates a detection hypothesis subject to validation based on center heatmap in accordance with at least one embodiment of the present technology.

[0058] FIG. 17D illustrates validation of the detection hypothesis based on a corresponding center heatmap in accordance with at least one embodiment of the present technology.

[0059] FIG. 18 is a flowchart of an example method for detecting objects using center heatmaps in accordance with at least one embodiment of the present technology.DESCRIPTION OF EMBODIMENTS

[0060] Robotic systems used in warehouses and distribution centers utilize a combination of hardware components (e.g., a robotic arm, an end effector, sensors, and the like) and software to move items, such as boxes, without direct human handling or intervention. The robotic systems typically depend on vision systems that capture both visual imagery and depth information, providing a sense of locations of surfaces, gaps, and / or obstacles.

[0061] Prior to making any operative decisions, the raw sensor data is cleaned up and simplified. This can include combining views from multiple cameras, filtering out obstacles and / or parts of the scene that are not relevant, and generating basic geometric descriptions such as surface orientation or relative height. The clean-up process may not provide perfect reconstruction but provides a usable representation of the scene that stays consistent as objects are removed and the environment changes.

[0062] Based on the environment, perception is about making sense of cluttered visual cues. Objects in the environment, such as boxes, may look similar to one another, lack64932-7414-0331UP335240.W0.01clear marking, and / or sit adjacent to neighboring boxes, making boundaries difficult to identify for computing systems. Surfaces can be reflective or damaged, which adds noise and uncertainty. Such factors often increase the difficulties in computing a grip location for each object. In transferring the objects, it is essential for the robotic system to identify where it can safely make contact with and grip an object, as grip stability typically improves with the grip location moving away from edges and corners. Such aspects push perception beyond merely finding outlines toward understanding how interior surface regions are distributed across an object surface.

[0063] Existing approaches to object detection in such environments often begin by extracting visual features, typically edges and comers, and grouping the features into bounding regions that are assumed to correspond to individual boxes or other objects. In some instances, segmentation or template matching is applied to refine the regions, and depth information is layered to estimate object boundaries or to prioritize objects closest to the robot. More recent systems incorporate learned models that classify or segment objects directly from sensor data. Such techniques have permitted partial automation, particularly in controlled scenarios such as single-stock-keeping-unit (SKU) stacks viewed from a top-down perspective, in which surfaces tend to be planar and separations between objects are relatively clear.

[0064] However, the limitations of such approaches become apparent under different parameters, such as for mixed-SKU stacks and / or densely packed scenes, particularly in instances when cartons are adjacent, partially occluded, rotated, or covered with reflective tapes and / or labels. Moreover, when approaching the objects along a lateral direction, such as when removing objects from a container or placing objects into a container, some physical factors (e.g., box rotations, consideration of depth against stacked / supporting objects) can expose even greater limitations of the existing approaches. Additionally, feature extraction based on edges and corners tend to degrade under glare, noise, and / or incomplete depth returns, and small local errors may propagate into incorrect object boundaries. Template matching struggles in instances objects deviate from known poses or in instances surfaces lack distinguishing markings. Vision-in-hand solutions (e.g., visual sensors attached to the robotic arm and / or end-effectors) can improve visibility but cause slower overall throughput due to repeated repositioning. Such limitations may lead to problems such as double picks, missed objects, crushed objects, dropped objects, and the like. Using the conventional methods to account for such limitations may result in overly conservative grasp strategies that reduce efficiency.74932-7414-0331UP335240.W0.01

[0065] Embodiments of the present technology may address such issues based on a parallel mechanism for reasoning about object interiors, rather than merely relying on object boundaries. At a high level, the system generates center-based probability maps or heatmaps that represent the likelihood that points in the image or point cloud belong to the interior surface of a single object. In some embodiments, such heatmaps are produced alongside, rather than as a byproduct of traditional hypothesis generation, and the heatmaps serve as a complementary signal for expansion, refinement, and validation of object hypotheses. The overall flow still permits for comers, edges, normal, or minimum-viable regions to seed candidate detections, but the seeds can be treated as provisional and subject to further evaluation based on the heatmaps.

[0066] In some embodiments, the heatmaps are generated using algorithmic techniques grounded in geometry and spatial relationships inferred from sensor data. A candidate region can be first identified, often originating from a rough bounding box, a minimum viable region (MVR), and / or another provisional area believed to lie on the surface of a single object. The region can have flexibility or a relatively large error tolerance with respect to matching the true object boundaries. Rather, the region serves as a working area in which interior structure can be evaluated. A representative surface center can be estimated from the region, for example by locating the intersection of diagonals of an inferred rectangular face or by computing a central tendency of surface points. For example, for each pixel or three-dimensional (3D) point within the candidate region, a centeredness value can be calculated based on relative distances to the estimate center and to nearby region boundaries. Points closer to the interior tend to receive higher centeredness values, and points nearer to edges or transitions tend to receive lower values.

[0067] The centeredness values accumulated across the candidate region can produce a smooth probability landscape rather than a binary decision. Interior regions of an object can naturally reinforce one another, which may permit one or more dominant interior peaks to merge even in the presence of glare, missing depth data, and / or fragmented edges. The interior peaks are used to guide further processing. For example, a detection hypothesis can be expanded outward from the most likely interior region until surrounding evidence no longer supports inclusion. In some instances, multiple interior peaks may appear within the same candidate region, indicating that the region may include more than one object, prompting the hypothesis to be divided accordingly. In scenes in which adjacent objects are tightly packed, boundaries can be estimated by evaluating regions of reduced84932-7414-0331A1P335240.W0.01centeredness between neighboring heatmaps, which often correspond to separation between object interiors.

[0068] In some embodiments, the heatmaps may be generated using learned models. The centeredness is not derived directly from geometric distance calculations. Rather, pre-processed sensor data is provided to an artificial intelligence (Al) model that has been trained to output center probability distribution for object surfaces. The preprocessing stage used to generate and / or format inputs for the Al model may include masking portions of the scene that are known to be unreliable, extracting surface normal, emphasizing continuous planar regions, and / or otherwise shaping the input so that the most relevant interior cues are preserved while reducing unnecessary data complexity. As an illustrative example, a computing / robotic system implementing the preprocessing stage can take two-dimensional (2D) images and three-dimensional (3D) images (e.g., depth maps or point clouds) from the sensors, which would otherwise be evaluated on a per-pixel level by the Al model, and drastically reduce the overall size of the input data into predetermined constructs and parameters. The magnitude reduction and distillation into key features (e.g., represented by the predetermined constructs / parameters) can increase the efficiency, the accuracy, the resource management, and more in both training the Al models and implementing the Al models for inference computations.

[0069] Training data for the Al model may be obtained from prior detections and / or successful picks, permitting the model to learn what valid object interiors tend to look like under a range of conditions. For example, the training data may include various detections and / or picks associated with different object sizes, orientations, lighting effects, and surface artifacts such as tape or labels. The model may be trained to develop an internal representation of interior regions that does not depend on explicit corner or edge detection, such that the model may generate heatmaps serving the same functional role as the algorithmic heatmaps.

[0070] Embodiments of the present technology will be explained with reference to the accompanying drawings.

[0071] FIG. 1 illustrates an example environment 102 in which a robotic system 104 may be implemented, in accordance with at least one embodiment of the present technology. The environment 102 may correspond to an industrial, logistics, or warehouse setting, such as cargo container, trailer, storage area, or other workspace in which objects 116 are arranged for handling, transport, and / or manipulation. The objects 116 may include boxes, packages, containers, or other items positioned within the environment 10294932-7414-0331A1P335240.W0.01and accessible to the robotic system 104. In the present technology, a reference to a system may generally include a reference to the robotic system 104.

[0072] The robotic system 104 includes at least one processor 106, vision system 108, an end effector 110, an artificial intelligence (Al) model 112, and a memory 114. The components of the robotic system 104 may be located within a single robotic platform or distributed across multiple hardware modules / unites that are configured to communicate via wired and / or wireless interfaces.

[0073] In general, the processor 106 may include any suitable special purpose or general -purpose computer, computing entity, or processing device including various computer hardware or software modules and may be configured to execute instructions stored on any applicable computer-readable storage media. For example, the processor 106 may include a microprocessor, a microcontroller, a digital processor (DSP), an applicationspecific integrated circuit (ASIC), a Field-Programmable Gate Array (FPGA), a central processing unit (CPU), a graphical processing unit (GPU), accelerators, or any other digital or analog circuitry configured to interpret and / or to execute program instructions and / or to process data. Although illustrated as a single processor in FIG. 1, the processor 106 may include any number of processors configured to, individual or collectively, perform or direct performance of any number of operations described in the present technology.

[0074] The vision system 108 may be configured to acquire sensor data representing the environment 102 and the objects 116 in the environment 102. In some embodiments, the vision system 108 may include one or more 2D imaging sensors, one or more 3D depth / di stance sensors, or a combination thereof, arranged to capture overlapping and / or complementary views of the environment 102. In some embodiments, the vision system may include sensors arranged to provide complementary views of the environment 102. For example, the vision system 108 may include a left vision system and a right vision system positioned at laterally offset locations relative to the robotic system 104. Each of the left vision system and the right vision system may include one or more 2D imaging sensors and / or 3D depth sensors configured to capture overlapping but distinct perspectives of the objects 116.

[0075] The vision system 108 may capture visual information describing object appearance, such as intensity or texture, as well as spatial information describing depth, surface orientation, and / or relative positioning of object surfaces. In some embodiments, the vision system 108 may be configured to perform initial preprocessing operations, such104932-7414-0331UP335240.W0.01as sensor calibration, alignment of data from multiple viewpoints, and / or basic filtering to reduce noise or remove irrelevant regions.

[0076] The end effector 110 may be configured to physically interact with the objects 116 within the environment 102. In some embodiments, the end effector 110 may include one or more object-engagement mechanisms, such as a gripper, suction device, clamp, or other manipulation tool, capable of contacting, securing, lifting, and / or repositioning an object. The end effector 110 operates under control of the processor 106 and is actuated based on information derived from the image data obtained by the vision system 108. In some embodiments, the processor 106 generates control commands for the end effector 110 in response to a verified detection hypothesis, a generated center heatmap, and / or a determined grasp location. The end effector 110 may further provide feedback to the processor 106, such as position, contact state, or interaction status, which may be used to update subsequent perception or control operations.

[0077] The Al model 112 may be implemented as a software module executed by the processor 106 or as a dedicated hardware accelerator. The Al model 112 may be configured to process image data or intermediate representations generated by the processor 106 to perform inference tasks, such as estimating object properties, generating probability maps, and / or assisting in validation or refinement of perception results. In some embodiments, the Al model 112 operates in conjunction with non-AI algorithms, with the processor 106 combining outputs from both approaches to guide system behavior.

[0078] The memory 114 stores data and instructions used by the robotic system 104. In some embodiments, the memory 114 stores executable instructions for the processor 106, parameters for the Al model 112, reference data associated with the vision system 108, and intermediate or historical data such as previously captured image data, detection results, and / or interaction outcomes. The memory 114 permits the robotic system 104 to maintain state information across multiple perception and interaction cycles.

[0079] The robotic system 104 is configured to observe the environment 102, interpret the spatial arrangement and properties of the objects 116 within the environment 102, and perform physical interactions based on the interpretation. The robotic system 104 functions as a coordinated system in which sensing, computation, decision-making, and actuation are closely coupled. Rather than operating as independent subsystems, the components of the robotic system 104 exchange information continuously, permitting perception results to influence physical actions and, conversely, permitting physical actions to inform subsequent perception. The robotic system 104 may operate in such closed-loop manner114932-7414-0331UP335240.W0.01to adapt to changes in the environment 102, such as object removal, reorientation, and / or partial occlusion.

[0080] The robotic system 104 is capable of maintaining and updating an internal representation of the environment across multiple perception-action cycles. For example, the robotic system 104 may reuse previously obtained image data, remove regions corresponding to objects that have already been handled or removed, and / or reconstruct portions of the scene that were temporarily occluded in manipulation. The objection identification, and particularly the detection hypothesis generation and verification of the robotic system 104 may be described in further detail with respect to the subsequent figures. In the subsequent figures, a reference to a system may generally include a reference to the robotic system 104.

[0081] Modifications, additions, or omissions may be made to the robotic system 104 without departing from the scope of the present technology. For example, in some embodiments, the robotic system 104 may include any number of other components that may not be explicitly illustrated or described.

[0082] FIG. 2 is an illustration of an example environment in which a robotic system 200 with an object detection mechanism may operate. The robotic system 200 can include and / or communicate with one or more units (e.g., robots) configured to execute one or more tasks. Aspects of the coordinated transfer mechanism can be practiced or implemented by the various units. The robotic system 200 may correspond to the robotic system 104 of FIG. 1.

[0083] For the example illustrated in FIG. 2, the robotic system 200 can include an unloading unit 202, a transfer unit 204 (e.g., a palletizing robot and / or a piece-picker robot), a transport unit 206, a loading unit 208, or a combination thereof in a warehouse or a distribution / shipping hub. Each of the units in the robotic system 200 can be configured to execute one or more tasks. The tasks can be combined in sequence to perform an operation that achieves a goal, such as to unload objects from a truck or a van and store them in a warehouse or to unload objects from storage locations and prepare them for shipping. In some embodiments, the task can include placing the objects on a target location (e.g., on top of a pallet and / or inside a bin / cage / box / case).

[0084] In order to execute the tasks, the robotic system 200 can derive individual placement locations / orientations, calculate corresponding motion plans, or a combination thereof, such as for placing and / or stacking the objects. Each of the units can be configured to execute a sequence of actions (e.g., operating one or more components therein)124932-7414-0331UP335240.W0.01according to the placement locations / orientations, the motion plans, and / or the like to execute a task. In some embodiments, the robotic system 200 can include (1) sensors 201 that provide various input data and (2) at least one controller 209 that processes the sensor outputs to compute the locations / orientations, motion plans, and / or the like. For example, the sensors 201 can include a camera configured to obtain 2D and / or 3D image data of various locations and / or objects. The controller 209 can process the image data to derive the locations / orientations, the motion plan, and / or the like.

[0085] One example of the task executed through the robotic system 200 can include manipulation (e.g., moving and / or reorienting) of a target object 212 (e.g., one of the packages, boxes, cases, cages, pallets, etc. corresponding to the executing task) from a start / source location 214 to a task / destination location 216. For example, the unloading unit 202 (e.g., a devanning robot) can be configured to transfer the target object 212 from a location in a carrier (e.g., a truck) to a location on a conveyor. Also, the transfer unit 204 can be configured to transfer the target object 212 from one location (e.g., the conveyor, a pallet, or a bin) to another location (e.g., a pallet, a bin, etc.). For another example, the transfer unit 204 (e.g., a palletizing robot) can be configured to transfer the target object 212 from a source location (e.g., a pallet, a pickup area, and / or a conveyor) to a destination pallet. In completing the operation, the transport unit 206 (e.g., a conveyor, an automated guided vehicle (AGV), a shelf-transport robot, etc.) can transfer the target object 212 from an area associated with the transfer unit 204 to an area associated with the loading unit 208, and the loading unit 208 can transfer the target object 212 (by, e.g., moving the pallet carrying the target object 212) from the transfer unit 204 to a storage location (e.g., a location on the shelves). The object 212 may correspond to the objects 116 of FIG. 1.

[0086] For illustrative purposes, the robotic system 200 is described in the context of a packaging and / or shipping center; however, it is understood that the robotic system 200 can be configured to execute tasks in other environments / for other purposes, such as for manufacturing, assembly, storage / stocking, healthcare, and / or other types of automation. It is also understood that the robotic system 200 can include other units, such as manipulators, service robots, modular robots, etc., not shown in FIG. 2. For example, in some embodiments, the robotic system 200 can include a depalletizing unit for transferring the objects from cage carts or pallets onto conveyors or other pallets, a container-switching unit for transferring the objects from one container to another, a packaging unit for wrapping / casing the objects, a sorting unit for grouping objects according to one or more characteristics thereof, a piece-picking unit for manipulating (e.g., for sorting, grouping,134932-7414-0331UP335240.W0.01and / or transferring) the obj ects differently according to one or more characteristics thereof, or a combination thereof.Suitable System

[0087] FIG. 3 is a block diagram illustrating the robotic system 300 in accordance with one or more embodiments of the present technology. In some embodiments, for example, the robotic system 200 (e.g., at one or more of the units and / or robots described above) can include electronic / electrical devices, such as one or more processors 302, one or more storage devices 304, one or more communication devices 306, one or more input-output devices 308, one or more actuation devices 312, one or more transport motors 314, one or more sensors 316, or a combination thereof. The various devices can be coupled to each other via wire connections and / or wireless connections. For example, the robotic system 200 can include a bus, such as a system bus, a Peripheral Component Interconnect (PCI) bus or PCI-Express bus, a HyperTransport or industry standard architecture (ISA) bus, a small computer system interface (SCSI) bus, a universal serial bus (USB), an IIC (I2C) bus, or an Institute of Electrical and Electronics Engineers (IEEE) standard 1394 bus (also referred to as "Firewire"). Also, for example, the robotic system 200 can include bridges, adapters, processors, or other signal-related devices for providing the wire connections between the devices. The wireless connections can be based on, for example, cellular communication protocols (e.g., 3G, 4G, LTE, 5G, etc.), wireless local area network (LAN) protocols (e.g., wireless fidelity (Wi-Fi)), peer-to-peer or device-to-device communication protocols (e.g., Bluetooth, Near-Field communication (NFC), etc.), Internet of Things (loT) protocols (e.g., NB-IoT, LTE-M, etc.), and / or other wireless communication protocols.

[0088] The processors 302 can include data processors (e.g., central processing units (CPUs), special-purpose computers, and / or onboard servers) configured to execute instructions (e.g., software instructions) stored on the storage devices 304 (e.g., computer memory). In some embodiments, the processors 302 can be included in a separate / standalone controller (e.g., the controller 209 of FIG. 2) that is operably coupled to the other electronic / electrical devices illustrated in FIG. 3 and / or the robotic units illustrated in FIG. 2. The processors 302 can implement the program instructions to control / interface with other devices, thereby causing the robotic system 200 to execute actions, tasks, and / or operations.144932-7414-0331UP335240.W0.01

[0089] The storage devices 304 can include non-transitory computer-readable mediums having stored thereon program instructions (e.g., software). Some examples of the storage devices 304 can include volatile memory (e.g., cache and / or random-access memory (RAM)) and / or nonvolatile memory (e.g., flash memory and / or magnetic disk drives). Other examples of the storage devices 304 can include portable memory and / or cloud storage devices.

[0090] In some embodiments, the storage devices 304 can be used to further store and provide access to processing results and / or predetermined data / thresholds. For example, the storage devices 304 can store master data 352 that includes descriptions of objects (e.g., boxes, cases, and / or products) that may be manipulated by the robotic system 200. In one or more embodiments, the master data 352 can include a dimension, a shape (e.g., templates for potential poses and / or computer-generated models for recognizing the object in different poses), a color scheme, an image, identification information (e.g., bar codes, quick response (QR) codes, logos, etc., and / or expected locations thereof), an expected weight, other physical / visual characteristics, or a combination thereof for the objects expected to be manipulated by the robotic system 200. In some embodiments, the master data 352 can include manipulation-related information regarding the objects, such as a center-of-mass (CoM) location on each of the obj ects, expected sensor measurements (e.g., for force, torque, pressure, and / or contact measurements) corresponding to one or more actions / maneuvers, or a combination thereof.

[0091] The communication devices 306 can include circuits configured to communicate with external or remote devices via a network. For example, the communication devices 306 can include receivers, transmitters, modulators / demodulators (modems), signal detectors, signal encoders / decoders, connector ports, network cards, etc. The communication devices 306 can be configured to send, receive, and / or process electrical signals according to one or more communication protocols (e.g., the Internet Protocol (IP), wireless communication protocols, etc.). In some embodiments, the robotic system 200 can use the communication devices 306 to exchange information between units of the robotic system 200 and / or exchange information (e.g., for reporting, data gathering, analyzing, and / or troubleshooting purposes) with systems or devices external to the robotic system 200.

[0092] The input-output devices 308 can include user interface devices configured to communicate information to and / or receive information from human operators. For example, the input-output devices 308 can include a display 310 and / or other output154932-7414-0331UP335240.W0.01devices (e.g., a speaker, a haptics circuit, or a tactile feedback device, etc.) for communicating information to the human operator. Also, the input-output devices 308 can include control or receiving devices, such as a keyboard, a mouse, a touchscreen, a microphone, a user interface (UI) sensor (e.g., a camera for receiving motion commands), a wearable input device, etc. In some embodiments, the robotic system 200 can use the input-output devices 308 to interact with the human operators in executing an action, a task, an operation, or a combination thereof.

[0093] The robotic system 200 can include physical or structural members (e.g., robotic manipulator arms) that are connected at joints for motion (e.g., rotational and / or translational displacements). The structural members and the joints can form a kinetic chain configured to manipulate an end-effector (e.g., the gripper) configured to execute one or more tasks (e.g., gripping, spinning, welding, etc.) depending on the use / operation of the robotic system 200. The robotic system 200 can include the actuation devices 312 (e.g., motors, actuators, wires, artificial muscles, electroactive polymers, etc.) configured to drive or manipulate (e.g., displace and / or reorient) the structural members about or at a corresponding joint. In some embodiments, the robotic system 200 can include the transport motors 314 configured to transport the corresponding units / chassis from place to place.

[0094] The robotic system 200 can include the sensors 316 (e.g., the sensors 201 of FIG. 2) configured to obtain information used to implement the tasks, such as for manipulating the structural members and / or for transporting the robotic units. The sensors 316 can include devices configured to detect or measure one or more physical properties of the robotic system 200 (e.g., a state, a condition, and / or a location of one or more structural members / joints thereof) and / or of a surrounding environment. Some examples of the sensors 316 can include accelerometers, gyroscopes, force sensors, strain gauges, tactile sensors, torque sensors, position encoders, etc.

[0095] In some embodiments, for example, the sensors 316 can include one or more imaging devices 322 (e.g., visual and / or infrared cameras, 2D and / or 3D imaging cameras, distance measuring devices such as LIDARs or radars, etc.) configured to detect the surrounding environment. The imaging devices 322 can generate representations of the detected environment, such as digital images and / or point clouds, which may be processed via machine / computer vision (e.g., for automatic inspection, robot guidance, or other robotic applications). As described in further detail below, the robotic system 200 (via, e.g., the processors 302) can process the digital image and / or the point cloud to identify164932-7414-0331UP335240.W0.01the target object 212 of FIG. 2, the start location 214 of FIG. 2, the task location 216 of FIG. 2, a pose of the target object 212, a confidence measure regarding the start location 214 and / or the pose, or a combination thereof.

[0096] For manipulating the target object 212, the robotic system 200 (via, e.g., the various circuits / devices described above) can capture and analyze image data of a designated area (e.g., a pickup location, such as inside the truck or on the conveyor belt) to identify the target object 212 and the start location 214 thereof. Similarly, the robotic system 200 can capture and analyze image data of another designated area (e.g., a drop location for placing objects on the conveyor, a location for placing objects inside the container, or a location on the pallet for stacking purposes) to identify the task location 216. For example, the imaging devices 322 can include one or more cameras configured to generate image data of the pickup area and / or one or more cameras configured to generate image data of the task area (e.g., drop area). Based on the image data, as described below, the robotic system 200 can determine the start location 214, the task location 216, the associated poses, a packing / placement location, and / or other processing results.

[0097] In some embodiments, for example, the sensors 316 can include position sensors 324 (e.g., position encoders, potentiometers, etc.) configured to detect positions of structural members (e.g., the robotic arms and / or the end-effectors) and / or corresponding joints of the robotic system 200. The robotic system 200 can use the position sensors 324 to track locations and / or orientations of the structural members and / or the joints during execution of the task.Computer Vision System

[0098] Returning to image processing, the robotic system 200 can include a computer vision system implemented using the processor 302, the storage device 304, the imaging devices, portions thereof, or combinations thereof. The computer vision system can provide machine-level recognition, detection, identification, and / or related data processing for objects, things, persons, etc. depicted in the image data.

[0099] The computer vision system can process one or more of various types of data. For example, the computer vision system can process 2D and / or 3D image data. Even within one category, such as 3D image data, the computer vision system can process a variety of file types. For example, the computer vision system can receive and process a point cloud as the 3D image data. Moreover, the point cloud can have an unstructured format or a structured format.174932-7414-0331UP335240.W0.01

[0100] Unstructured image data typically corresponds to 3D image data being stored with an unspecified predetermined structure. For instance, the unstructured image data can store valid data points along with their coordinates (e.g., x, y, and z coordinates) while excluding exclude data points and / or locations determined to be NaN, such as when no object / structure is detected within a sensing distance / range. Stated differently the unstructured image data can positively identify data associated with detected objects / structures and communicate through inference that locations that have not been named represent NaN or no detections.

[0101] On the other hand, structured image data typically corresponds to 3D image data being stored with a specified predetermined structure, such as a grid-like format. For instance, the structured image data communicates and / or store the data according to a predetermined pattern of the x and y coordinates. Hence, the structured image data can include a value for each predetermined location (e.g., a unique x-y pairing) within a field-of-view. Accordingly, the structured image data can infer the x-y coordinates of each data point / value according to the predetermined format. For locations without any detected objects / structures, the structured image data can include NaN values (e.g., a predetermined value, typically a maximum or a minimum bit-value). In some usage cases, using an unstructured point cloud allows for more flexible applications and potentially greater accuracy, while a structured point cloud can result in reduced data usage and faster computation of that data. In other usage cases, such as for sparsely populated environment, the unstructured point cloud can reduce the overall data size, while the structure point cloud may provide additional details that may be missed or filtered out in the unstructured point cloud.

[0102] In communicating or processing the 3D image data, it may be desirable to reduce the size and / or complexity of the image data. Conventional systems use compression algorithms, such as standard (zstd). Zstd may be considered a lossless algorithm that provides relatively fast compression and decompression speeds. In exchange for the relatively fast compression / decompression, zstd trades away the overall compression rate, i.e., zstd provides relatively nominal decrease in the overall data size. Depending on the parameters selected, conventional systems can use zstd to reduce an unstructured point cloud data file of 26.09MB to 22.87MB.

[0103] In some situations, it may be possible to utilize structured point cloud data. Continuing with the example depiction above, the structured point cloud may depict the same scene with a data size of about 12.75MB. If a similar zstd compression algorithm is184932-7414-0331A1P335240.W0.01run on such data, the size may be further reduced to approximately 6.99MB. Despite the reduction in file size, zstd compression may still not be sufficient to serve certain purposes. Additionally, when the data compressed by zstd is to be used, it must first be decompressed before it can be used, which can take time and computational resources. Accordingly, more suitable compression methods may be desired.

[0104] FIG. 4 is a perspective view of a robotic system 400 in accordance with embodiments of the present technology. The robotic system 400 can be an example of the robotic system 104 (e.g., the endpoint unit 102) illustrated in and described above with respect to FIG. 1. In the illustrated embodiment, the robotic system 400 includes a chassis 402, a conveyor arm or first segment 404 (“first segment”) coupled to the chassis 402 and extending toward a distal portion 401a of the robotic system 400, a second segment 421 coupled to the chassis 402 and extending toward a proximal portion 401b of the robotic system 400, and a gripper 406 coupled to the first segment 404 at the distal portion 401a. As discussed above with respect to the robotic system 104, the robotic system 400 can be configured to execute one or more tasks to perform an operation that achieves a goal, such as to unload objects from a cargo carrier (e.g., a truck, a van) and store them in a warehouse, or to unload objects from storage locations and prepare them for shipping (e.g., load them into the cargo carrier). For example, in some embodiments, the robotic system 400 can be positioned such that the second segment 421 is adjacent to a warehouse conveyor (e.g., a conveyor previously / already in place within the operating environment). Objects can then be conveyed along a path formed by the warehouse conveyor, the second segment 421, the chassis 402, and the first segment 404 toward or away from the gripper 406. As discussed in further detail herein, the chassis 402, the first segment 404, the second segment 421, and the gripper 406 can be actuated to various positions and / or angular positions, or otherwise operated, such that objects can be conveyed or transferred between the warehouse to the cargo carrier in a desired and efficient manner.

[0105] The robotic system 400 can also include supporting legs 410 coupled to the chassis 402, one or more controllers 438 supported by the chassis 402, first joint rollers 409 coupled between the first segment 404 and the gripper 406, and second joint rollers 437 coupled between the first segment 404 and the second segment 421. The chassis 402, the first segment 404, the second segment 421, the sensor arms 430, the supporting legs 410, and / or other components of the robotic system 400 can be made from metal (e.g., aluminum, stainless steel), plastic, and / or other suitable materials.194932-7414-0331UP335240.W0.01

[0106] The chassis 402 can include a frame structure that supports the first segment 404, the second segment 421, the controllers 438, and / or one or more sensor arms 430 coupled to the chassis 402. In the illustrated embodiment, two sensor arms 430 each extend vertically on either side of the first segment 404. An upper sensor 424 (e.g., an upper vision sensor) and a lower sensor 425 (e.g., a lower vision sensor) are coupled to each sensor arm 430 along a vertical direction and are positioned to generally face toward the distal portion 401a.

[0107] The first segment 404 is coupled to extend from the chassis 402 toward the distal portion 401a in a cantilevered manner. The first segment 404 supports a first conveyor 405 (e.g., a conveyor belt) extending along and / or around the first segment 404. Similarly, the second segment 421 is coupled to extend from the chassis 402 in a cantilevered manner, but toward a proximal portion 401b of the robotic system 400. The second segment 421 supports a second conveyor 422 (e.g., a conveyor belt) extending along and / or around the second segment 421. In some embodiments, one or more actuators 436 (e.g., motors) configured to move the first and second conveyors 405, 422 are coupled to the chassis 402. In some embodiments, the actuators are positioned elsewhere (e.g., housed in or coupled to the first and / or second segments 404, 421). The actuators 436 can also be operated to rotate the first segment 404 about a first axis Ai and / or a second axis A2. As illustrated in FIG. 4, the first axis Ai can be generally orthogonal to a transverse plane of the chassis 402 while the second axis A2 can be generally parallel to the transverse plane of the chassis 402. Said another way, the first axis Ai can be in a first plane that is generally orthogonal to a second plane containing the second axis A2. In some embodiments, the actuators 436 can also pivot the second joint rollers 437 about the first and second axes Ai, A2 or different axes.

[0108] As mentioned above, the gripper 406 can be coupled to extend from the first segment 404 toward the distal portion 401a with the first joint rollers 409 positioned therebetween. In some embodiments, the gripper 406 is configured to grip objects using a vacuum and to selectively release the objects. The gripper 406 can include suction cups 440 (and / or any other suitable gripping element, such as a magnetic component, a mechanical gripping component, and / or the like, sometimes referred to generically as “gripper elements,” “gripping elements,” and / or the like) and / or a distal conveyor 442. The suction cups 440 can pneumatically grip objects such that the suction cups 440 can carry and then place the object the distal conveyor 442, which in turn transports the object in the proximal direction.204932-7414-0331UP335240.W0.01

[0109] In some embodiments, one or more actuators 408 (e.g., motors) are configurated to rotate the gripper 406 and / or the first joint rollers 409 relative to the first segment 404 about a third axis A3 and / or a fourth axis A4. As illustrated in FIG. 4 and discussed in more detail below with reference to FIGS. 40-44C, the third axis A3 can be generally parallel to a longitudinal plane of the gripper 406 while the fourth axis A4 can be generally orthogonal to the longitudinal plane of the gripper 406. Additionally, or alternatively, the third axis A3 can be generally orthogonal to a transverse plane of the gripper 406 while the fourth axis A4 can be generally parallel orthogonal to the transverse plane of the gripper 406. Said another way, the third axis A3 can be in a third plane that is generally orthogonal to a fourth plane containing the fourth axis A4. In some embodiments, as discussed in more detail below, the robotic system 400 can maintain the transverse plane of the gripper 406 generally parallel with the transverse plane of the chassis 402 (e.g., such that rotation about the second axis A2 is met with an opposite rotation about the fourth axis A4. As a result, for example, in some embodiments, the third axis A3 can be generally orthogonal to the transverse plane of the chassis 402 and / or the fourth axis A4 can be generally parallel to the transverse plane of the chassis 402.

[0110] In some embodiments, the actuators 408 are configured to operate the suction cups 440 and / or the distal conveyor 442. In some embodiments, the actuators 408 are coupled to the first segment 404, the first joint rollers 409, and / or the gripper 406. Movement and / or rotation of the gripper 406 relative to the second segment 404 and components of the gripper 406 are described in further detail below.

[0111] In the illustrated embodiment, two supporting legs 410 are rotatably coupled to the chassis 402 about pivots 416 positioned on either side of the chassis 402. A wheel 412 is mounted to a distal portion of each supporting leg 410. The chassis 402 also supports actuators 414 (e.g., linear actuators, motors) operably coupled to the supporting legs 410. In some embodiments, the robotic system 400 includes fewer or more supporting legs 410, and / or supporting legs 410 configured in different positions and / or orientations. In some embodiments, the wheels 412 can be motorized to move the chassis 402, and thus the rest of the robotic system 400, along linear direction LI. Operation of the actuators 414 is described in further detail below with respect to FIG. 4.

[0112] The controllers 438 can be operably coupled (e.g., via wires or wirelessly) to control the actuators 408, 436, 414. In some embodiments, the controllers 438 are positioned to counterbalance the moment exerted on the chassis 402 by, for example, the214932-7414-0331UP335240.W0.01cantilevered first segment 404. In some embodiments, the robotic system 200 includes counterweights coupled to the chassis 402 to counterbalance such moments.

[0113] As an illustrative example, the robotic system 400 can be configured to provide an interface and operate between (1) the cargo carrier located at or about the distal portion 401a and (2) an existing object handling component, such as a conveyor preinstalled at the truck bay, located at or about the proximal portion 401b. The supporting legs 410 can allow the chassis 402 and / or the second segment 421 to be positioned over and / or overlap the existing object handling component. For example, the supporting legs 410 can be adjacent to or next to peripheral surfaces of the warehouse conveyor and position the chassis 402 and / or the second segment 421 over and / or partially overlapping an end portion of the warehouse conveyor.

[0114] Based on the relative arrangement described above, the robotic system 400 can automatically transfer target objects between the cargo carrier and the warehouse conveyor. Using the devanning process as an illustrative example, the robotic system 400 can use the first segment 404 to successively / iteratively position the EOAT (e.g., the gripper 406) adjacent to or in front of target objects located / stacked in the cargo carrier. The robotic system 400 can use the EOAT to (1) grasp and initially remove the target object from the cargo carrier and (2) place / release the grasped target object onto the first joint rollers 409 and / or the first conveyor 405. The robotic system 400 can operate the connected sequence of rollers and conveyors, such as the first joint rollers 409, the first conveyor 406, the second joint rollers 437, the second conveyor 422, etc., to transfer the target object from the EOAT to the warehouse conveyor.

[0115] In transferring the target objects, the robotic system 400 can analyze the sensor information, such as one or more image data (e.g., 2D and / or 3D data) and other observed physical characteristics of the objects. For example, the mixed SKU environment can have objects of different types, sizes, etc. stacked on top of and adjacent to each other. The coplanar surfaces (e.g., front surfaces) of the stacked objects can form walls or vertical planes that extend at least partially across a width and / or a height inside the cargo carrier. The robotic system 400 can use 2D and / or 3D image data from the vision sensors 424 and / or 425 to initially detect objects within the cargo carrier. The detection operation can include identifying edges, calculating and assessing dimensions of or between edges, assessing surface texture, such as visual characteristics including codes, numbers, letters, shapes, drawings, or the like identifying the object or its contents. The robotic system 400 can compare the sensor outputs and / or the derived data to attributes of known or expected224932-7414-0331UP335240.W0.01objects as listed in the master data 352 of FIG. 3. When the compared attributes match, the robotic system 400 can detect the object depicted in the image data by determining the type or the identifier for the object and the estimated real -world location of the objects (e.g., the peripheral edges of the object). The robotic system 400 can perform additional operations and analyses, such as to confirm detection or related data and / or when portions of the image fail to match the attributes listed in the master data 352, indicating that the corresponding portions may be depicting unrecognized objects.

[0116] FIG. 5 illustrates a block diagram of a method 500 of operating the robotic system 104 of FIG. 1 and / or the robotic system 200 of FIG. 2 in accordance with some embodiments of the present technology. The method can be for generating one or more detection hypothesis for objects depicted in imagine information (also referred to as image data) captured by sensors (e.g. the imaging sensors 424 / 425 of FIG. 4). The robotic system 104 can use the detection hypothesis to generate instructions for a robot (e.g. the robotic system 400 of FIG. 4) to grasp and remove one or more objects corresponding to the one or more detection hypothesis from a stack of objects. The method can include image preprocessing in block 502 to generate processed image data, feature extraction in block 504 from the processed image data, hypothesis generation in block 506 to generate the detection hypothesis, a motion planning in block 508 to generate a motion plan for the robot to interact with a target object (e.g. the object corresponding to the detection hypothesis), and optionally, detection registration in block 510 to generating templates for subsequent detections. They method 500 can be implemented by the robotic system 104 and / or the robotic system 200.

[0117] In some embodiments, image preprocessing of block 502 can include subprocesses for combining images from two different imaging systems (e.g. from two-dimensional and three-dimensional cameras and / or multiple cameras at different physical positions with overlapping fields of view), generating surface normal, logically removing (e.g. cropping) portions of the image, imaging data adjustment (e.g. adjustment or compensation from noise or inaccuracies in imaging data such as due to reflective surfaces), point cloud segmentation, or a combination thereof. Each of these sub- processes will be discussed in detail below.

[0118] In some embodiments, feature extraction of block 504 can include subprocesses of line detection, corner detection, center heat map generation corresponding to object surface center estimation, center heat map validation, or a combination thereof. Each of these sub-processes will be discussed in detail below.234932-7414-0331UP335240.W0.01

[0119] In some embodiments, the hypothesis generation of block 506 can include subprocesses of initial detection hypothesis generation, hypothesis refinement, hypothesis validation, or a combination thereof based on, for example, candidate edges, corners, surface center estimation, template matching, or a combination thereof. In some embodiments, the method 500 can include reprocessing the image to remove validated detection hypothesis (also referred to as verified detection hypothesis) from the image information. Each of these sub-processes will be discussed in detail below.

[0120] In some embodiments, the motion planning of block 508 can include generating a motion plan for operation of a robot, such as the robotic system 400 of FIG. 4. In some embodiments, generating the motion plan can include determining a target object to grasp based on the corresponding validated detection results, generating instructions for positioning of the gripper 406 to grasp and remove the target object. In some embodiments, block 508 can include a sub-process of sending the motion plan to a robotic system, such as the robotic system 400, for execution of the motion plan and receiving information from the robotic system to determine whether the target object was successfully or unsuccessfully grasped and removed by the robotic system. For example, the removed target object can be added to a pick history that includes the successful or unsuccessful attempts to remove the target object. The method 500 can further include using pick history in a subsequent image preprocessing sequence, as noted by the flow line from block 508 to 502.

[0121] In some embodiments, the template registration of block 510 can include storing of validated detection results based on successful removal of objects. Template registration will be discussed in detail below. In some embodiments, the templates registered in block 510 can be used by the robotic system 104 in subsequent hypothesis generation, as noted by the flow line.

[0122] FIG. 6A illustrates example 2D image data 602 after preprocessing operations that remove non-object regions in accordance with at least one embodiment of the present technology. The robotic system 104 of FIG. 1 can analyze and process an image data 120 obtained via the vision system 108. For the illustrated example, the depicted image corresponds to an image from a visual camera taken along a lateral orientation to capture objects stacked within a shipping container. While described in greater detail below with respect to a corresponding 3D image for the same scene, the image data 120 can depict side views of objects that are arranged form different layers or walls, where each layer or244932-7414-0331A1P335240.W0.01wall includes objects that are located at distances within threshold ranges of each other from the image sensor.

[0123] In particular, FIG. 6A illustrates an example of the image data 602 following one or more preprocessing operations. The preprocessing operations refer to operations performed to transform raw sensor data into a form that is more suitable for analysis by downstream processes. Preprocessing may include alignment and registration of the image data obtained from multiple vision sensors, such as left and right imaging systems. In such instances, the system (e.g., the robotic system 104 of FIG. 1) spatially registers 2D and 3D image data into a common coordinate frame based on calibration parameters, permitting corresponding portions of the scene to be analyzed together.

[0124] The preprocessing may include stitching or combining the 2D image data captured by the left sensor and the right sensor to form a unified 2D representation of the scene. For example, overlapping portions of the left image and the right image may be aligned and stitched together, and / or selected regions from each image may be retained based on image quality or visibility of object surfaces.

[0125] In some embodiments, the preprocessing includes logical removal or masking of the image data that are not relevant to object analysis. For example, image regions corresponding to known structures such as container walls, floors, ceilings, and / or portions of the robotic system 104 (e.g., the end effector 110) may be identified and excluded from further processing. The robotic system 104 may determine such regions based on depth thresholds, geometric constraints, reference models, and / or kinematic information.

[0126] FIG. 6B illustrates another example image data 604 with one or more preprocessing operations applied with emphasis on 3D spatial information derived from the captured image data. For illustrative purposes, the image data 604 shown in FIG. 6B corresponds to a 3D depiction of the scene captured by image data 602 of FIG. 6A.

[0127] The image data 604 illustrates different depth values or relative distances of object surfaces from the vision system 108. Portions of the image data 604 corresponding to objects positioned closer to the vision system 108 may be represented with one shade, and portions corresponding to objects positioned farther away may be represented with a different shade. For example, higher intensity may indicate surfaces that are closer to the sensors, and lower intensity may indicate surfaces located further away. The stacked arrangement of objects may be visible as a set of planar surface regions at varying depths, reflecting a 3D spatial layout rather than a flat 2D projection. For the illustrated example, the objects are arranged in three layers or walls, with the nearest wall having the lowest254932-7414-0331A1P335240.W0.01stack height and the farthest wall having the highest stack height, such as according to a stacking rule and / or an object removal or devanning rule.

[0128] In some instances, the image data 604 may be combined from multiple vision systems, such as a left depth sensor and a right depth sensor positioned to observe the same scene from different lateral perspectives. The depth measurements from the multiple sensors are registered into a common coordinate frame, permitting corresponding 3D points to be merged into a single composite point-cloud or depth representation. The combination improves spatial coverage across the scene and reduces gaps that might otherwise occur due to occlusion from a single viewpoint.

[0129] The image data 604 includes portions of the image that have been selectively removed during preprocessing. Such removed or blank regions (e.g., lower middle portion for the example in FIG. 6B) represent portions of the 3D data that may not be present due to occlusion, shadows, and / or insufficient sensor return. Additionally or alternatively, the depth data may be removed in response the analysis determining certain points correspond to non-forward-facing surfaces, such as side faces of objects, container walls, and / or background structures that are not relevant to object interaction. Depth values that fall outside expected ranges or exhibit characteristics associated with noise, glare, or unreliable reflections may also be excluded.

[0130] In some embodiments, the preprocessing may include logical removal of depth data corresponding to regions known as a priori to be outside the area of interest, such as portions of the container interior, structural supports and / or locations in which a robotic end effector is expected to appear. The removal regions may be determined based on sensor calibration, geometric constraints, and / or known motion states of the robotic system. By removing such regions, the remaining 3D image data may be more focused on object surfaces that are suitable for further analysis. For illustrative purposes, the image data 602 and 604 are referenced as starting points for various processes described below.

[0131] FIG. 7A illustrates example features extracted 702 from the preprocessed image data in accordance with at least one embodiment of the present technology. FIG. 7A illustrates a stack of objects, such as boxes, arranged in multiple rows and columns within a bounded environment.

[0132] In some embodiments, the robotic system (e.g., the robotic system 104) extracts and classifies comer features that represent physically meaningful points on object surfaces. The comers are defined not solely in 2D image space, but with reference to 3D spatial structure derived from depth data. The features illustrated in the image data 702264932-7414-0331UP335240.W0.01correspond to candidate object comers that have been evaluated in relation to surrounding geometry to determine whether the comers are open, exposed, or obstructed in the vertical dimension.

[0133] In the embodiments of the present technology, a comer may refer to a location on an object surface at which two boundary segments or surface edges intersect, such as where a top surface meets a vertical face of a box. Initial corner candidates may be identified using preprocessed 2D and / or 3D images. For example, the initial comer candidates may be determined based on the intersections of line segments or edges (e.g., results of edge detection processes, such as results of applying a Sobel filter to the images 602 and / or 604), intensity gradients, and / or segmented object boundaries in the preprocessed 2D image data. The 2D-derived corner locations and the 3D-derived corner locations may be compared based on corresponding positions, permitting each comer candidate to be associated with depth and height measurements.

[0134] Further, the robotic system 102 (via, e.g., the processor 106 of FIG. 1) can determine 3D characterization of a detected / verified corner by analyzing the spatial position of that comer relative to other detected comers and / or surrounding surface measurements. For example, the corners can be characterized as (1) open corners that do not have another object / surface within a threshold distance or (2) internal comers that correspond to corners that are near or abutting adjacent object(s). In some embodiments, the system may evaluate corner candidates by grouping the corners according to similar depth values measured along a viewing direction of the vision system. Comers that share approximately the same depth may be considered to lie on surfaces that are co-planar or positioned at similar distances from the imaging sensors, such as for corners on a common surface / object and for corners for objects within a common layer.

[0135] As an example, the image data 702 illustrates an object 704 positioned in a top row of the stack, in which the object has four comer features designated as a first comer 708a, a second comer 708b, a third corner 708c, and a fourth comer 708d (collectively referred to as the corners 708). The corners 708 correspond to intersections of object boundary edges identified during earlier feature extraction steps using the preprocessed 2D image data, such as intersections of detected edges or boundaries of segmented regions. Each of the corners 708 is mapped into the registered 3D image data such that each comer is associated with spatial coordinates, including depth and height relative to a gravity-referenced axis.274932-7414-0331UP335240.W0.01

[0136] In some embodiments, the system first evaluates the depth values associated with the comers 708. The depth values may represent distances from a reference point of the visions system or from a known container boundary. Corners having depth values within a predefined tolerance may be grouped together, indicating that the comers lie approximately the same object surface and at a similar distance from the sensors. For example, the four comers 708 may be determined to have similar depth values, consistent with the comers belonging to the same object surface.

[0137] In some embodiments, the system evaluates the vertical positions of the corners relative to the direction of gravity, such as for images data captured by laterally-facing sensors. Height information may be derived from the 3D image data following calibration and registration. For example, the system may determine that the first corner 708a and the second corner 708b have the greater height among the four comers 708, and the third comer 708c and the fourth comer 708d are located lower in the vertical direction. The system may further determine whether the height difference between the first corner 708a and the second corner 708b is within a defined tolerance, permitting both comers to be considered as candidate upper corners.

[0138] The system assesses whether each candidate upper corner is exposed or obstructed based on analysis of presence of other object surfaces above the corner location. For instance, the system may search the 3D image data for depth points located above the first corner along the gravity axis. In instances surface points, planar regions, and / or other comers are not detected within a threshold distance above the first corner 708a, the first comer 708a may be classified as an exposed or open 3D comer. The second comer 708b may be analyzed in a similar manner to be determined as an open comer.

[0139] Contrastingly, the third corner 708c and the fourth comer 708d may be examined based on whether the surfaces are present above the comers. In some instances, the 3D image data may indicate that portions of another object extend above the third comer 708c and the fourth corner 708d, either due to an adjacent object overlapping in the vertical direction or as the object is partially covered by another layer of the stack. Even in instances the third corner 708c and the fourth corner 708d appear visually prominent in the 2D image, the depth-based analysis may reveal obstruction. In such instances, the third comer 708c and the fourth comer 708d are classified as obstructed comers.

[0140] In some embodiments, additional checks may be applied to reduce misclassification. For example, surface normal information derived from the preprocessed 3D image data may be used to confirm whether detected surfaces above the third corner284932-7414-0331UP335240.W0.01708c and the fourth comer 708d correspond to separate objects rather than slanted faces or noise artifacts. Segmentation results from earlier processing steps may also be used to determine whether the overlaying surfaces belong to distinct objects. Corners associated with unreliable or missing depth measurements may be temporarily excluded or assigned reduced confidence.

[0141] With 3D image data, the robotic system 104 may not identify all edges and comers of every object. For example, the objects may be occluded by another object (e.g., by an object positioned closer to the image sensor) and / or the objects may be packed tightly. As a result, the edge detection results may be unable to detect the object edges of every object depicted in the captured scene. As described in further detail below 2D image data may present other challenges with respect to edge detection. As a result, the robotic system 104 can be configured to account for incomplete edge and corner detections in further processing the image data. For example, the robotic system 104 can use the detected edges and corners to compute bounding regions that are encircled or defined by the detected edges / comers. Such bounding regions can serve as initial detection hypotheses, with each detection hypothesis corresponding to an estimate that the corresponding portion of the image is representative of a single object. Stated differently, an object detection can be an estimate that one object or one surface thereof is located at coordinates matching a corresponding portion of the image data. A hypothesis of such obj ect detection can serve as a starting point or an initial detection result that is to be further processed, adjusted when needed, and finalized.

[0142] FIG. 7B is image data 710 illustrating bounding regions corresponding to detection hypotheses generated from the extracted corner features described with respect to FIG. 7A. Each bounding region represents a preliminary hypothesis that a corresponding portion of the image data 710 depicts one portion / surface of one object, such as a face of a box, that may be individually identified and interacted with by a robotic system.

[0143] In some embodiments, comers classified as exposed (e.g., the first corner 708a and the second comer 708b) represent locations on object surfaces that are unobstructed in the vertical direction and / or lateral direction and thus provide strong geometric evidence of object accessibility. The corners classified as internal or obstructed (e.g., the third corner 708c and the fourth corner 708d) indicate surfaces that may be partially or fully covered by other objects.

[0144] The detection hypotheses may be derived based on the classified corner information. For example, bounding regions may be assigned around clusters of comers294932-7414-0331UP335240.W0.01that are likely to belong to a single object. The bounding regions may be illustrated as rectangular outlines surrounding portions of the image data 710 corresponding to individual boxes or box faces, such as the bounding box 712. The robotic system 104 can compute the bounding regions based on identifying edges that at least partially extend linearly between a pair of adjacent corners. The robotic system 104 can compute each bounding region based on connecting a set (e.g., four) comers with qualifying linear edges (e.g., In some embodiments, the robotic system 104 may be configured to focus on or limit to detection hypotheses that include at least one exposed or open corner to reduce instances of false detections. The presence of one or more exposed comer may indicate that the corresponding bounding region is not vertically occluded and therefore may be interacted with independently of neighboring objects.

[0145] The robotic system 104 can calculate a confidence score for the bounding box 712. For example, the robotic system 104 can calculate the confidence score to represent a likelihood that the bounding box 712 matches one surface of one object. In some embodiments, the robotic system 104 can calculate the confidence score according to comer characterizations and associated quantities / locations, lengths and / or orientations of the edges, and / or the like. Subsequently, the robotic system 104 can use the confidence score to determine (e.g., based on a comparison to a predetermined threshold confidence score) whether the bounding box 712 is sufficient for subsequent processing.

[0146] In the example of the image data 710 shown in FIG. 7B, detection regions associated with such sufficient-confidence hypotheses are visually illustrated and treated as verified or conditionally verified detections. Such regions may be determined to correspond to objects that are likely accessible from the current viewpoint. In some embodiments, in response to computing a detection hypothesis having the sufficient confidence level, the robotic system 104 may logically remove the corresponding portion of the image data. Logical removal may include cropping, masking, or otherwise excluding or temporarily altering the pixel values and / or depth points associated with the region from subsequent processing stages. In the present technology, a hypothesis confidence level represents a quantitative or qualitative measure of how likely a detection hypothesis corresponds to a valid, distinct, and accessible object within the scene. The confidence level is derived from multiple features extracted from preprocessed 2D and 3D image data and is used to determine whether to accept, refine, defer, or logically remove a detection hypothesis from further consideration.304932-7414-0331UP335240.W0.01

[0147] In some embodiments, the system assigns a confidence level to a detection hypothesis based at least in part on the properties of the features supporting that hypothesis. For example, hypotheses associated with one or more comers classified as exposed and upper may be assign a higher confidence level than hypotheses associated solely with obstructed corners. The presence of exposed three-dimensional comers indicates that the corresponding object surface is unobstructed in the vertical direction and is therefore more likely to represent an independently accessible object.

[0148] In addition to comer exposure, the confidence level may incorporate geometric consistency metrics. For instance, the system may evaluate whether the detected corners of a hypothesis are spatially consistent with in terms of depth similarity, relative orientation, and expected object geometry. A hypothesis with comers at similar depths and approximately orthogonal edge relationships may receive a higher confidence level than a hypothesis with features that are inconsistent or fragmented. Surface normal information derived from the 3D preprocessing may further contribute to the assessment by indicating whether supporting surfaces form a planar region consistent with an object surface.

[0149] In some embodiments, the system further evaluates the hypothesis confidence level based on occlusion analysis. A detection hypothesis that does not overlap other hypothesis in the vertical direction, or that is determined not to be positioned beneath another object based on 3D spatial reasoning, may be considered non-occluded and therefore assigned a higher confidence. Conversely, hypotheses that share vertical overlaps with other hypotheses or are determined to support overlying surfaces may be assigned lower confidence values or flagged as obstructed.

[0150] The confidence level may incorporate measures of data reliability. For example, in instances the image region supporting a hypothesis corresponds to areas of reliable 3D depth measurements, such as regions not affected by glare, shadowing, or sensor noise, the system may increase the confidence level. Hypotheses relying on sparse or unreliable depth data may be assigned lower confidence.

[0151] In some implementations, the confidence level may be represented as a numerical score, such as a value normalized within a defined range, or as a discrete classification, such as “high confidence,” “medium confidence,” or “low confidence.” One or more thresholds may be applied to the confidence level to guide subsequent actions. For example, in instances the confidence level of a detection hypothesis exceeds a first threshold, the hypothesis may be considered sufficiently reliable to be acted upon, such as by designating the corresponding region for removal or interaction. In instances the314932-7414-0331UP335240.W0.01confidence level does not meet such a threshold, the hypothesis may be retained for further refinement, deferred until additional objects are removed, or excluded from immediate consideration.

[0152] Referring back to the logical removal of the bounding regions, the robotic system 104 can iteratively perform or repeat the detection process following the logical removal. Since these bounding regions have sufficiently high likelihood of belonging to one surface of one object, the robotic system 104 can reconsider the image data after excluding such regions or treating such regions as empty space (e.g., as if the corresponding objects have been transferred out from the scene). For example, the robotic system 104 can reclassify corners or edges as now being open or exposed after having removed bounded boxes adjacent thereto. As a result, the robotic system 104 can increase the detection results using the same image, such as by simulating removal of detected objects without performing the actual transfer and reimaging.

[0153] FIG. 8A illustrates image data 802 for identification and removal of image regions corresponding to a robotic end effector in accordance with at least one embodiment of the present technology.

[0154] The image data 802 shows a stack of objects in a forward-facing view. The image data 802 may be obtained during active robotic manipulation and / or at a time period in which the robotic end effector or gripper is positioned within the field of view of the vision systems.

[0155] In some embodiments, the robotic system may identify portions of the image data 802 that correspond to the end effector, such as using a predetermined filter or template, as distinct from object surfaces analyzed for feature extraction and detection hypothesis generation. In some embodiments, identification of the end effector within the image data 802 may be performed based on a combination of geometric modeling, depthbased analysis, temporal reasoning, and / or kinematic awareness of the robotic system. The robotic system actively distinguishes image regions corresponding to the robotic tool from regions corresponding to objects of interest.

[0156] In some embodiments, the system maintains one or more reference models of the end effector, which may include 2D appearance characteristics, 3D geometry, or both. Such reference models may be obtained during a calibration phase and may represent the expected shape, size, and / or orientation of visible portions of the end effector from the perspective of the vision sensors. In instances the 2D image data is obtained, the robotic system 104 may compare localized image regions against the reference models to identify324932-7414-0331UP335240.W0.01candidate matches based on shape similarity, edge patterns, and / or texture characteristics. Regions exhibiting a certain level of similarity to the reference model may be classified as belonging to the end effector.

[0157] Additionally or alternatively, the system may utilize 3D image data to identify the end effector. For example, depth information derived from one or more 3D sensors may be analyzed to determine portions of the image data corresponding to depth ranges that are inconsistent with the expected depth of object surfaces. As the end effector typically operates closer to the vision sensors than the object stack, image regions having depth measurements within a predefined near-field range may be flagged as potential endeffector regions. For example, a first region 804 may correspond to region in which the end effector is operating. Contrastingly, depth measurements associated with surfaces lying beyond the predefined range may be considered a part of the object scene. For example, a second region 806, including parts of the image data 802 not included in the first region 804, may be considered a part of the object scene.

[0158] In some embodiments, the system may logically remove, mask and / or ignore the identified end-effector region, such that transient robotic components do not interfere with feature extraction processes such as segmentation, line detection, corner extraction, and / or hypothesis confidence evaluation. In some embodiments, the system may associate temporal information, such as a timestamp, with the image data 802. The temporal information may permit the system to reason about the time the image data 802 was captured relative to robot motion and to later combine the image data 802 from different times.

[0159] FIG. 8B illustrates image data 808 generated based on temporal combination of three-dimensional image data obtained at least different times to compensate for end-effector occlusion in accordance with at least one embodiment of the present technology. In some embodiments, 3D image data at different times may be obtained as part of continuous operation of a robotic manipulation system. Each instance of image data may be associated with a timestamp indicating the time the data was captured relative to robot motion.

[0160] The image data 808 includes two sets of 3D image data. Data points in a first region 810 may correspond to 3D image data obtained at a first time period. For example, the first region may be associated with a time at which the end effector was present such as the image data 802 of FIG. 8A. In these and other embodiments, the first region may include a removed region corresponding to the region in which the end effector was334932-7414-0331UP335240.W0.01present. Data points in a second region 812 may correspond to 3D image data obtained at a second time period different from the first time period. For example, the second time period may be a point in time earlier or later than the first time period in which the end effector does not occlude the same portion of the scene.

[0161] In these and other embodiments, the image data in the second region 812 may be used to fill in the removed regions of the first region 810. For example, the 3D image data obtained at the earlier or later timestamp, in which the end effector was not obstructing the removed region, may be used to populate the corresponding spatial region in the later image data. The selection of replacement data may be based on temporal proximity, spatial alignment, and / or similarity of surrounding geometry to improve accuracy of the reconstructed region with respect to the current state of the scene.

[0162] FIG. 8C illustrates a reconstructed three-dimensional view of the scene formed by combining depth data captured at different times to compensate for temporary occlusion by the robotic end effector in accordance with at least one embodiment of the present technology. A first region 814 corresponds to 3D image data retained from a later capture and represents object surfaces that were visible and unobstructed at that time. A second region 816 represents reconstructed depth data inserted to bridge between temporally distinct measurements, often using earlier image data selected based on spatial alignment and consistency with neighboring surfaces. A third region 818 corresponds to 3D image data obtained at an earlier time and used to fill portions of the scene that were removed from the later image due to end-effector presence. The regions together may form a single, spatially consistent 3D representation restoring object surface information lost to transient occlusion, permitting subsequent processing to operate on a more complete depiction of the scene.

[0163] FIG. 8D illustrates a reconstruction process applied to 2D image data in accordance with at least one embodiment of the present technology. In some embodiments, portions of 2D image data that were removed or ignored due to end-effector presence are filled using corresponding 2D image data obtained at a different time. The system aligns the earlier and later 2D images using known sensor calibration and scene geometry and then inserts the earlier image content into the removed regions of the later image. The resulting 2D image provides a visually complete view of the object stack while excluding transient features of the robotic end effector.

[0164] FIG. 9A illustrates preprocessed image data 902 (similar to the image data 702 of FIG. 7A). in accordance with at least one embodiment of the present technology. The344932-7414-0331UP335240.W0.01image data 902 may include and / or exclude different features from the preprocessing. For example, the image data 902 may include comer estimations for the objects in the scene. As another example, the image data 902 may exclude a portion of the image data 902 corresponding to the end effector. In some embodiments, the robotic system 104 may analyze different portions of the image data 902 to determine and / or verify that the segmented data corresponds to one or more objects within the scene. Stated differently, the preprocessed image data 402 can correspond to a result of feature detection for corners and edges, serving as a starting point for computing bounding regions. The verification process of the segmented image data and the corresponding computation of the bounding regions may be described with respect to FIGS. 9B-9K.

[0165] In some embodiments, the robotic system 104 can divide the image data 902 into sub portions for further analysis. The robotic system 104 may divide the image data 902 based on detected corners, such as portions between an adjacent pairing of comers that have coordinates (e.g., lateral or vertical positions / coordinates) within a threshold range of each other and / or having one or more detected edges linearly extending between the pairing of corners.

[0166] For example, FIG. 9B illustrates segmented image data 904 corresponding to a portion of the imaged data 902. The segmented image data 904 can correspond to the portion of the image data 902 referenced from a corner 906 identified from the preprocessing. The segmented image data 904 can represent an area between a set of detected corners, and may further include a set of edges, such as a first line 908a and a second line 908b, extending from the referenced corner 906. The first line 908a and the second line 908b represent adjacent edges of the segmented image data 904.

[0167] FIG. 9C illustrates an initial detection hypothesis 910 defined based on the comer 906, the first line 908a, and the second line 908b in accordance with at least one embodiment of the present technology. In some embodiments, the edges (the first line 908a and the second line 908b) may be extended from the corner 906 to define a bounded region that is likely to correspond to a portion of a single object surface. For example, the robotic system 104 can define the bounded region by generating (1) one complementary edge that extend from a distal end of the first line 908a and extends parallel to the second line 908b (e.g., laterally) for a length matching that of the second line 908b and (2) another complementary edge that extends from a distal end of the second line 908b and extends parallel to the first line 908a (e.g., downward) for a length matching that of the first line 908a. The robotic system 104 may further extend the lines 908a and / or 908b according to354932-7414-0331UP335240.W0.01minimum required lengths (e.g., corresponding to smallest dimension of expected objects). The initial detection hypothesis 910 represents a provisional object region based on local geometric evidence and does not necessarily describe the full extent of the object.

[0168] FIG. 9D illustrates results of line detection performed on 3D image data corresponding to the segmented image region in accordance with at least one embodiment of the present technology. The system (e.g., the robotic system 104) analyzes the 3D image data to detect linear features associated with object surfaces, such as edges of boxes, wrinkles in packaging labels / tapes, and / or other contour features on the depicted surface(s). For example, the image 902 or a corresponding 3D image processed to detect the comers, can be obtained at a first granularity / sensitivity, and the line detection shown in FIG. 9D can be performed at a second or finer granularity / sensitivity.

[0169] Using the finer granularity, the detected lines can represent locations where there is sufficient geometric consistency, for example, along relatively smooth or planar regions of an object surface. Accordingly, the detected lines can include edges corresponding to the other physical edges complementary to those represented by lines 908a and 908b of FIG. 9B and defining the corresponding physical surface of the depicted object. However, due to the finer granularity, the robotic system 104 can detect other lines influenced by surface characteristics derived from the 3D image data, such as changes in surface orientation or depth gradients. In some embodiments, the line detection may be incomplete or fragmented in certain areas, for example due to surface irregularities, wrinkles, printed / raised text, or regions affected by reflective materials that degrade depth measurement reliability.

[0170] In continuing to detect and locate the physical edges of depicted objects, FIG.9E illustrates a subsequent processing step in which the system expands the detected lines to generate expanded lines 914 in accordance with at least one embodiment of the present technology. In some embodiments, the robotic system 104 can generate the expanded lines 914 by increasing / extending the lengths and / or widths of the detected line segments to form continuous regions representing candidate object edges. The expansion may be performed by extending line segments along their detected orientation or by thickening the segments to cover adjacent pixels or depth points that share similar geometric properties. The robotic system 104 can generate the expanded lines 914 by lengthening / thickening the detected lines of FIG. 9D by a predetermined amount (e.g., fixed distance, a percentage of detected edge, etc.) and / or connecting or interpolating between detected segments that have separation distances less than a predetermined threshold. In some embodiments, the364932-7414-0331UP335240.W0.01robotic system 104 can expand along predetermined axes, such as horizontal and vertical axes for processing image data generated by laterally facing sensors. The expanded lines 914 provide a more robust representation of potential object boundaries, permitting the system to better evaluate surface continuity and to support later stages of detection hypothesis refinement.

[0171] FIG. 9F illustrates determination of continuous surface regions 916 derived from the expanded lines 914 shown in FIG. 9E in accordance with at least one embodiment of the present technology. The robotic system 104 can analyze the lengthened and / or widened line segments of FIG. 9E to identify areas in which adjacent or overlapping expanded lines form / encircle / define continuous regions. Such regions represent candidate surface areas that are likely to belong to a single object surface rather than isolated or fragmented features. In some embodiments, the system may group expanded lines based on spatial proximity, orientation similarity, and connectivity / overlap, and define each resulting continuous area as a distinct region.

[0172] FIG. 9G illustrates refinement of the initial detection hypothesis based on overlap between the initial hypothesis region and the continuous surface regions 916 identified in FIG. 9F, thus generating an intermediate detection hypothesis 918 in accordance with at least one embodiment of the present technology. The robotic system 104 can evaluate which of the continuous surface regions 916 formed from lengthened and widened line features correspond to, or are associated with, the initial detection hypothesis. In some embodiments, a continuous surface region 916 is determined to correspond to the initial detection hypothesis 910 in instances the continuous area at least partially overlaps, connects with, or intersects the region defined by the initial detection hypothesis. Additionally or alternatively, the continuous surface region 916 can be determined to correspond to the initial detection hypothesis 910 when the corresponding depth value are within a threshold range from the initial detection hypothesis 910 and / or have surface normal parallel to that of the initial detection hypothesis 910 (e.g., as indicative of a higher likelihood of portions belonging to the same physical surface).

[0173] The continuous areas that overlap the initial detection hypothesis can be retained and visually distinguished, and continuous areas that do not overlap the initial hypothesis may be excluded from further consideration. Likewise, the continuous areas having physical traits of likely belonging to the same continuous physical surface as the initial detection hypothesis may be retained and others may be excluded. For the illustrated example, the robotic system can ignore the continuous regions overlapping the object374932-7414-0331A1P335240.W0.01located on the right half of FIG. 9G since they don’t overlap with the initial detection hypothesis 910 and / or have depth values / surface normal that deviate from those of the initial detection hypothesis 910.

[0174] By selectively maintaining the continuous regions that are spatially / logically connected to the initial detection hypothesis, the robotic system 104 may increase the likelihood that subsequent processing focuses on image data corresponding to the same object surface rather than on unrelated texture, printed markings, or features belonging to adjacent objects. In some embodiments, retaining only overlapping continuous areas may reduce the impact of texture artifacts, text, labels, or surface irregularities that may otherwise generate spurious lines and continuous regions. The filtering step may improve the robustness of the initial detection hypothesis by narrowing the supporting feature set to areas that are both geometrically continuous and spatially consistent with the hypothesis.

[0175] FIG. 9H illustrates expansion of retained continuous regions to approximate an object surface in accordance with at least one embodiment of the present technology. The boundaries of each retained continuous area may be extended to form an expanded hypothesis region 920 that better approximates the full extent of the object surface. In some embodiments, the robotic system 104 can generate the expanded region 920 by extending edges of the continuous areas in one or more directions, such as along detected edge orientations or until a termination condition is met, including a change in depth, segmentation boundary, or loss of surface continuity. Effectively, the robotic system 104 can generate the expanded hypothesis region 920 by including the expanded edges of FIG.9E that boarder the separated portions of the intermediate detection hypothesis 918 of FIG.9G. Stated differently, the robotic system 104 can generate the expanded hypothesis region 920 by including the expanded edges that contact and / or are disposed between the portions of the intermediate detection hypothesis9. Further, in generating the expanded hypothesis region 920, the robotic system 104 can superimpose or overlap the initial detection hypothesis 910 defined in FIG. 9C with the expanded continuous areas 918 of FIG. 9G

[0176] FIG. 91 illustrates access to segmented image information corresponding to the expanded hypothesis region 920 in accordance with at least one embodiment of the present technology. Portions of the segmentation result that overlap with the expanded regions are highlighted in FIG. 41, indicating areas in which the segmentation supports the hypothesis that the expanded region corresponds to a single object surface. By referencing segmentation results, the robotic system 104 can incorporate an additional consistency384932-7414-0331UP335240.W0.01check beyond line-based and region-based features. Overlap between the expanded regions and segmented areas increases confidence that the region represents a coherent portion of an object rather than background texture or features of an adjacent object.

[0177] The robotic system 104 can remove the expanded hypothesis region 920 from the segmented image data 904 and effectively repeat the processes described above. For example, the robotic system 104 can mask or otherwise treat the expanded hypothesis region 920 as having default depth values (e.g., corresponding to empty space) and process the resulting modified image to re-perform edge / comer detection, segmentation, initial hypothesis computation, edge expansion, continuous surface region determination, detection hypothesis refinement, and detection hypothesis expansion, as described above. As a result, the robotic system 104 can generate a set of one or more detection hypotheses 922 for the segmented image data 904, where the set 922 includes the expanded hypothesis region 920 and the subsequently computed hypothesis region for the remaining portions.

[0178] FIG. 9J illustrates determination of a modified detection hypothesis 924 based on evaluating the overlap between the expanded regions and the segmentation information in accordance with at least one embodiment of the present technology. In some embodiments, the portions of the expanded area substantially corresponding to the segmented image data are determined, and the robotic system 104 can retain or focus on the portions that satisfy spatial and geometric consistency criteria. Portions of the expanded regions that fall outside of the segmented areas, or that otherwise fail consistency checks, may be excluded. The retained region is designated as a modified detection hypothesis 924, shown in FIG. 9J as a refined, bounded region. Effectively, the robotic system 104 can further expand the expanded hypothesis region 920 of FIG. 9H according to bottom and / or top edge conditions. Details regarding outer edge (e.g., the bottom / top / side edge) conditions and the related expansion are described below. This modified detection hypothesis 924 provides a more accurate representation of the object surface than the initial hypothesis and is suitable for subsequent validation, confidence evaluation, or downstream processing stages.

[0179] FIG. 9K illustrates determination of representative comers for the expanded and refined detection hypothesis resulting from the processes described with respect to FIGS. 4H-4J in accordance with at least one embodiment of the present technology. The modified detection hypothesis may be analyzed to identify corner locations that best represent the geometry of the corresponding object surface. For example, a first corner 926a, a second comer 926b, a third corner 926c, and a fourth comer 926d (collectively394932-7414-0331UP335240.W0.01referred to as the comers 926) may be identified. The comers 926 may be determined based on intersections of refined boundary edges, changes in direction of boundary segments, or characteristic transitions in surface geometry derived from the underlying image data.

[0180] In some embodiments, the corners 926 of the expanded detection hypothesis may be determined based on evaluation of the outermost extents of the refined region (e.g., starting from the modified detection hypothesis 924 and moving outward) in one or more directions. The robotic system 104 may determine where adjacent boundary segments intersect at approximately orthogonal angles and designate those locations as representative 2D comers associated with the detection hypothesis. In some embodiments, 3D image data may be used to confirm that the identified corner locations correspond to physical corners of an object surface rather than artifacts caused by noise, texture, or segmentation errors.

[0181] FIG. 10A illustrates determination of object edges and estimation of object width based on line analysis within segmented image data in accordance with at least one embodiment of the present technology. In some embodiments, one or more candidate edges corresponding to an object may be identified based on line features between adjacent comers identified with respect to FIG. 9K. For example, the robotic system 104 may determine lines extending between estimated comer locations. Moreover, the robotic system 104 can analyze the 2D image to detect the edges.

[0182] In some embodiments, the system (e.g., the robotic system 104 of FIG. 1) may access the segmented image data associated with the detection hypothesis and apply a line detection or edge detection algorithm to identify line segments corresponding to object boundaries. The detected line segments are evaluated based on geometric characteristics, including orientation and length. For instance, the system may determine the angle of each line segment and identify segments oriented within a predefined angular range corresponding to a particular direction, such as vertical. For the illustrated example of sideview images, the robotic system 104 can be configured to identify segments that extend horizontally and / or vertically. The system may also determine whether individual line segments, or portions of the line segments, exceed a minimum length threshold. Line segments that do not satisfy length or orientation criteria may be logically removed from further consideration.

[0183] The robotic system 104 can evaluate the remaining line segments to determine whether a sufficient total length of aligned line segments is present in a given direction,404932-7414-0331UP335240.W0.01such as the vertical direction. In instances the total length exceeds a threshold, the system may extend one or more line segments in the direction to compensate for gaps caused by incomplete detection, sensor noise, and / or surface irregularities. In these and other embodiments, the minimum length threshold, total length threshold, and other thresholds may be defined based on expected shape and dimensions of the objects. For example, in instances the robotic system is expected to identify and engage boxes, the thresholds may be defined based on known dimensions and shapes of the boxes.

[0184] FIG. 10B illustrates further refinement of candidate object edges identified based at least on line-based analysis described with respect to FIG. 10A in accordance with at least one embodiment of the present technology. In some embodiments, in response to identifying multiple candidate edges and possible object widths, the system evaluates the relative confidence of each edge candidate. The system may designate a first edge 1006 of the detection hypothesis as a high-confidence edge (e.g., corresponding to open 3D edges) and a second edge 1008 of the detection hypothesis as a low-confidence edge. The high-confidence edge may correspond to an edge location supported by strong and consistent geometric evidence, such as continuous line segments aligned with expected object boundaries. Contrastingly, the low-confidence edge may correspond to an edge location in which detection line segments are fragmented, incomplete, or affected by noise, occlusion, or surface artifacts.

[0185] FIG. 10C illustrates determination of an updated or refined object edge within the low-confidence edge region (e.g., the second edge 1008) identified in FIG. 10B using depth-based analysis in accordance with at least one embodiment of the present technology. FIG. 10C shows a graph 1010 representative of the depth measures at positions along a lateral line left and right of the second edge 1008. The system analyzes 3D depth measurements within the low-confidence region to identify a depth profile representative of the object boundary. In some embodiments, the robotic system 104 can obtain multiple depth measurements for the low-confidence edge region. For the multiple depth measurements, the robotic system 104 can aggregate depth measurements across columns or rows of image data to reduce noise and misalignment effects. For example, the y-axis of the graph 1010 may represent a mean or aggregate depth value, and the x-axis may represent spatial position within the image data.

[0186] The system can identify a depth maximum within the low-confidence edge region, which may correspond to the deepest measured point indicative of a gap between adjacent objects or free space beyond the object surface. In some embodiments, the system414932-7414-0331UP335240.W0.01may compute an offset from the depth maximum based on the slope or curvature of the depth profile to assign the refined edge. For instance, the system may identify a steep transition region in the depth data and determine the refined edge location as a point offset from the maximum by a defined percentage of the run. For example, the refined edge location may be selected at approximately a midpoint along the steepest portion of the depth transition. Given the context of ultimately deriving a heatmap representative of a surface of an object and / or a center portion therein and that the analyzed region is associated with lower confidence level, the robotic system 104 can use the offset to increase the likelihood of selecting the refined edge location more likely to be included within or overlapping the corresponding physical surface and reducing the risk of selecting / positioning a refined edge past / outside of the actual surface. The refined edges can be used to adjust and finalize the detection hypotheses (e.g., such as by adjusting outer edges of the modified detection hypothesis 924)

[0187] Referring back to FIG. 1, the robotic system 104 may be configured to verify a detection hypothesis (e.g., following incorporation of the refined edges into the modified detection hypothesis) based on interior-based reasoning derived from center heatmaps. The processor 106 may evaluate one or more detection hypotheses generated from preprocessed image data. The processor 106 evaluates each detection hypothesis based on whether the interior of the hypothesized region exhibits geometric characteristics consistent with a single object surface, in addition to the boundary or feature consistency. Particularly, the robotic system 104 may generate and analyze a center heatmap associated with a candidate region corresponding to a detection hypothesis to determine whether the spatial distribution of image data points within that region supports the presence of a single object. The verification process may include assessing the strength, uniqueness, and spatial coherence of interior likelihood values within the center heatmap. In instances the center heatmap indicates a strong and well-defined interior region aligned with the detection hypothesis, the robotic system 104 may verify the hypothesis as corresponding to a valid object. Conversely, in instances the heatmap indicates weak interior support, multiple interior regions, or misalignment with the hypothesized boundaries, the robotic system 104 may reject, split, or further refine the detection hypothesis prior to proceeding with physical interaction. Example process of generating a heatmap may be described in further detail with respect to FIGS. 11 A and 1 IB.

[0188] For example, FIG. 11A illustrates a portion of preprocessed image data 1102 depicting a surface of an object for which center-based analysis is to be performed in424932-7414-0331UP335240.W0.01accordance with at least one embodiment of the present technology. The image data 1102 illustrates a generally rectangular object surface with four detected comer features, including a first comer 1604a, a second comer 1604b, a third comer 1604c, and a fourth comer 1604d (collectively referred to as the corners 1604). The corners 1604 may correspond to refined and validated corners determined through one or more of the processes described above, such as comer extraction, detection hypothesis refinement, and edge validation. The corner positions can define the extents of the object surface region under consideration and may serve as geometric anchors for subsequent center heatmap computation.

[0189] In some embodiments, the system uses the detected corners 1104 to generate an estimated region that approximates the boundaries of the object surface. For example, FIG.11B illustrates the estimated 1106 defined based on the comers 1104 in accordance with at least one embodiment of the present technology. The estimated region 1106 approximates the geometric extent of the object surface under evaluation. The estimated region 1106 may serve as a reference area for interior analysis and need not correspond exactly to a final object boundary.

[0190] FIG. 11B further illustrates a center heatmap 1108 corresponding to the estimated region 1106. The center heatmap 1108 encodes, for each image data point within the bounding region, a centeredness value representing the relative likelihood that the point lies toward the interior of the object surface rather than near the boundaries. In the illustrated embodiment, grayscale intensity is used to visually represent the centeredness values, with higher intensity or darkness indicating higher centeredness and lower intensity or darkness indicating lower centeredness.

[0191] To generate the center heatmap 1108, the system (e.g., the robotic system 104) can first estimate a surface center for the object surface region. In some embodiments, the estimated center is determined as the intersection of diagonals formed by connecting opposite corners of the estimated region 1106. The estimated center does not necessarily correspond to a true center of mass or grasp location; instead, the estimated center functions as a geometric reference point for evaluating relative spatial relationships within the object surface.

[0192] In some embodiments, for each image data point p within the bounding region, the system can compute distances from p to the nearest boundary edge of the bounding region 1106 and a distance from p to the estimated center. The distance from p to the closest edge may be represented as d-min, which is selected from multiple edge distances434932-7414-0331UP335240.W0.01(for example, deiand de2) based on which edge is nearer. The distance from p to the estimated center is represented as de. Based on such distances, the system may calculate a centeredness value Cv (e.g., a measure of proximity to the estimated center for p) for the image data point according to the relationship. In these and other embodiments, the centeredness value Cv may be mathematically represented as Cv = (d-miri) I (d-miri + de). In these and other embodiments, image data points located near the estimated center may yield higher centeredness values, while points near the edges of the bounding region yield lower centeredness values.

[0193] The resulting center heatmap 1108 provides a spatial likelihood distribution that emphasizes the interior of the object surface and deemphasizes boundary regions. Such representation is robust to local edge detection errors and partial boundary inaccuracies, as centeredness values are derived from relative geometry rather than discrete feature presence. In some embodiments, the center heatmap 1108 is used to validate or refine detection hypotheses, identify multiple objects within an oversized hypothesis, estimate boundaries between adjacent objects, or assist in selecting interior grasp locations.

[0194] With reference back to FIG. 1, in some embodiments, the center heatmaps may be generated via the Al model 112. The Al model 112 is configured to produce, from image data associated with a candidate detection hypothesis, a spatial map representing interior likelihoods of object surfaces. The Al model 112 learns to infer interior structure directly from data patterns observed in 2D and 3D image information, rather than computing centeredness values solely through explicit geometric formulas.

[0195] In some embodiments, the Al model 112 receives input that includes preprocessed 2D image data, 3D depth data, or derived representations such as surface normal, segmented regions, or bounding regions defined by detected corners, among others. The Al model 112 may additionally infer interior structure by learning correlations between visual appearance, depth continuity, and object geometry from training data. For example, the Al model 112 may be trained to recognize spatial patterns indicative of a single object surface and to assign higher interior likelihood values to image data points that are centrally located within such patterns and to assign lower values to image data points near object boundaries or transitions between objects.

[0196] In some embodiments, the Al model 112 may be trained based on training examples including image data of objects together with corresponding ground-truth interior representations. For example, training data may include labeled object surfaces in which interior regions are identified based on known object geometry, successful grasp444932-7414-0331UP335240.W0.01locations, and / or validated detection hypotheses from prior system operation. In some embodiments, training labels may be derived from successful robotic interactions, such as cases in which an object was grasped and removed without interference, indicating that the hypothesized interior region corresponded to a valid object surface. The Al model 112 learns to associate visual and depth patterns, such as depth continuity, surface uniformity, and relative position within a bounding region, with higher interior likelihoods, and to associate boundary transitions, occlusions, and / or mixed object features with lower likelihoods.

[0197] In some embodiments, the Al model 112 may be trained to generalize across variations in object size, orientation, surface texture, and partial occlusion. In these and other embodiments, the Al model 112 may generate center heatmaps that remain robust in instances explicit edge detection is incomplete and / or in instances object boundaries are ambiguous. During inference, the Al-generated center heatmap may exhibit one or more interior peaks corresponding to likely object centers, with likelihood values gradually decreasing toward object boundaries. The processor 106 may analyze the resulting heatmap to evaluate accuracy of the detection hypothesis and / or presence of more than one object within the heatmap.

[0198] In some embodiments, the robotic system 104 may combine Al-generated center heatmap with geometrically computed heatmaps segmentation results, and / or comer-based information to improve robustness. For example, the processor 106 may compare the spatial alignment of the Al-generated heatmap with a bounding region defined by detected corners or may use the Al output to adjust thresholds or offsets used in geometric verification.

[0199] In some embodiments, the robotic system 104 may generate multiple estimated regions corresponding to multiple sets of detected comers. For example, FIG. 12 illustrates multiple estimated regions 1202 generated based on comer detection in accordance with at least one embodiment of the present technology. FIG. 12 illustrates only a few estimated regions for illustrative purposes. Each bounding box represents a candidate region that may correspond to an object surface or a portion thereof. The estimated regions 1202 may be generated based on techniques described in the present technology, such as defining bounding boxes from sets of comers that approximate object edges.

[0200] In some embodiments, the estimated regions 1202 may serve as candidate regions for further interior-based analysis. In these and other embodiments, the estimated regions 1202 need not be mutually exclusive and may overlap, particularly in dense or454932-7414-0331UP335240.W0.01cluttered scenes. In some embodiments, a center heatmap may be generated for each bounding box. For example, FIG. 13 illustrates center heatmaps 1302 generated for the estimated regions 1202 of FIG. 12 in accordance with at least one embodiment of the present technology. The heatmaps 1302 encode interior likelihood values for image data points within corresponding estimated regions 1202. In some embodiments, the heatmaps 1302 may be generated using center heatmap computation described with respect to FIGS.6A and 6B, such as estimating a center for each bounding box and computing centeredness values for image data points relative to the estimated center and bounding edges. Additionally or alternatively, the heatmaps 1302 may be generated based on Al models (e.g., the Al model 112 of FIG. 1).

[0201] In some embodiments, each center heatmap may be generated independently for each bounding box, permitting multiple hypotheses to be evaluated in a substantially parallel manner. In some instances, the heatmaps 1302 may overlap spatially, reflecting that multiple bounding boxes may cover adjacent and / or partially overlapping object regions.

[0202] Referring back to FIG. 1, in some embodiments, the robotic system 104 may revise and / or expand the initial detection hypotheses based on the heatmaps. The initial detection hypotheses generated based on corner and line detection alone may lead to inaccurate results due to quality of the image data or information. For instance, the vision system 108 may have limited or insufficient information due to point cloud noise or glare due to reflections off packing tapes, among others. For example, FIG. 14 A illustrates a reflective region 1402, in which packing tape causes reflection of the light in accordance with at least one embodiment of the present technology. The reflection may cause noise in the image information, leading to inaccurate feature extraction and / or hypothesis generation.

[0203] FIG. 14B illustrates an example initial detection hypothesis 1404 to be revised and / or expanded in accordance with at least one embodiment of the present technology. In some embodiments, the robotic system may generate the initial detection hypothesis 1404 as a bounded region corresponding to a minimum possible object size. The minimum possible object size may represent the smallest 2D surface dimensions of an object that the robotic system is expected to encounter in the environment. The robotic system may use the minimum size as a conservative starting point for the hypothesis generation.

[0204] In some embodiments, the initial detection hypothesis 1404 may be aligned with an open corner 1906 identified in the image data. The open corner 1406 refers to a comer464932-7414-0331UP335240.W0.01that is not adjacent to or overlapped by other objects and may be determined based on 3D image data. For example, no other object surfaces may be located above or immediately adjacent to the open corner 1406 based on the 3D image data.

[0205] The quality and / or size of the initial detection hypothesis 1404 may be affected by noise and / or occlusions in the image data. For example, the initial detection hypothesis 1404 may be affected by noisy region 1408 which may represent glare from packing tape, reflections, or unreliable depth measurements. In some embodiments, the robotic system may identify such noisy regions from preprocessing or feature extraction. In these and other embodiments, the initial detection hypothesis 1404 may be generated to avoid the noisy region 1908.

[0206] FIG. 14C illustrates center heatmaps 1410 corresponding to the objects in FIG.14B in accordance with at least one embodiment of the present technology. Multiple center heatmaps are illustrated superimposed on preprocessed image data, with each heatmap corresponding to a candidate object region derived from an initial detection hypothesis.

[0207] In some embodiments, the center heatmaps 1410 may be generated based on heatmap generation process, such as described with respect to FIGS. 13A and 13B. For example, for each initial detection hypothesis generated in FIG. 14B, the robotic system may define an estimated region based on detected corners or a minimum expected object size. Based on the estimated region, the system can estimate a surface center and computes centeredness values for image data points within the region. The centeredness values form a spatial distribution emphasizing interior portions of the object surface.

[0208] FIG. 14D illustrates determination of centered regions 1412 derived based on the center heatmaps 1410 in accordance with at least one embodiment of the present technology. The centered regions 1412 can correspond to subsets of image data points selected based on centeredness values computed from the respective center heatmaps. In some embodiments, the system evaluates centeredness values associated with individual image data points within the heatmap. The system can identify the centered regions 1412 as image data points having centeredness values greater than a predefined centeredness threshold. As the estimated center of a center heatmap has the highest likelihood of belonging to the interior of an object surface, image data points exceeding the point-centeredness threshold are more likely to correspond to interior portions of the same object as the estimated center.

[0209] The image data points determined to exceed the centeredness threshold are collectively grouped to form a centered region. Each centered region therefore represents474932-7414-0331UP335240.W0.01a spatially coherent area around the estimated center of an object surface where the system has an increased degree of confidence that the image data belongs to a single object.

[0210] In these and other embodiments, the system may incorporate the centered regions to expand, constrain, and / or otherwise modify the initial detection hypothesis (e.g., the initial detection hypothesis 1404). For example, the system may expand the initial detection hypothesis to include image data points within the centered region. In some instances, portions of the centered region may extend beyond the original hypothesis boundary. Conversely, portions of the initial detection hypothesis that fall outside the centered region may be deprioritized or excluded from further refinement.

[0211] FIG. 14E illustrates verification of the initial detection hypothesis 1406 using the centered region 1412 derived from the corresponding center heatmap in accordance with at least one embodiment of the present technology. The robotic system 104 can verify the initial detection hypothesis 1406 when a sufficient portion (e.g., according to a predetermined threshold) overlaps the center heat map.

[0212] For the determination, the initial detection hypothesis 1406 generated in FIG. 14B can be overlapped and / or compared with the centered region 1412 determined in FIG. 14D. In some embodiments, the system overlays or compares the spatial extent of the initial detection hypothesis 1906 with the centered region 1412 to confirm that both representations correspond to the same object in the scene. As the centered region 1412 represents image data points with high interior likelihood relative to the estimated center of the object surface, substantial overlap between the centered region 1412 and the initial detection hypothesis 1906 can indicate that the hypothesis is anchored to the interior of a single object rather than to noise or adjacent objects.

[0213] FIG. 14F illustrates expansion of the detection hypothesis using information derived from candidate edges and the extent of the centered region in accordance with at least one embodiment of the present technology. In some embodiments, in response to verifying the initial detection hypothesis using the centered region, the system expands the initial detection hypothesis outward to better approximate the full extent of the object surface. The system identifies candidate edges 1418 based on available line detection results, such as open edges that are not adjacent to other objects. For example, an open edge 1422 may be used as a left boundary of the expanded hypothesis, while detected candidate edges 1418 may be used to define upper and right boundaries.

[0214] In some embodiments, when the initial detection hypothesis 1406 overlaps but leaves a qualifying amount of the heat map uncovered, the robotic system 104 can expand484932-7414-0331UP335240.W0.01the detection hypothesis to include the area covered by the heat map. After merging the initial detection hypothesis 1406 and the center heat map, the robotic system 104 can detect and consider additional edges outside of or surrounding the combined region. The robotic system 104 can use one or more of the processes described above, such as expanding the edges, examining regions encircled by the edges, evaluating the encircled regions for overlap with the combined region, and / or the like, to further expand the combined region and arrive at the detection hypothesis.

[0215] For portions of the object boundary in which reliable edge information is unavailable, such as regions affected by depth noise or glare, the system may rely on the spatial extent of the centered region to define a boundary. For example, the bottom boundary 1420 of the detection hypothesis may be determined based on the bottom-most extent of the centered region, rather than on edge detection results. The initial detection hypothesis may incorporate the candidate edges and centered-region-based boundary to define expanded detection hypothesis 1424. The expanded detection hypothesis 1424 may correspond more closely to the full object surface than the initial detection hypothesis. In some embodiments, the candidate edges 1418 may be determined based on analysis and filtering of line segments detected via line detection.

[0216] FIG. 15A illustrates detected line segments 1502 in accordance with at least one embodiment of the present technology. For example, FIG. 15A illustrates results of line detection performed on preprocessed image data corresponding to a detection hypothesis. The system can implement a line detection algorithm to the image data, resulting in multiple detected line segments 1502 overlaid on the image depicting the object surfaces. The detected line segments 1502 represent candidate edges derived from geometric transitions in the image data, such as pixel values, changes in depth, surface orientation, or image gradients.

[0217] In some embodiments, the system analyzes and filters the detected line segments to identify candidate edges suitable for refining or validating the detection hypothesis. For example, the system may evaluate each detected line segment based on geometric properties such as length, orientation, and spatial consistency. For instance, the line segments 1502 whose lengths fall below a threshold may be removed, as such segments may correspond to noise, surface texture, printed markings, or glare rather than true object boundaries. The system may further group line segments 1502 based on orientation or clustering, retaining those segments that align with expected object edges, such as predominantly vertical or horizontal edges.494932-7414-0331UP335240.W0.01

[0218] The initial line detection may produce numerous line segments, including fragmented or spurious segments caused by surface irregularities or noisy image regions. Accordingly, the system does not treat all detected line segments as valid edges. Instead, the system selectively retains line segments 1502 that satisfy one or more filtering criteria and uses those retained segments to identify candidate edges associated with the detection hypothesis, such as the candidate edges described with respect to FIG. 14F.

[0219] FIG. 15B illustrates a superimposed representation of 2D and 3D image data for a portion of the scene that includes one or more objects in accordance with at least one embodiment of the present technology. For some instances, the superimposed representation can correspond to a portion of the image shown in FIG. 15A resulting from a segmentation process as described above. The combined image data highlights a noisy region 1504 in which the underlying image information may be unreliable. Such noise may arise, for example, from point-cloud degradation caused by glare, reflections from packing tape, surface irregularities, or sensor limitations. Although portions of the image data in the noisy region 1504 may be deemed unreliable for precise geometric analysis, the image data may still be usable for certain processing steps in instances the noise remains below a defined threshold.

[0220] FIG. 15C illustrates results of line detection performed on the image data depicted in FIG. 15B, such as using a finer granularity / sensitivity as described above, in accordance with at least one embodiment of the present technology. The system can implement a line detection algorithm that may utilize surface normals calculated from the 3D image data or on gradients derived from the combined image representation. For example, unlike top view images, side view images may depict object that have been rotated and not directly facing the image device. Hence, the depicted surfaces may be angled or rotated. The robotic system 104 can be configured to account for such orientations by computing the surface normal for an evaluated portion of the image. The robotic system 104 can consider portions of the image data having parallel surface normal and / or consistent / linear gradients as possibly belonging to the same physical surface. Due to the presence of noise in the underlying image data, the line detection process may produce numerous false, fragmented, or inaccurately oriented line segments. The spurious line segments may be referred to as feature artifacts 1506.

[0221] FIG. 15D illustrates generation of center heatmaps 1508 based on the same image data and extracted features used in FIG. 15C in accordance with at least one embodiment of the present technology. Unlike the line detection results, the center504932-7414-0331A1P335240.W0.01heatmaps 1508 emphasize interior likelihoods of object surfaces rather than boundary features. The center heatmaps 1508 exhibit coherent interior-focused patterns even in regions in which line detection produces significant feature artifacts (e.g., the feature artifacts 1506).

[0222] FIG. 15E illustrates estimation of an object boundary (e.g., representative of peripheral edges of the one physical surface of one object) using centeredness values derived from one or more center heatmaps 1514 in accordance with at least one embodiment of the present technology. In some embodiments, the system may utilize interior-based information, rather than direct edge detection alone, to determine a likely boundary location between adjacent objects or between an object and surrounding space.

[0223] In some embodiments, the system evaluates centeredness values across a region of image data that spans two or more adjacent center heatmaps, such as a center heatmap corresponding to a target object and a directly adjacent center heatmap corresponding to a neighboring object. For example, the system aggregates centeredness values along a set of rows or columns of image data points that traverse the region between the two heatmaps. The aggregation may include computing a sum, mean, median, or other statistical measure of the centeredness values for each row or column.

[0224] FIG. 15E further illustrates a chart in which the horizontal axis represents the aggregated centeredness values and the vertical axis represents a row number or spatial position within the image data in accordance with at least one embodiment of the present technology. The plotted curve 1512 corresponds to the aggregated centeredness values derived from the image data, such as from the center heatmap 1514. As the curve 1512 progresses, the aggregated centeredness values increase as the analysis approaches the estimated center of the target object, forming a peak near the interior of the object. The aggregated centeredness values then decrease as the analysis moves away from that center toward an adjacent object or boundary region.

[0225] In some embodiments, the system identifies an object boundary estimate 1516 at or near a minimum of the aggregated centeredness values between adjacent interior peaks. The minimum may represent a location at which interior likelihood is lowest and thus corresponds to a transition between objects or between an object and background space.

[0226] The system may then use the object boundary estimate 1516 to determine or refine an edge for a detection hypothesis. For example, the system may identify the boundary of the detection hypothesis as the row or column of image data points514932-7414-0331UP335240.W0.01corresponding to the object boundary estimate. In other embodiments, the system may apply an offset to the boundary estimate, such as selecting a boundary location at a fixed distance or a percentage of the distance between the boundary estimate and a neighboring interior peak, to account for noise or gradual transitions in centeredness values.

[0227] In some embodiments, the system further uses the object boundary estimate as a basis for analyzing image information to identify candidate edges. For instance, the system may examine a band or region of image data centered around the boundary estimate to identify line segments that align with the estimated boundary. For instance, the line segment 1516 identified within the region and aligned with the boundary estimate 1516 may be selected as a representative edge of the object. The selected edge may then be used to generate, refine, or validate the detection hypothesis, such as described with respect to FIG. 15F.

[0228] FIG. 15F illustrates a validated detection hypothesis 1518 generated following refinement and validation of a detection hypothesis based on center heatmap-based boundary estimation in accordance with at least one embodiment of the present technology. The validated detection hypothesis 1518 is shown as a bounded region corresponding to an object surface determined to be accurately aligned with the physical boundaries of the object depicted in the image data.

[0229] In some embodiments, the system may refine an initial or expanded detection hypothesis based on center heatmap-derived information rather than relying solely on direct edge detection. For example, in instances the direct edge detection produces unreliable or fragmented results due to noise in the image data — such as noise caused by reflections from packing tape or other reflective materials — the system may estimate object boundaries based on aggregated centeredness values. As described with respect to FIG. 10E, the system may identify an object boundary estimate at or near a minimum of aggregated centeredness values between interior regions.

[0230] Based on the object boundary estimate, the system refines one or more edges of the detection hypothesis. In these and other embodiments, the system may more accurately align a bottom edge of the detection hypothesis with the true physical bottom edge of the object. Such alignment may be achieved even in situations in which the 3D image data is degraded near the boundary due to reflective surfaces, glare, or depth dropouts.

[0231] In response to refining the detection hypothesis based on center heatmap-based edge estimation, the system may designate the resulting region as the validated detection hypothesis 1518. The validated detection hypothesis 1518 represents a detection region524932-7414-0331UP335240.W0.01that satisfies geometric consistency, interior likelihood coherence, and boundary alignment criteria. In some embodiments, the validated detection hypothesis 1518 may be used directly for downstream processes, such as generating grasp locations, issuing control commands to an end effector, updating detection history, and / or logically removing the corresponding region from further image analysis.

[0232] FIG. 16A illustrates an example of a center heatmap generated for a candidate object surface as part of a center heatmap validation subprocess in accordance with at least one embodiment of the present technology. For example, in some embodiments, the robotic system may determine the confidence of center heatmaps. In general, the center heatmap encodes interior likelihood values across a bounding region associated with a detection hypothesis. The region of highest centeredness, illustrated as a high-intensity area, represents an estimated center of the object surface. However, in some instances, the estimated center 1602 may not align with the actual physical center of the object surface depicted in the underlying image data. Such misalignment may occur, for example, in instances the bounding region is inaccurate, the detection hypothesis includes portions of adjacent objects, and / or the image noise or occlusion distorts the spatial distribution of centeredness values.

[0233] FIG. 16B illustrates a distribution 1604 of centeredness values derived from the center heatmap of FIG. 16A in accordance with at least one embodiment of the present technology. In some embodiments, the system evaluates the center heatmap by aggregating centeredness values along one spatial dimension, such as by summing the centeredness values along each column or row of the heatmap. With respect to the distribution 1604, the horizontal axis represents a spatial index, and the vertical axis represents the aggregated centeredness value for that index. The resulting curve represents how interior likelihood varies across the bounding region. The distribution 1604 reflects the spatial structure of the heatmap and provides a one-dimensional representation suitable for comparison against an expected interior-likelihood distribution.

[0234] FIG. 16C illustrates an expected distribution 1606 of centeredness values corresponding to a valid center heatmap in accordance with at least one embodiment of the present technology. In some embodiments, the expected distribution 1606 is modeled as a smooth, unimodal curve, such as a second-order polynomial or bell-shaped distribution, centered near the true interior of an object surface. The expected distribution may be generated analytically, empirically, or using regression techniques such as a y-hat linear regression graph. By comparing the distribution 1604 derived from the generated534932-7414-0331A1P335240.W0.01center heatmap with the expected distribution 1606, the system can evaluate whether the heatmap exhibits the characteristics of a valid interior representation.

[0235] In some embodiments, the system may determine an alignment value by comparing corresponding points of the derived distribution and the expected distribution, such as by summing differences between corresponding coordinates. In response to the alignment value falling below a predefined alignment threshold, the system may determine that the center heatmap is invalid for verifying or refining a detection hypothesis. Conversely, in instances the alignment satisfies the threshold, the center heatmap may be considered valid and used in subsequent processing.

[0236] FIG. 16D illustrates another example of center heatmap validation in accordance with at least one embodiment of the present technology. FIG. 16D illustrates a bounding box 1608 corresponding to a detection hypothesis enclosing a center heatmap that exhibits more than one region of elevated interior likelihood. For example, the center heatmap may include a first elevated region 1610 and a second elevated region 1612. The elevated regions correspond to multiple estimated surface centers within the same bounding box 11108. In some embodiments, the system analyzes the spatial distribution of the estimated surface centers and determines that the presence of multiple interior peaks is inconsistent with the expected structure of a single object surface. In such instances, the system may determine that the center heatmap is invalid for hypothesis verification. In some embodiments, as the bounding box used to generate the heatmap was derived from detected comers, the system may further flag the corresponding image region as having incomplete or unreliable extracted features and may trigger additional feature extraction, hypothesis splitting, or rejection of the detection hypothesis.

[0237] FIG. 17A illustrates a detection hypothesis 1702 generated prior to center heatmap-based validation in accordance with at least one embodiment of the present technology. The detection hypothesis 1102 is illustrated as a bounded region overlaid on preprocessed image data. The detection hypothesis 1702 may have been generated using comer and line detection processes described earlier. The detection hypothesis 1702 may represent a candidate object region based primarily on boundary -based features and has not yet been validated using interior-based reasoning.

[0238] FIG. 17B illustrates validation of the detection hypothesis 1702 of FIG. 17A using a corresponding center heatmap 1704 in accordance with at least one embodiment of the present technology. In some embodiments, the system evaluates alignment between the spatial extent of the detection hypothesis 1702 and the interior likelihood distribution544932-7414-0331A1P335240.W0.01encoded by the center heatmap 1704. For example, the system may overlay the center heatmap 1704 onto the detection hypothesis 1702 or otherwise compare the spatial correspondence. In instances a sufficient portion of the detection hypothesis 1702 overlaps with regions of high interior likelihood in the center heatmap 1704, and the alignment satisfies a predefined alignment threshold, the system may determine that the detection hypothesis 1702 is valid.

[0239] FIG. 17C illustrates a detection hypothesis 1706 subject to validation based on center heatmap in accordance with at least one embodiment of the present technology. The detection hypothesis 1706 may be generated using boundary -based feature extraction techniques, such as corner detection and line detection, including the processes described with respect to FIGS. 9A-9K and FIGS. 10A-10C. The detection hypothesis 1706 represents a candidate object region inferred from geometric features but has not yet been verified using interior-based reasoning.

[0240] FIG. 17D illustrates validation of the detection hypothesis 1706 of FIG. 17C based on a corresponding center heatmap in accordance with at least one embodiment of the present technology. The center heatmap may indicate one or more interior likelihood regions corresponding to actual object interiors, shown as regions of high centeredness. As illustrated, a single interior-based object region, referred to as a center-based object, aligns with only a portion of the detection hypothesis 1706. The remaining portions of the detection hypothesis do not correspond to regions of elevated interior likelihood.

[0241] In some embodiments, the system evaluates alignment between the detection hypothesis 1706 and the center heatmap 1710 by determining how many image data points within the detection hypothesis 1706 overlap with regions of high centeredness in the center heatmap. In instances the degree of alignment falls below a predefined alignment threshold, the robotic system 104 may determine that the detection hypothesis 1706 is invalid. In the example of FIG. 17D, as the center heatmap 1710 indicates that only part of the detection hypothesis corresponds to a true object interior, the system may reject the detection hypothesis.

[0242] FIG. 18 is a flow chart of an example method 1800 of operating a robot system, arranged in accordance with at least one embodiment of the present technology. The method 1800 may be implemented by any suitable element of a robotic system such as the robotic system 104 of FIG. 1 as described above. Although illustrated as discrete steps, various steps of the method 1800 may be divided into additional steps, combined into fewer steps, or eliminated, depending on the desired implementation. Additionally, the554932-7414-0331A1P335240.W0.01order of performance of the different steps may vary depending on the desired implementation.

[0243] In some embodiments, the method 1800 may begin at block 1802. At block 1802, image data of a scene including a set of objects may be obtained. In some embodiments, the image data may be obtained using a vision system, such as the vision system 108 of FIG. 1. In some embodiments, the vision system may include one or more vision sensors, such as left and right vision systems. The image data may include 2D image data and / or 3D depth data. The obtained image data may represent object surfaces, depth relationships, and / or spatial arrangements of the set of objects within the scene.

[0244] At block 1804, a detection hypothesis representing that at least one candidate region within the image data corresponds to an object of the set of objects may be generated for detecting and locating the set of objects or a subset thereof depicted in the image data. The robotic system may analyze the image data to generate one or more initial detection hypotheses. The detection hypotheses represent candidate object regions within the scene. As described with respect to FIGS. 4A-4C of present technology, the robotic system may identify geometric features such as comers, edges, surface normals, and / or segmented regions from the preprocessed image data and use the features to define an initial candidate region. For example, a detection hypothesis may be generated by anchoring a candidate region to an identified comer extending along adjacent edges, or by defining a minimum viable region corresponding to a graspable portion of an object.

[0245] At block 1806, a center heatmap for at least a portion of the at least one candidate region may be generated. The center heatmap may represent relative likelihoods of image data points being a part of the object surface of an object of the set of objects. The center heatmap may be generated by estimating a surface center for the candidate region and assigning centeredness values to image data points based on the relative distances to the estimated center and to boundaries of the region. In some embodiments, the estimated surface center may be determined based on inferred bounding region defined based on boundary-based features, such as detected comers, edges, surface normals, segmentation results, and / or a minimum viable region associated with a graspable portion of an object. In some embodiments, the center heatmap may be generated using an Al model trained to infer likelihoods from preprocessed image data. In some embodiments, the Al model may be trained using historical detection results and the pre-processing image data associated with the historical detection results, in which the Al model is configured to compute inferences at least partially based on the pre-processed image data.564932-7414-0331UP335240.W0.01

[0246] At block 1808, the detection hypothesis may be verified based on the center heatmap. For example, the robotic system may evaluate whether the detection hypothesis is supported by the interior-likelihood information encoded in the center heatmap. In some embodiments, the verification may include measuring an alignment value between the candidate region of the detection hypothesis and regions of elevated centeredness in the center heatmap. The alignment may analyzed with respect to an alignment threshold, and in response to the alignment satisfying the alignment threshold, the detection hypothesis may be verified as corresponding to an object. In response to the alignment being below the threshold, the detection hypothesis may be adjusted or rejected. In verifying the detection hypothesis based on the center heatmap, the robotic system can adjust (e.g., expand, shrink, combine, etc.) the existing detection hypothesis according to the center heatmap, such as to cover a thresholding portion of the center heatmap.

[0247] For instance, the verification process of the detection hypothesis based on the center heatmap may indicate that the detection hypothesis is not sufficiently accurate (e.g., above a threshold). The detection hypothesis may be adjusted to improve the accuracy. In some embodiments, the adjustment may include expanding the detection hypothesis outward from a region of elevated centeredness until a threshold condition is met. For example, in instances the detection hypothesis only includes a portion of the corresponding object, the detection hypothesis may be expanded, based on the center heatmap, to match the object surface more accurately.

[0248] Additionally or alternatively, the adjustment may include identifying multiple regions of elevated interior likelihood based on the center heatmap. For example, the detection hypothesis may include multiple estimated center regions based on the elevated centeredness. Such multiple estimated center regions may indicate possibility of the detection hypothesis including multiple adjacent objects. In these and other embodiments, a boundary between adjacent objects may be estimated based on a relative decrease in aggregated centeredness values between the multiple regions of elevated interior likelihood. The detection hypothesis may be divided into multiple objects or multiple detection hypotheses corresponding to the multiple regions.

[0249] Modifications, additions, or omissions may be made to the method 1800 without departing from the scope of the present technology. For example, the outline steps and operations are only provided as examples, and some of the steps and operations may be optional, combined into fewer steps and operations, or expanded into additional steps and operations without detracting from the essence of the disclosed embodiments.574932-7414-0331UP335240.W0.01

[0250] In some embodiments, the method 1800 may further include identifying a grasp location on the object based on the detection hypothesis and the center heatmap. The grasp location on the object may refer to a location on the object surface that is suitable for physical interaction by the robotic system, and particularly an end effector (e.g., the end effector 110 of FIG. 1) of the robotic system. The regions of elevated centeredness may provide indicators of interior surface areas that are less likely to coincide with object edges, adjacent objects, and / or unstable contact regions.

[0251] In some embodiments, the robotic system may select a grasp location corresponding to a local maximum or a region of elevated centeredness within the center heatmap, subject to one or more constraints such as minimum distance from detected boundaries, compatibility with the geometry of the end effector, or accessibility based on the robot’s kinematic limits. The selected grasp location may lie near the estimated surface center of the object or within a centered region identified during hypothesis refinement. In some embodiments, the robotic system may further adjust the grasp location based on feedback from previous grasp attempts, object removal history, or updated image data obtained after partial manipulation of the scene.

[0252] The identified grasp location may then be used to generate control commands for the end effector to engage the object. By basing grasp selection on the detection hypothesis and the center heatmap, the robotic system increases the likelihood of stable and reliable object interaction, particularly in dense or cluttered environments where boundary-based grasp selection alone may be unreliable.

[0253] In some embodiments, the method 1800 may further include obtaining updated image data in response to removal of one or more objects from the scene. For example, in response to a removal of an object corresponding to a verified detection hypothesis, the vision system may obtain new image data of the scene to reflect the updated arrangement of remaining objects. In some embodiments, the robotic system may also update previously obtained image data by logically removing regions corresponding to the removed object or by combining newly obtained image data with earlier image data.

[0254] For this and other processes and methods disclosed herein, the functions performed in the processes and methods may be implemented in differing order. Furthermore, the outlined operations are provided only as examples, and some of the operations may be optional, combined into fewer operations, or expanded into additional operations without detracting from the essence of the embodiments.Examples584932-7414-0331UP335240.W0.01

[0255] The subject technology is illustrated, for example, according to various aspects described below. Various examples of aspects of the present disclosure are described as numbered examples (1, 2, 3, etc.) for convenience. These are provided as examples and do not limit the present technology. The aspects of the various implementations described herein may be omitted, substituted for aspects of other implementations unless context dictates otherwise. For example, one or more aspects of example 1 below may be omitted, substituted for one or more aspects of another example (e.g., example 2) or examples, or combined with aspects of another example. The following is a non-limiting summary of some example implementations presented herein.1. An example method of operating a robot system, comprising: obtaining image data of a scene including a plurality of objects;identifying, based on the image data, features representative of one or more objects within the plurality;based on the identified features, generating a center heatmap representing relative likelihoods of image data points being a part of an object in the plurality of objects.2. The method of one or more examples herein, one or more portions thereof, or a combination thereof, wherein the features represent detection hypotheses representing that at least one candidate region within the image data corresponds to an object of the plurality of objects for detecting and locating the plurality of objects or a subset thereof depicted in the image data.3. The method of one or more examples herein, one or more portions thereof, or a combination thereof, further comprising:generating a center heatmap for at least a portion of the at least one candidate region, the center heatmap representing relative likelihoods of image data points being a part of the object surface of an object of the plurality of objects.4. The method of one or more examples herein, one or more portions thereof, or a combination thereof, further comprising:verifying the detection hypothesis based on the center heatmap.5. The method of one or more examples herein, one or more portions thereof, or a combination thereof, further comprising:594932-7414-0331UP335240.W0.01generating the detection hypothesis based on the center heatmap; or verifying the detection hypothesis based on the center heatmap.6. The method of one or more examples herein, one or more portions thereof, or a combination thereof, wherein the central heatmap is generated as a result of inference computation using a machine learning model configured to process the features as inputs.7. The method of one or more examples herein, one or more portions thereof, or a combination thereof, further comprising:providing the center heatmap and the identified features as training data; and training a machine learning model using the training data, wherein the machine learning model is trained to generate real-time heatmaps based on input features derived from a real-time image.8. The method of one or more examples herein, one or more portions thereof, or a combination thereof, wherein verifying the detection hypothesis includes adjusting the detection hypothesis based on the center heatmap.9. The method of one or more examples herein, one or more portions thereof, or a combination thereof, wherein the adjusting comprises expanding the detection hypothesis outward from a region of elevated centeredness until a threshold condition is met.10. The method of one or more examples herein, one or more portions thereof, or a combination thereof, wherein the adjusting comprises:identifying multiple regions of elevated interior likelihood based on the center heatmap;estimating a boundary between adjacent objects based on a relative decrease in aggregated centeredness values between the multiple regions of elevated interior likelihood;dividing the detection hypothesis into multiple objects corresponding to the multiple regions of elevated interior likelihood and the boundary.11. The method of one or more examples herein, one or more portions thereof, or a combination thereof, wherein the generating comprises:604932-7414-0331A1P335240.W0.01determining an estimated surface center and boundaries of the object; and assigning centeredness values to the image data points based on relative distances to the estimated surface center and to the boundaries of the object.12. The method of one or more examples herein, one or more portions thereof, or a combination thereof, wherein the estimated surface center is determined based on an inferred bounding region derived from one or more detected corners, edges, surface normal, or minimum viable regions.13. The method of one or more examples herein, one or more portions thereof, or a combination thereof, wherein the center heatmap is generated using an artificial intelligence (Al) model configured to generate the center heatmap based on pre-processed image data.14. The method of one or more examples herein, one or more portions thereof, or a combination thereof, wherein the pre-processed image data includes one or more of:masked regions corresponding to unreliable image areas;surface normal data;segmented planar surface data; andfiltered image data derived from prior detections.15. The method of one or more examples herein, one or more portions thereof, or a combination thereof, wherein the Al model is trained using historical detection results and the pre-processing image data associated with the historical detection results, wherein the Al model is configured to compute inferences at least partially based on the pre-processed image data.16. The method of one or more examples herein, one or more portions thereof, or a combination thereof, wherein the verifying comprises measuring an alignment value between the candidate region of the detection hypothesis and the center heatmap.17. The method of one or more examples herein, one or more portions thereof, or a combination thereof, wherein the detection hypothesis is adjusted in response to the alignment value being below an alignment threshold.614932-7414-0331UP335240.W0.0118. The method of one or more examples herein, one or more portions thereof, or a combination thereof, wherein the detection hypothesis includes a minimum viable region associated with a graspable portion of the object.19. The method of one or more examples herein, one or more portions thereof, or a combination thereof, wherein the minimum viable region is expanded or adjusted based on the center heatmap.20. The method of one or more examples herein, one or more portions thereof, or a combination thereof, further comprising obtaining updated image data in response to removal of one or more objects from the scene.21. The method of one or more examples herein, one or more portions thereof, or a combination thereof, further comprising identifying a grasp location on the object based on the detection hypothesis and the center heatmap.22. A robotic system, comprising:a communication interface configured to communicate with (1) one or more image sensors to obtain image data of a plurality of objects in an environment and (2) an end effector to interact with the plurality of objects; andone or more processors configured to perform the method of one or more examples 1-21, one or more portions thereof, of a combination thereof.23. A non-transitory computer-readable medium storing instructions that, when executed by one or more processors of a robotic system, cause the robotic system to perform the method of one or more examples 1-21, one or more portions thereof, of a combination thereof.

[0256] Terms used in the present technology and especially in the appended claims (e.g., bodies of the appended claims) are generally intended as “open” terms (e.g., the term “including” should be interpreted as “including, but not limited to,” the term “having” should be interpreted as “having at least,” the term “includes” should be interpreted as “includes, but is not limited to,” etc.).624932-7414-0331UP335240.W0.01

[0257] Additionally, if a specific number of an introduced claim recitation is intended, such an intent will be explicitly recited in the claim, and in the absence of such recitation no such intent is present. For example, as an aid to understanding, the following appended claims may contain usage of the introductory phrases "at least one" and "one or more" to introduce claim recitations. However, the use of such phrases should not be construed to imply that the introduction of a claim recitation by the indefinite articles "a" or "an" limits any particular claim containing such introduced claim recitation to embodiments containing only one such recitation, even when the same claim includes the introductory phrases "one or more" or "at least one" and indefinite articles such as "a" or "an" (e.g., “a” and / or “an” should be interpreted to mean “at least one” or “one or more”); the same holds true for the use of definite articles used to introduce claim recitations.

[0258] In addition, even if a specific number of an introduced claim recitation is explicitly recited, those skilled in the art will recognize that such recitation should be interpreted to mean at least the recited number (e.g., the bare recitation of "two recitations, " without other modifiers, means at least two recitations, or two or more recitations). Furthermore, in those instances where a convention analogous to “at least one of A, B, and C, etc.” or “one or more of A, B, and C, etc.” is used, in general such a construction is intended to include A alone, B alone, C alone, A and B together, A and C together, B and C together, or A, B, and C together, etc. Additionally, the use of the term “and / or” is intended to be construed in this manner.

[0259] Further, any disjunctive word or phrase presenting two or more alternative terms, whether in the description, claims, or drawings, should be understood to contemplate the possibilities of including one of the terms, either of the terms, or both terms. For example, the phrase “A or B” should be understood to include the possibilities of “A” or “B” or “A and B” even if the term “and / or” is used elsewhere.

[0260] All examples and conditional language recited in the present technology are intended for pedagogical objects to aid the reader in understanding the present technology and the concepts contributed by the inventor to furthering the art, and are to be construed as being without limitation to such specifically recited examples and conditions. Although embodiments of the present technology have been described in detail, various changes, substitutions, and alterations could be made hereto without departing from the spirit and scope of the present technology.634932-7414-0331U

Claims

P335240.W0.01CLAIMSWhat is claimed is:

1. A method of operating a robot system, comprising:obtaining image data of a scene including a plurality of objects;generating, based on the image data, a detection hypothesis representing that at least one candidate region within the image data corresponds to an object of the plurality of objects for detecting and locating the plurality of objects or a subset thereof depicted in the image data;generating a center heatmap for at least a portion of the at least one candidate region, the center heatmap representing relative likelihoods of image data points being a part of the object surface of an object of the plurality of objects; and verifying the detection hypothesis based on the center heatmap.

2. The method of claim 1, wherein verifying the detection hypothesis includes adjusting the detection hypothesis based on the center heatmap.

3. The method of claim 2, wherein the adjusting comprises expanding the detection hypothesis outward from a region of elevated centeredness until a threshold condition is met.

4. The method of claim 2, wherein the adjusting comprises:identifying multiple regions of elevated interior likelihood based on the center heatmap;estimating a boundary between adjacent objects based on a relative decrease in aggregated centeredness values between the multiple regions of elevated interior likelihood;dividing the detection hypothesis into multiple objects corresponding to the multiple regions of elevated interior likelihood and the boundary.644932-7414-0331UP335240.W0.

015. The method of claim 1, wherein the generating comprises:determining an estimated surface center and boundaries of the object; and assigning centeredness values to the image data points based on relative distances to the estimated surface center and to the boundaries of the object.

6. The method of claim 5, wherein the estimated surface center is determined based on an inferred bounding region derived from one or more detected corners, edges, surface normal, or minimum viable regions.

7. The method of claim 1, wherein the center heatmap is generated using an artificial intelligence (Al) model configured to generate the center heatmap based on pre-processed image data.

8. The method of claim 7, wherein the pre-processed image data includes one or more of:masked regions corresponding to unreliable image areas;surface normal data;segmented planar surface data; andfiltered image data derived from prior detections.

9. The method of claim 8, wherein the Al model is trained using historical detection results and the pre-processing image data associated with the historical detection results, wherein the Al model is configured to compute inferences at least partially based on the pre-processed image data.

10. The method of claim 1, wherein the verifying comprises measuring an alignment value between the candidate region of the detection hypothesis and the center heatmap.

11. The method of claim 10, wherein the detection hypothesis is adjusted in response to the alignment value being below an alignment threshold.654932-7414-0331UP335240.W0.0112. The method of claim 1, wherein the detection hypothesis includes a minimum viable region associated with a graspable portion of the object.

13. The method of claim 12, wherein the minimum viable region is expanded or adjusted based on the center heatmap.

14. The method of claim 1, further comprising obtaining updated image data in response to removal of one or more objects from the scene.

15. The method of claim 1, further comprising identifying a grasp location on the object based on the detection hypothesis and the center heatmap.

16. A robotic system, comprising:one or more image sensors configured to obtain image data of a plurality of objects in an environment;an end effector configured to interact with the plurality of objects; andone or more processors configured to:generate, based on the image data, a detection hypothesis including at least one candidate region corresponding to an object of the plurality of objects; generate a center heatmap for at least a portion of the at least one candidate region, the center heatmap representing relative likelihoods of image data points being a part of the object surface of an object of the plurality of objects;verify the detection hypothesis based on the center heatmap; and cause the robotic manipulator to interact with the object based on the detection hypothesis and the center heatmap.

17. The robotic system of claim 16, wherein the one or more processors are further configured to adjust the detection hypothesis based on the center heatmap.664932-7414-0331UP335240.W0.0118. The robotic system of claim 16, wherein the robotic manipulator is controlled to engage the object at a location corresponding to elevated centeredness values in the center heatmap.

19. A non-transitory computer-readable medium storing instructions that, when executed by one or more processors of a robotic system, cause the robotic system to:obtain image data of a scene including a plurality of objects;generate, based on the image data, a detection hypothesis including at least one candidate region corresponding to an object of the plurality of objects; generate a center heatmap for at least a portion of the at least one candidate region, the center heatmap representing relative likelihoods of image data points being a part of the object surface of an object of the plurality of objects; and verify the detection hypothesis based on the center heatmap.

20. The non-transitory computer-readable medium of claim 19, wherein the instructions further cause the robotic system to adjust the detection hypothesis based on the center heatmap.674932-7414-0331U