System and related methods for object identification, verification, and manipulation

Abstract

A system for identifying, verifying, and manipulating objects comprises a navigation module configured to navigate to a target location based on stored location data, a segmentation module configured to generate segmented regions from an input image, and a classification module operably connected to the segmentation module to identify a target object from the segmented regions. A pose estimation module determines the spatial pose of the target object, and an object manipulation module interacts with the target object using the spatial pose data. A verification module confirms the target object by capturing identifying information, and a control unit coordinates operation of the navigation, segmentation, classification, pose estimation, object manipulation, and verification modules. The system provides robust identification and interaction capabilities by integrating visual processing, spatial analysis, and verification techniques to handle complex operational scenarios, including partial object visibility, occlusions, or ambiguous identifying features.

Description

Title: System and Related Methods for Object Identification, Verification, and ManipulationField of Invention.

[0001] The present invention relates to systems and methods for automated object identification, verification, and manipulation for use in automated retail or logistics environments.Background

[0002] The automation of retail shelf stocking presents unique challenges that remain unsolved by existing solutions. While significant advances have been made in general object recognition and classification, the specific requirements of autonomous retail stocking -where robots must reliably identify, locate, and manipulate products in dynamic environments - create demands that exceed the capabilities of current computer vision approaches.

[0003] A fundamental challenge lies in the simultaneous need for both broad product recognition and precise verification in retail environments. While existing computer vision systems can achieve reasonable accuracy in either general classification tasks or specific verification tasks, the retail stocking use case requires both capabilities working in concert. A stocking robot must not only identify the correct product category but also verify exact SKUs, detect damaged items, and determine precise grip points and placement orientations - all while operating at commercially viable speeds.

[0004] Current approaches typically rely on either pure computer vision methods or traditional barcode-based identification systems, each with significant limitations. Pure vision-based systems, while flexible, struggle with similar-looking products, packaging variations, and partial occlusions common in retail environments. These systems often fail to provide the near-perfect accuracy required for automated stocking, where errors can lead to significant operational disruptions. Conversely, barcode-based systems offer precise identification but require specific product orientations and unobstructed views of identifying markers, making them impractical for autonomous robotic applications.

[0005] The challenge is further complicated by the dynamic nature of retail environments. Products are frequently updated with new packaging, shelf layouts change regularly, and lighting conditions vary throughout the day. Traditional computer vision approaches, which often rely on static databases of product images or fixed feature matching, struggle to adapt to these continuous changes. Moreover, the need to operate in real-time while products are being handled by robots adds additional complexity, as the system must maintain accuracy while processing images from multiple angles and under varying conditions.

[0006] Existing solutions have attempted to address these challenges through various means. Some systems employ deep learning models trained on large product databases, but these often fail when encountering new packaging variations or when products are partially obscured. Others combine multiple cameras or sensors, but these approaches typically increase system complexity and cost while still failing to achieve the required reliability. Additionally, many current solutions require controlled lighting conditions or specific product presentations, making them impractical for real-world retail applications.

[0007] Another significant limitation of current approaches is their inability to effectively handle the verification stage of the stocking process. While a system might correctly identify a product category, it must also verify specific attributes such as expiration dates, price tags, and proper orientation before placement. This multi-stage verification requirement, combined with the need for real-time processing to support robotic operations, creates a technical challenge that existing single-pipeline vision systems have failed to address adequately.

[0008] Furthermore, current computer vision solutions often struggle with the physical context required for robotic stocking. Beyond simple identification, the system must provide precise spatial information for grip point selection and placement orientation. This requirement for both semantic understanding and spatial awareness represents a unique challenge that goes beyond traditional object recognition tasks.

[0009] It is within this context that the present invention is provided.Summary

[0010] The present invention provides a system for identifying, verifying, and manipulating objects in an automated manner. The system comprises a navigation module for navigating to a target location, a segmentation module for generating segmented regions from an input image, and a classification module for identifying a target object from among the segmented regions. A pose estimation module determines the spatial pose of the identified target object, while an object manipulation module interacts with the object using the spatial pose data. A verification module confirms the target object by capturing identifying information, such as barcodes or alphanumeric text, and a control unit coordinates the operation of these components.

[0011] The invention provides a reliable multi-layered approach to object identification and verification by integrating computer vision methods with conventional object recognition techniques. The combination of segmentation, classification, pose estimation, and verification modules ensures robust handling of object identification even in challenging scenarios such as occlusions or damaged identifying features. The navigation module and control unit provide precise movement to target locations while enabling coordination between components for efficient operation.

[0012] In some embodiments, the classification module includes a visual language model configured to prioritize segmented regions based on semantic relationships between an input prompt and the visual features of the objects. This allows the system to handle complex queries, including ambiguous or partial descriptions.

[0013] In further embodiments, the classification module includes an embeddings-based search model to compare segmented regions with a stored database of object descriptors. This enables robust identification of objects with variable visual appearances or incomplete database entries.

[0014] In some embodiments, the segmentation module uses machine learning-based segmentation models to generate masks corresponding to distinct objects in the input image. This approach enhances the system's ability to identify objects even in cluttered or overlapping scenarios.

[0015] In yet further embodiments, the pose estimation module determines the six-degree-of-freedom pose of the target object and refines the pose data when the object is partially visible or occluded. This improves the accuracy of object manipulation, particularly for misoriented or obstructed objects.

[0016] In some embodiments, the verification module includes a barcode scanning module for reading identifying barcodes and an optical character recognition (OCR) module for extracting alphanumeric data. This dual verification process ensures reliable object confirmation using multiple identifying features. In other embodiments the verification module includes an RFID reader.

[0017] In further embodiments, the barcode scanning module is configured to detect a variety of barcode symbologies, including linear and two-dimensional barcodes, providing compatibility with existing product labeling systems.

[0018] In yet further embodiments, the verification module cross-references identifying information, such as expiry dates or batch codes, with a stored dataset to validate the accuracy and quality of the object. This feature enables additional verification parameters beyond product identity

[0019] In some embodiments, the object manipulation module is configured to manipulate the target object, including rotation, to facilitate identification or verification by aligning specific features, such as barcodes, with the verification module.

[0020] In further embodiments, the navigation module includes both two-dimensional navigation for general movement to a target location and three-dimensional navigation for refined positioning within a localized area, such as a shelf or storage zone. This combination ensures efficient and accurate navigation.

[0021] In yet further embodiments, the segmentation, classification, and pose estimation modules operate iteratively to refine object identification and selection. This iterative process enhances accuracy when handling ambiguous or incomplete visual data.

[0022] In some embodiments, the system includes a control unit that updates a digital map with the position and identification of objects. This enables continuous tracking and updating of object locations within a defined environment.

[0023] In further embodiments, the system includes user prompts for manual interaction when verification or identification processes are unsuccessful. This feature provides a fallback for exceptional scenarios where automated processes require human input.

[0024] In yet further embodiments, the system processes partial visibility of objects through the segmentation and classification modules, enabling identification when only a portion of the target object is visible.

[0025] In some embodiments, the classification module generates error notifications when no segmented regions match the target object. This ensures the system can report and manage identification failures efficiently.

[0026] In further embodiments, the navigation module incorporates planogram data to determine an initial search space, reducing processing time by limiting the system's operational area.

[0027] In yet further embodiments, the object manipulation module comprises an actuator capable of executing grasping, picking, and placing operations. This allows the system to interact with the target object effectively based on its spatial pose.

[0028] In some embodiments, the segmentation module operates in real time on a video stream, enabling dynamic and continuous object detection and segmentation.

[0029] In yet further embodiments, the classification module includes a logic processing unit configured to interpret partial or ambiguous prompts dynamically, improving the system's ability to handle task-specific contexts.Brief Description of the Drawings

[0030] Various embodiments of the invention are disclosed in the following detailed description and accompanying drawings.

[0031] FIG. 1 illustrates an example system architecture showing the modules and components for object identification, verification, and manipulation.

[0032] FIG. 2 illustrates an example process flow diagram representing the sequential operations of the system for identifying, verifying, and manipulating objects.

[0033] FIG. 3 illustrates an example modular architecture of the computer vision system, showing interactions between nodes, user inputs, and outputs.

[0034] Common reference numerals are used throughout the figures and the detailed description to indicate like elements. One skilled in the art will readily recognize that the above figures are examples and that other architectures, modes of operation, orders of operation, and elements / functions can be provided and implemented without departing from the characteristics and features of the invention, as set forth in the claims.Detailed Description and Preferred Embodiment

[0035] The following is a detailed description of exemplary embodiments to illustrate the principles of the invention. The embodiments are provided to illustrate aspects of the invention, but the invention is not limited to any embodiment. The scope of the invention encompasses numerous alternatives, modifications and equivalent; it is limited only by the claims.

[0036] Numerous specific details are set forth in the following description in order to provide a thorough understanding of the invention. However, the invention may be practiced according to the claims without some or all of these specific details. For the purpose of clarity, technical material that is known in the technical fields related to the invention has not been described in detail so that the invention is not unnecessarily obscured.DEFINITIONS:

[0037] The terminology used herein is for the purpose of describing particular embodiments only and is not intended to be limiting of the invention.

[0038] As used herein, the term "and / or" includes any combinations of one or more of the associated listed items.

[0039] As used herein, the singular forms "a," "an," and "the" are intended to include the plural forms as well as the singular forms, unless the context clearly indicates otherwise.

[0040] It will be further understood that the terms "comprises" and / or "comprising," when used in this specification, specify the presence of stated features, steps, operations, elements, and / or components, but do not preclude the presence or addition of one or more other features, steps, operations, elements, components, and / or groups thereof.

[0041] The term "navigation module" refers to any system, component, or combination of components configured to enable movement to a specified location within an environment. This includes, but is not limited to, two-dimensional (2D) and three-dimensional (3D) navigation systems, global positioning systems (GPS), indoor localization systems, and pathplanning algorithms. In one example implementation, the navigation module may include a 2D map for general store-level navigation using SLAM (Simultaneous Localization and Mapping) or ACML (Adaptive Monte-Carlo Localizer) and a 3D localization system based on depth point clouds for precise shelf-level positioning.

[0042] The term "segmentation module" refers to any software, hardware, or combination thereof configured to generate segmented regions corresponding to distinct objects in an input image or video stream. This includes, but is not limited to, machine learning-based segmentation models, computer vision algorithms, and edge-detection methods. In one example implementation, the segmentation module may employ a neural network model, such as the Segment Anything Model 2 (SAM2) or U-Net, to process a video feed and produce masks representing individual objects on a retail shelf.

[0043] The term "classification module" refers to any system, process, or component configured to identify, categorize, or prioritize one or more objects based on input data. This includes, but is not limited to, visual language models (VLMs), machine learning classifiers, embeddings-based search models, and object recognition algorithms. In one example implementation, the classification module may use a pre-trained CLIP model to compare segmented objects with a set of descriptors or database entries to identify the target object. The classification module may also handle partial, ambiguous, or informal user queries by interpreting task-specific context dynamically.

[0044] The term "pose estimation module" refers to any system or algorithm configured to determine the spatial position and orientation of an object in three-dimensional space. This includes, but is not limited to, 6D pose estimation algorithms, point-cloud analysis, and geometric reconstruction techniques. In one example implementation, the pose estimation module may employ point cloud bounding box fitting or algorithms like foundation or diffusion algorithms to calculate a six-degree-of-freedom pose of the target object, allowing for reliable pose estimation in the presence of occlusions and incomplete data.

[0045] The term "object manipulation module" refers to any mechanical, robotic, or electromechanical system configured to interact with an object based on spatial pose information. This includes, but is not limited to, robotic grippers, actuators, robotic arms, and manipulation planning algorithms. In one example implementation, the object manipulation module may comprise a robotic arm equipped with a mechanical gripper for picking up, rotating, and placing the target object based on the pose determined by the pose estimation module.

[0046] The term "verification module" refers to any system or component configured to confirm the identity or characteristics of an object. This includes, but is not limited to, barcode scanning modules, optical character recognition (OCR) systems, and database crossreferencing tools. In one example implementation, the verification module may use a barcode scanner, such as a laser-based or image-based reader, to decode linear or two-dimensional barcodes. The module may also include an OCR system to extract expiry dates, product names, or batch codes from printed labels.

[0047] The term "control unit" refers to any hardware, software, or combination thereof configured to coordinate and manage the operations of the navigation module, segmentation module, classification module, pose estimation module, object manipulation module, and verification module. This includes, but is not limited to, microcontrollers, processors, or software platforms capable of sending commands, receiving data, and managing workflows. In one example implementation, the control unit may execute pre-programmed instructions to integrate real-time data streams, synchronize component operations, and update digital maps with verified object locations.

[0048] The term "input image" refers to any visual representation, including still images, frames from a video feed, or data from depth sensors. This includes, but is not limited to, RGB images, infrared images, point cloud data, or multi-spectral imagery. In one example implementation, the input image may be captured by a depth camera, such as Intel RealSense camera and processed to detect and segment objects within a retail environment.

[0049] The term "barcode scanning module" refers to any system or device configured to detect and decode barcodes or similar identifying markers. This includes, but is not limited to, optical barcode scanners, image-based scanners, and software libraries such as pyzbar. In one example implementation, the barcode scanning module may extract an EAN (European Article Number) from a product label and compare it with a stored dataset for verification.

[0050] The term "planogram data" refers to any stored representation of the layout, arrangement, or positioning of objects within a defined environment. This includes, but is not limited to, digital maps, shelf layouts, product placement data, or inventory management databases. In one example implementation, the planogram data may be used to determine the initial search space for locating a target object on a shelf.

[0051] The term "six-degree-of-freedom pose" refers to the position and orientation of an object in three-dimensional space, including translation along three axes and rotation about those axes. This includes, but is not limited to, representations in Cartesian coordinates or Euler angles. In one example implementation, the pose is determined using a combination of visual data and geometric reconstruction algorithms.

[0052] The term "stored dataset" refers to any collection of data, including identifying information, visual descriptors, or contextual information related to objects. This includes, but is not limited to, product databases, SKU lists, batch information, and expiry date records. In one example implementation, the stored dataset may include product identifiers and associated metadata for cross-referencing during verification.

[0053] The invention may be implemented using alternative materials, devices, or algorithms suitable for achieving the described functionality. For instance, the barcode scanner may be replaced by other identifying mechanisms, such as RFID readers or NFC systems. Similarly, theobject manipulation module may employ alternative gripper designs, such as vacuum-based suction devices or soft robotic actuators, depending on the target environment.DESCRIPTION OF DRAWINGS

[0054] The present invention relates to a system and method for automated object identification, verification, and manipulation, particularly in environments requiring precise and reliable handling of objects, such as retail, logistics, and warehouse operations. The invention provides a multi-layered perception system that combines advanced computer vision techniques with verification processes to ensure robust and accurate identification and interaction with target objects. By integrating components such as navigation, segmentation, classification, pose estimation, object manipulation, and verification modules, the invention overcomes the limitations of existing systems that rely on single-method identification approaches, rigid database dependencies, or require specialized infrastructure modifications.

[0055] Existing systems in the prior art often suffer from significant shortcomings, including reliance on a single mode of verification such as barcode scanning or visual recognition alone. Such systems fail when products are misoriented, barcodes are damaged, or objects are partially visible due to occlusions. Furthermore, many prior solutions require exact SKU matches or perfect database entries, rendering them inflexible when handling ambiguous queries, packaging variations, or incomplete product data. The inability to operate dynamically in such conditions reduces their reliability and operational effectiveness.

[0056] The present invention addresses these shortcomings by combining multiple identification and verification steps into a cohesive workflow. Specifically, it incorporates advanced segmentation techniques to isolate individual objects, classification models capable of interpreting visual features and semantic relationships, and pose estimation to determine object positioning and orientation. The invention further includes a verification module to validate the target object through identifying information such as barcodes or printed text, ensuring reliable confirmation even when visual or positional conditions are suboptimal. By combining these processes, the invention reduces the dependency on perfect visibility, flawless product data, or manual intervention.

[0057] In addition, the invention improves operational efficiency by leveraging planogram data to reduce the search space and incorporating flexible navigation capabilities for precise movement to target locations. It is designed to integrate seamlessly with existing infrastructure, such as standard shelving systems, barcodes, and product packaging, without requiring specialized modifications. The invention further supports dynamic query processing, enabling it to respond to partial, informal, or ambiguous descriptions of target objects using advanced classification models, including visual language models and embeddings-based search.

[0058] Referring now to the drawings, FIG. 1 illustrates a system architecture for automated object identification, verification, and manipulation.

[0059] At the core of the system is a control unit 100, which coordinates the operation of various interconnected modules. The control unit 100 may be implemented as a processorbased platform, such as a microcontroller or a remote industrial computing server system, configured to send commands and receive operational data from other modules to manage the overall workflow. The control unit 100 may store pre-programmed instructions and manage real-time data flow.

[0060] An input acquisition module includes a video or image input source 102, which is used to capture images or video streams of a target environment. The input source 102 is depicted as a camera but may, in some implementations, include depth sensors, multi-spectral imaging devices, or infrared cameras. The input acquisition module may preprocess the visual input to filter noise or adjust lighting and may output RGB data, point cloud representations, or other image formats suitable for processing.

[0061] The system further comprises a navigation module 104, which enables movement to a specified location within a target environment. The navigation module 104 includes a two-dimensional navigation component for coarse movement to an initial target area and a three-dimensional navigation component for refined positioning relative to a shelf or subregion. The navigation module 104 may utilize mapping systems such as SLAM, ACML, or other indoor positioning tools. In some embodiments, navigation may be informed by planogram datastored in a planogram database, allowing the system to reduce the search space prior to detailed object identification.

[0062] Connected to the control unit 100 is a segmentation module 106, which generates segmented regions corresponding to individual objects within the input image. The segmentation module 106 may utilize machine learning-based models, such as the Segment Anything Model 2 (SAM2), U-Net, or similar neural networks capable of isolating objects in cluttered or overlapping conditions. Each identified region may be assigned a segmentation mask, which is passed to downstream modules for further processing.

[0063] A classification module 108 is operably connected to the segmentation module 106 and is configured to identify and categorize objects within the segmented regions. The classification module 108 may employ visual language models (VLMs), embeddings-based search methods, or pre-trained object recognition algorithms such as CLIP. The classification module 108 prioritizes segmented objects based on an input prompt, which may be dynamically interpreted to account for partial, ambiguous, or informal descriptions. The classification module 108 allows the system to respond to complex queries, such as identifying misplaced items or products described using semantic relationships.

[0064] Once an object has been identified, its spatial pose is determined using a pose estimation module 110. The pose estimation module 110 calculates the six-degree-of-freedom pose of the target object, determining its position and orientation in three-dimensional space. Pose data generated by the pose estimation module 110 enables precise interaction with objects, even when they are misoriented, partially visible, or occluded. The pose estimation module 110 may use algorithms such as pointcloud-based bounding boxes, foundation or diffusion models or similar geometric reconstruction techniques.

[0065] An object manipulation module 112 is configured to interact with the identified object based on the spatial pose data provided by the pose estimation module 110. The object manipulation module 112 may include an actuator, such as a robotic arm with a gripper, suction-based end effectors, or soft robotic mechanisms, to perform operations such as grasping, rotating, or placing the target object. In some implementations, the objectmanipulation module 112 may include an alignment component that adjusts the object's position to facilitate further verification, such as aligning a barcode for scanning.

[0066] Verification of the target object is achieved using a verification module 114, which may include a barcode scanning module and an optical character recognition (OCR) input 115. The verification module 114 ensures that the identified object matches a stored dataset by capturing identifying information such as barcodes, product names, expiry dates, or batch codes. The barcode scanning component may detect various barcode symbologies, including linear and two-dimensional formats, while the OCR module extracts text-based identifiers for further validation.

[0067] A storage and data management component includes a stored dataset containing product descriptors, SKUs, and metadata for cross-referencing during verification. The planogram database may also be used to store pre-defined shelf layouts and product positioning data, which informs the navigation module 104 and supports efficient object identification by reducing the search area.

[0068] The user interaction interface 116 allows for manual input or intervention when automated processes require refinement. For instance, the system may prompt the user to provide additional context during ambiguous classification tasks or to assist in aligning objects for verification. The interface 116 may include a graphical user interface (GUI) displayed on a monitor, a touchscreen interface, or command-line inputs.

[0069] Although the system is illustrated with specific components, it is understood that variations may be implemented without departing from the scope of the invention. For example, the navigation module 104 may employ alternative localization technologies, and the object manipulation module 112 may incorporate different gripper designs or materials based on the nature of the objects being handled. Similarly, the segmentation module 106 and classification module 108 may be replaced with alternative computer vision or recognition algorithms capable of achieving similar functionality. The verification module 114, while shown as including barcode scanning and OCR capabilities, may also incorporate alternative identification systems, such as RFID readers or NFC scanners, in some embodiments.

[0070] FIG. 2 illustrates a high-level process flow diagram of a system for automated object identification, verification, and manipulation.

[0071] The process begins at step 200, where the system receives an input task or prompt that specifies the required operation, such as identifying, verifying, or manipulating a target object. The input task may be provided as a direct instruction, such as "find the 2-liter Irish whole milk," or as an ambiguous request, like "pick up any misplaced item." At step 202, the system acquires input images or video streams of the environment using an input acquisition module, such as an RGB camera or a depth sensor. The input data may undergo preprocessing, including noise filtering, lighting adjustments, or format conversion, to ensure optimal image quality for subsequent analysis.

[0072] At step 204, the navigation module enables the system to navigate to the general vicinity of the target location based on pre-stored location data, such as a digital map or planogram database. The navigation module may include a two-dimensional navigation system for broad store-level positioning and a three-dimensional navigation system for fine movement to a shelf or designated subregion. Variants of this navigation step may incorporate SLAM-based algorithms or other localization systems, depending on the operating environment.

[0073] Once the system reaches the approximate location, step 206 refines the positioning to align the input source with the target area, ensuring the relevant portion of the environment is captured.

[0074] At step 208, the segmentation module processes the acquired image or video data to generate segmented regions, each corresponding to a distinct object within the visual input. The segmentation module may utilize machine learning-based models, such as the Segment Anything Model 2 (SAM2), or alternative segmentation tools that isolate objects even in scenarios with occlusions, overlapping products, or partial visibility. Each segmented region is assigned a mask, which is passed to subsequent modules for further processing. In some implementations, real-time segmentation may be performed on video streams, enabling continuous identification as the system navigates through the environment.

[0075] In step 210, the classification module analyzes the segmented regions to identify the target object based on visual features. The classification module may include visual language models (VLMs) or embeddings-based search models capable of comparing the segmented regions with stored object descriptors. If an input taskincludes partial, ambiguous, orinformal descriptions, the classification module dynamically interprets the task context to prioritize likely matches. For instance, the system may process a query such as "front all orange juice bottles" by identifying objects that match the visual and semantic characteristics of orange juice products. In some implementations, if no suitable match is found, the system may prompt user input for clarification.

[0076] Once the target object has been identified, step 212 determines the spatial pose of the object using the pose estimation module. The pose estimation module calculates the six-degree-of-freedom pose of the object, including its position and orientation, to enable precise interaction. Algorithms such as point cloud bounding box fitting or foundation models may be used to refine pose estimation, particularly in cases where objects are partially visible, misoriented, or obstructed by other items. The pose data is then provided to the object manipulation module for execution.

[0077] At step 214, the object manipulation module allows the robotic arm of the device being controlled to interact with the target object based on the determined pose. The manipulation may involve grasping, rotating, or repositioning the object using an actuator, such as a robotic arm with a gripper ora suction-based mechanism. In some implementations, the object may be rotated to align identifying features, such as a barcode or label, for verification purposes. Alternative manipulation means may include soft robotic actuators or other end effectors designed to handle delicate or irregularly shaped objects.

[0078] Step 216 involves verifying the target object using the verification module. The verification module may include a barcode scanning component for reading identifying information, such as European Article Numbers (EANs) or other symbologies, and an optical character recognition (OCR) component for extracting alphanumeric data, such as product names, expiry dates, or batch codes. The extracted information is cross-referenced with a stored dataset to confirm the identity or characteristics of the target object. If verification isunsuccessful, the system may return to previous steps, such as segmentation or classification, to identify an alternative candidate or prompt for manual intervention.

[0079] At step 218, the system determines whether the verification process was successful. If the target object is verified, the process proceeds to step 220, where the system updates the stored dataset or digital map to record the object's position, status, or metadata. For example, the digital map may be updated to reflect that an identified product has been placed in the correct location on a shelf. If verification fails, the system may issue an error notification 221, re-attempt identification 222, or prompt a user for manual input 223.

[0080] Variants of this process flow may include additional error handling, iterative refinement loops, or real-time adjustments to accommodate unexpected conditions, such as incomplete product data or dynamic changes in the environment. For instance, the system may implement user-assisted manual alignment if a barcode cannot be located automatically.

[0081] While FIG. 2 depicts a specific flow of operations, it is understood that variations in the sequence or implementation of modules may be applied without deviating from the scope of the invention. For example, the segmentation and classification modules may operate iteratively to refine object identification, or alternative pose estimation algorithms may be used to improve accuracy under different visibility conditions. Similarly, the verification module may incorporate alternative identification techniques, such as RFID readers or NFC systems, in some embodiments. The object manipulation module, while shown with a gripper, may include other actuators, such as suction-based or soft robotic mechanisms, depending on the nature of the objects being handled.

[0082] FIG. 3 illustrates a specific example implementation of the modular architecture of a computer vision system of FIG.l, configured for object identification, classification, pose estimation, and verification.

[0083] The diagram shows the interaction between the various nodes, components, and user inputs to achieve reliable product identification and confirmation.

[0084] The system begins with video input 300, where a visual feed of an environment containing objects, such as representative grocery products, is captured. The input sourcemay include devices such as an RGB camera, depth camera, or similar imaging systems. In the depicted example, a Depth Camera Node 302 receives the video input, processes the visual data, and forwards it to the computer vision node 304 for analysis. The Depth Camera Node 302 may collect RGB image data, depth maps, or a combination thereof to ensure detailed perception of the objects and their positions.

[0085] The Computer Vision Node 304 serves as the primary processing hub, coordinating the data flow between modules and executing core operations. It receives the visual input and interacts with multiple Robotic Operating System 2 (ROS2) nodes to perform specific tasks. The first module linked to the computer vision node is the SAM2 R0S2 Node 306, which performs object segmentation. This module generates segmentation masks for individual objects detected in the input video or image, assigning a unique mask number to each segmented object. The SAM2 ROS2 node 306 enables the system to distinguish between multiple objects, even in cluttered or partially visible scenarios.

[0086] The segmented masks are subsequently passed to the SKU Classifier ROS2 Node 308. The classifier module processes the unmasked segmentation results to identify objects based on visual features. The classification node may incorporate models such as Visual Language Models (VLMs) or embeddings-based search frameworks (e.g., CLIP) to compare segmented masks with stored object descriptors. The classifier outputs the mask corresponding most closely to the user-specified object, as determined by semantic or visual similarity.

[0087] The 6D Pose Estimation ROS2 Node 310 is responsible for performing pose estimation on the identified object. Specifically, the 6D Pose Estimation ROS2 node calculates the six-degree-of-freedom pose of the object, including its spatial position and orientation. This information enables subsequent object manipulation and verification tasks by determining the exact location and alignment of the object within the environment. The 6D Pose Estimation ROS2 node continuously provides updated pose data to ensure reliable interaction with the identified object, especially when the object is partially visible or misaligned.

[0088] The Barcode Scan ROS2 Service Node 312 provides ground truth verification of the identified object. Once the pose estimation step is complete, the barcode scan node is invoked to capture and decode identifying information from the object, such as a barcode orother visual identifiers. The barcode scanning service may use libraries such as pyzbar to extract data, including the EAN, which is cross-referenced with a stored product database to confirm the object's identity. The barcode scan ROS2 service node 312 ensures robust verification, particularly in cases where classification results require further confirmation.

[0089] The figure also depicts a user interaction interface 314, where the system prompts the user to provide input at key stages of the process. For instance, after segmentation, the computer vision node 304 displays the segmented video stream and requests the user to specify which product corresponds to the task prompt. User input may be provided through a graphical user interface (GUI) or command-line interaction, enabling manual selection or refinement when automatic classification yields ambiguous results. Additionally, the user is prompted to assist in aligning the object during the verification step by rotating the object to make the barcode visible.

[0090] The system ultimately outputs confirmation of the identified and verified product, informing the user whether the selected product is correct. In cases where no suitable match is found, the system may generate an error notification, prompting additional input or manual intervention.

[0091] The system also incorporates functionality to verify expiration dates on products as part of its verification pipeline. In the video input 300, the expiration date, denoted as "XXX" on one of the products, is captured by the imaging system. The Depth Camera Node 302 processes this visual data and forwards it to the Computer Vision Node 304 for analysis. The Computer Vision Node 304 uses an Optical Character Recognition (OCR) module to extract text-based information, such as the expiration date, from the product label. The extracted data is parsed and interpreted to identify the specific expiration date, which is then passed to the barcode verification process for additional validation.

[0092] After decoding the expiration date, the system cross-references it with current taskspecific requirements, such as ensuring that the product is not expired or meets a defined freshness criterion. If the expiration date is valid and within the required parameters, the system allows the product to proceed to the next verification stage. In cases where the expiration date is invalid, expired, or missing, the system generates an error notification,alerting the user or prompting the system to discard the product from further processing. This functionality enhances the system's ability to validate products not only based on their identity but also on their usability, ensuring compliance with quality and operational requirements during product handling tasks.

[0093] An example sequence of operations for the system to perform product fronting and placing tasks begins when the system receives a command from a planner or user specifying an operation, such as "Front / Place in Y product X." This command initiates a series of actions for locating, identifying, verifying, and positioning the specified product in its designated location. The system retrieves a location for product X using planogram data, which provides a pre-defined layout of the environment. A general location is identified on a two-dimensional map as an initial shelf guess. The system then performs two-dimensional navigation to move to the approximate area of the shelf location using navigation methods such as path-planning algorithms, simultaneous localization and mapping (SLAM), or stored map data. After navigating to the general location, the system refines its positioning to align with a specific shelf using three-dimensional localization techniques for greater precision.

[0094] The system verifies its position by scanning the shelf barcode using a barcode scanning module. The barcode scan provides ground truth confirmation of the correct shelf location. If the barcode is correct, the system proceeds to process the image of the shelf. If the barcode is not correct, an error message is generated indicating that the shelf could not be found, and the process may terminate until the planogram data is updated. Once the shelf is confirmed, the system begins segmentation of the shelf image using a segmentation module. The segmentation process generates a list of masks Z, each corresponding to a segmented product in the image. Segmentation may employ machine learning models, such as Segment Anything Model 2 (SAM2), capable of accurately isolating individual objects, including in cluttered or overlapping conditions.

[0095] The system then classifies the segmented masks to identify which mask z E Z corresponds to the target product X. The classification module processes the unmasked segmentation results using visual classifiers, such as visual language models (VLMs) or embeddings-based search frameworks like CLIP. The classification step compares the segmented masks to stored descriptors for product X, producing an initial product guess formask z. A decision is then made regarding the selected mask z. If the classification module determines that mask z corresponds to product X, the process continues to pose estimation. If the selected product does not match, the system deletes mask z from the list Z and determines whether any masks remain in Z. If Z is not empty, the system loops back to reanalyze the remaining masks. If Z is empty, the system generates an error message indicating that the product could not be found and may require an update to the product list, terminating the process.

[0096] Once the product is identified, the system performs 6D pose estimation using a pose estimation module. This step determines the spatial position and orientation of the product represented by mask z. Pose estimation calculations may account for partial visibility, occlusions, or objects that are misaligned. Using the pose data, the system interacts with the product through an object manipulation module. The module picks up the product and, in some cases, rotates or navigates around it to locate a barcode for verification.

[0097] The barcode scanning module then validates the product identity by reading the barcode and comparing it to stored product data. This step provides ground truth confirmation that the selected product matches product X. If the barcode verification is successful, the system proceeds to place or front the product in the designated location Y. The system ensures that the product is positioned accurately according to predefined criteria, such as alignment with the front edge of the shelf or a specified arrangement. If the barcode verification fails, the system deletes the mask z and returns to analyze the remaining masks in Z. If no masks remain, the system generates an error message and terminates the process.

[0098] The sequence of operations concludes successfully when the system verifies and places product X in location Y. If at any point the system encounters an error, such as failure to locate the shelf, identify the product, or verify the barcode, an appropriate error message is generated, and the operation may terminate. This example demonstrates a structured workflow that integrates segmentation, classification, iterative refinement of mask z, pose estimation, barcode validation, and object placement to ensure accurate and reliable fronting or placement of products.

[0099] Although described in a specific sequence herein, the operations of the system may in various embodiments be performed in other suitable orders. For example, the system may perform segmentation first and then classification; in other embodiments, classification, segmentation and pose estimation may be performed simultaneously, or classification may precede segmentation.

[0100] Furthermore, the use of the term "modules" is used herein for clarity to define the various functions described herein. It will be understood by a person of ordinary skill that the functions of a multiple modules may in practice be performed by a single computational entity, model, or component. Similarly, the functions of one module may in practice be divided across multiple computational entities, models or components. For example, the classification, segmentation, and pose-estimation are performed in a single end-to-end neural network.CONTROLLER / PROCESSOR COMPONENTS

[0101] A processor or controller as described herein may include any suitable type of computing device, such as a central processing unit (CPU), microcontroller, graphics processing unit (GPU), system on a chip (SoC), or digital signal processor (DSP). It may operate with one or more cores and may be configured to execute the functions described in this disclosure.

[0102] The processor may be operably connected to one or more memory devices, such as random access memory (RAM), read-only memory (ROM), flash storage, or solid-state drives (SSD). These memory devices store computer-readable instructions that, when executed by the processor, perform the methods described. The processor and memory communicate via data buses or other suitable communication pathways.

[0103] The computing device may also include input / output (I / O) devices, such as a touchscreen, mouse, keyboard, display, or speaker, to facilitate interaction with users or other systems. Additionally, it may include a network interface, such as a wired or wireless communication module, for connecting to networks.

[0104] Control logic or software instructions may be stored in memory and executed by the processor to implement specific functionalities. This logic may be modular, consisting of software components, processes, or functions that work together to perform the operations described herein.

[0105] The described computing operations involve the manipulation of data represented as electrical, optical, or magnetic signals stored or transferred within the system. These operations are machine-executed and do not require manual intervention, though they may interface with human operators through appropriate user interfaces.

[0106] The systems and methods described are not limited to any particular hardware configuration or programming language and may be implemented on general-purpose or specialized computing devices.CONCLUSION

[0107] Unless otherwise defined, all terms (including technical terms) used herein have the same meaning as commonly understood by one having ordinary skill in the art to which this invention belongs. It will be further understood that terms, such as those defined in commonly used dictionaries, should be interpreted as having a meaning that is consistent with their meaning in the context of the relevant art and the present disclosure and will not be interpreted in an idealized or overly formal sense unless expressly so defined herein.

[0108] The disclosed embodiments are illustrative, not restrictive. While specific configurations of the system and methods of the invention have been described in a specific manner referring to the illustrated embodiments, it is understood that the present invention can be applied to a wide variety of solutions which fit within the scope and spirit of the claims. There are many alternative ways of implementing the invention.

[0109] It is to be understood that the embodiments of the invention herein described are merely illustrative of the application of the principles of the invention. Reference herein to details of the illustrated embodiments is not intended to limit the scope of the claims, which themselves recite those features regarded as essential to the invention.

Claims

ClaimsWhat is claimed is:

1. A system for identifying, verifying, and manipulating objects, the system comprising:a navigation module configured to navigate to a target location based on stored location data;a segmentation module configured to generate a plurality of segmented regions from an input image, each segmented region corresponding to a distinct object in the image;a classification module operably connected to the segmentation module, the classification module configured to receive the segmented regions and determine, for each segmented region, a classification output corresponding to a target object; a pose estimation module configured to determine a spatial pose of the target object based on the segmented region associated with the classification output; an object manipulation module configured to interact with the target object using the spatial pose determined by the pose estimation module;a verification module configured to verify the target object by capturing identifying information from the target object; anda control unit operably connected to the navigation module, the segmentation module, the classification module, the pose estimation module, the object manipulation module, and the verification module, the control unit configured to coordinate operation of the system.

2. The system of claim 1, wherein the classification module comprises a visual language model configured to interpret an input prompt and prioritize one or more segmented regions based on semantic similarity between the input prompt and visual features associated with the segmented regions; and wherein the classification module further includes an embeddings-based search model configured to compare the segmented regions with a database of object descriptors.

3. The system of claim 1, wherein the segmentation module comprises a machine learning-based segmentation model configured to generate masks corresponding to individual objects within the input image.

4. The system of claim 1, wherein the pose estimation module is configured to determine a six-degree-of-freedom pose of the target object, the pose estimation module further configured to refine the pose estimation based on partial visibility or occlusions of the target object.

5. The system of claim 1, wherein the verification module comprises:a barcode scanning module configured to read a barcode on the target object to extract identifying data; andan optical character recognition (OCR) module configured to extract alphanumeric information from the target object; andwherein the verification module is configured to verify the target object by cross-referencing the identifying data and the alphanumeric information with a stored dataset; andwherein the barcode scanning module comprises a barcode reader configured to identify one or more barcode symbologies, including linear and two-dimensional barcode formats; andwherein the verification module is configured to verify product-specific information selected from the group consisting of: product identity, expiry date, and batch code.

6. The system of claim 1, wherein the object manipulation module is configured to manipulate the target object by rotating the target object to align identifying features with the verification module.

7. The system of claim 1, wherein the navigation module comprises a two-dimensional navigation system for navigating to a general location and a three-dimensional navigation system for navigating to a specific shelf or subregion within the target location.

8. The system of claim 1, wherein the segmentation module, classification module, and pose estimation module are configured to operate iteratively to identify and refine the classification output of the target object.

9. The system of claim 1, wherein the control unit is further configured to update a digital map with information corresponding to the position and identification of the target object.

10. The system of claim 1, wherein the control unit is configured to prompt a user to manually interact with the target object in response to an unsuccessful verification by the verification module.

11. The system of claim 1, wherein the segmentation module and classification module are configured to handle partial object visibility by analyzing portions of the target object that are not occluded.

12. The system of claim 1, wherein the classification module is further configured to determine whether any of the segmented regions fail to match the target object and provide an error notification.

13. The system of claim 1, wherein the navigation module is configured to utilize planogram data to determine an initial search space for the target object.

14. The system of claim 1, wherein the object manipulation module comprisesan actuator configured to execute grasping, picking, and placing operations based on the spatial pose of the target object.

15. The system of claim 1, wherein the classification module further includes a logic processing unit configured to handle partial or ambiguous prompts by dynamically interpreting task-specific context.

Images