Adaptive robotic learning system with privacy-preserving data processing

The method adapts robotic models to diverse environments and user preferences through user feedback and privacy-preserving data processing, enhancing adaptability and privacy in robotic task performance.

WO2026236279A1PCT designated stage Publication Date: 2026-11-12CLUTTERBOT INC
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
PCT/US2026/026769
Authority / Receiving Office
WO · WO
Patent Type
Applications
Current Assignee / Owner
Priority Date
2025-05-05
Filing Date
2026-05-05
Publication Date
2026-11-12

AI Technical Summary

Technical Problem

Conventional robotic solutions lack adaptability to diverse environments and user-defined preferences, and fail to safeguard privacy and provide reliable user feedback for task performance.

Method used

A computer-implemented method for refining or personalizing a robot task control model using local and cloud-based models, incorporating user feedback, and employing privacy-preserving data processing techniques.

Benefits of technology

Enhances robotic adaptability and privacy protection while allowing for efficient model refinement or personalization based on user input, improving task performance and user satisfaction.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure US2026026769_12112026_PF_FP_ABST
    Figure US2026026769_12112026_PF_FP_ABST
Patent Text Reader

Abstract

A method and system for adapting a robot task control model to improve its operation in an intended environment with intended tasks. The system receives a user report through a user interface, which includes text data describing a failed robotic task and a desired robot behavior. The system then retrieves relevant sensor frames from a stored dataset, processes these frames using local and cloud-based task control models to generate confidence values, and refines the local model based on performance. If the cloud-based model's confidence values exceed a threshold, the system refines the local model by training candidate models and selecting the best-performing candidate. If the cloud-based model's confidence values fail to meet the threshold, the system personalizes the local model by training candidate models on new classes and selecting the best-performing candidate. The refined or personalized model is then deployed to the robot.
Need to check novelty before this filing date? Find Prior Art

Description

Docket No. FSP2375PCTADAPTIVE ROBOTIC LEARNING SYSTEM WITH PRIVACY-PRESERVING DATA PROCESSINGCROSS-REFERENCE TO RELATED APPLICATIONS

[0001] This application claims benefit and priority to Foreign National provisional application serial no. 202541043530, filed in India on May 5, 2025, titled ADAPTIVE ROBOTIC LEARNING SYSTEM FOR HOME ENVIRONMENTS WITH PRIVACY-PRESERVING DATA PROCESSING, which is incorporated herein by reference in its entirety.BACKGROUND

[0002] Robotic solutions for automating task performance are increasingly of interest in multiple contexts. Household tidying and cleaning robots that do more than simply vacuum a floor are becoming increasingly capable, both in the tasks they perform and the granularity of detail they recognize in performing those tasks. Similarly configured robots are also becoming increasingly useful in individual and shared commercial environments, such as movie theaters, food courts, stadiums, hotels, and transit terminals.

[0003] Conventional task performance robotic solutions often rely on a single global object recognition model. Such models fail to adapt to special task target objects (such as toys, shoes, clothing, etc., in a household context, or handheld point of sale devices, unique merchandise, company-differentiated equipment, etc. in individual or shared commercial spaces, and so on). Conventional models also perform weakly in supporting user-defined preferences (e.g., pushing shoes against a wall, picking up food trays for a single specific food court vender, etc.).

[0004] Conventional solutions also fail to reliably safeguard the privacy of persons and data within a robot's sensed operational environment. Capturing raw video, as well as lidar and depth data, as part of robot training, planning, and operation, has the potential to reveal personal or sensitive content when solutions lack robust safeguarding measures. This poses even greater problems for robots operating in a public area where identifying, personal, and private attributes of hundreds of people may be captured during a single day of operation.

[0005] Conventional frameworks also lack a direct feedback method for users to address failures (e.g., “the robot doesn’t pick up these plush toys” or “the robot threw away an abandoned smartphone”) that automatically modifies the model based on a user's complaint or request. There is, therefore, a need for task performance robotic solution adaptable to a broadDocket No. FSP2375PCTarray of operational environments and use cases, that may automatically incorporate feedback provided by a user while reliably safeguarding the privacy of persons within the operational environment.BRIEF SUMMARY

[0006] A computer-implemented method is disclosed for adapting a robot task control model. The method involves receiving a user report through a user interface, retrieving relevant sensor frames from a stored dataset, and processing these frames using a local and cloud-based task control model. The method compares the classification outputs from both models and refines the local model based on the results. If the cloud-based model performs better, the local model is refined or personalized based on user feedback. The refined or personalized model is then deployed to the robot. In the refinement process, multiple candidate models are trained using the relevant sensor frames and a global dataset. In the personalization process, new candidate models are trained on a specific class using user-labeled example sensor frames. The bestperforming candidate model for refinement or personalization is then selected and the refined or personalized model deployed to the robot.

[0007] Also disclosed is a robotic system configured to execute the disclosed method, the system comprising a user interface and a robot. The robot is equipped with a sensing system including at least one camera. The robot is further equipped with a robot memory storing sensor frames captured by the sensing system, associated vector embeddings extracted from the sensor frames, and a local robot task control model. The robot also includes a robot data processor configured to execute the local robot task control model. The robotic system further comprises a cloud-based data processor configured to receive data from the robot data processor, a cloudbased teacher robot task control model, a robotic control system in at least one of the robot and a cloud server, and a task control model adaptation controller. The task control model adaptation controller is configured to receive input from the user interface, the robot memory, the local robot task control model, the robot data processor, the cloud-based teacher robot task control model, and the cloud-based data processor, to support improved operation of the robot in an intended environment, with intended tasks. The task control model adaptation controller is configured with logic, to perform the disclosed method.BRIEF DESCRIPTION OF THE SEVERAL VIEWS OF THE DRAWINGSDocket No. FSP2375PCT

[0008] To easily identify the discussion of any particular element or act, the most significant digit or digits in a reference number refer to the figure number in which that element is first introduced.

[0009] FIG. 1 illustrates a system architecture 100 in accordance with one embodiment.

[0010] FIG. 2 illustrates a report and recommendation routine 200 in accordance with one embodiment.

[0011] FIG. 3 illustrates a data harvest routine 300 in accordance with one embodiment.

[0012] FIG. 4 illustrates a preliminary data sampling routine 400 in accordance with one embodiment.

[0013] FIG. 5 illustrates a data sampling and deduplication routine 500 in accordance with one embodiment.

[0014] FIG. 6 illustrates a model deployment routine 600 in accordance with one embodiment.

[0015] FIGs. 7A-7B illustrate a side elevation view and a top plan view of the robot 700 as illustrated in FIG. 1, respectively.

[0016] FIG. 8 illustrates a simplified plan view of the bottom of a chassis 702 of the robot 700, illustrating details of a mobility system 704 in accordance with one embodiment.

[0017] FIGs. 9A-9B illustrate a simplified side view and top view of a chassis 702 of the robot 700, respectively, illustrating details of a sensing system 706 in accordance with one embodiment.

[0018] FIG. 10 illustrates robotic control system interfaces 1000 in accordance with one embodiment.

[0019] FIG. 11 illustrates a perception logic 1100 in accordance with one embodiment.

[0020] FIG. 12 illustrates a navigation logic 1200 in accordance with one embodiment.

[0021] FIG. 13 illustrates a robot brain 1300 in accordance with one embodiment.

[0022] FIG. 14 illustrates an action logic 1400 in accordance with one embodiment.

[0023] FIG. 15A and FIG. 15B illustrate a robot brain flow diagram 1500 in accordance with one embodiment.

[0024] FIGs. 16-28 provide exemplary visual representations of data that is input to, processed in, and output by the perception logic 1100 and navigation logic 1200 embodiments herein.Docket No. FSP2375PCT

[0025] FIG. 29 illustrates an embodiment of a robotic control system 2900 to implement components and process steps of the system described herein.

[0026] FIG. 30 illustrates sensor input analysis 3000 in accordance with one embodiment.

[0027] FIG. 31 illustrates an image processing routine 3100 in accordance with one embodiment.

[0028] FIG. 32 illustrates a video-feed segmentation routine 3200 in accordance with one embodiment.

[0029] FIG. 33 illustrates a system environment 3300 in accordance with one embodiment.

[0030] FIG. 34 illustrates a computing environment 3400 in accordance with one embodiment.

[0031] FIG. 35 illustrates a set of functional abstraction layers 3500 in accordance with one embodiment.

[0032] FIG. 36 illustrates an artificial intelligence or Al system 3600 in accordance with one embodiment.

[0033] FIG. 37 illustrates an exemplary tokenizer 3700 in accordance with one embodiment.

[0034] FIG. 38 illustrates a deep neural network 3800 in accordance with one embodiment.

[0035] FIG. 39A illustrates inference and / or training logic 3900a in accordance with one embodiment.

[0036] FIG. 39B illustrates inference and / or training logic 3900b in accordance with one embodiment.

[0037] FIG. 40 depicts a convolutional neural network 4000 in accordance with one embodiment.

[0038] FIG. 41 depicts a convolutional neural network layers 4100 in accordance with one embodiment.

[0039] FIG. 42 depicts a VGG network 4200 architecture known as VGG16 in one embodiment.

[0040] FIG. 43 depicts a VGG net 4300 in accordance with one embodiment.

[0041] FIGs. 44A-44B illustrate convolution layer filtering 4400 in accordance with one embodiment.

[0042] FIG. 45 illustrates a pooling layer function 4500 in accordance with one embodiment.Docket No. FSP2375PCT

[0043] FIG. 46 illustrates a basic deep neural network 4600 in accordance with one embodiment.

[0044] FIG. 47 illustrates an artificial neuron 4700 in accordance with one embodiment.DETAILED DESCRIPTION

[0045] This disclosure is generally directed to feedback for model updates in home robotics, including user-input components. In one embodiment, the disclosed solution dynamically improves the flexibility and power of tidying robot operation while optimizing and improving usage of robotic memory and processing hardware, providing a technical improvement over conventional tidying robot task control model deployment solutions. Privacy -preserving multimodal recognition based on data from visual sensors (such as red, green, and blue (RGB) cameras) and depth sensors (such as light detecting and ranging (LiDAR) sensors may be employed. Cloud-based data sampling and deduplication may optimize data for embedded hardware constraints. Adaptive Al capable of detection or segmentation may be used for improved household object recognition. Real-time customization for novel, user-specified objects or behaviors may be supported by the disclosed solution.

[0046] The disclosed system uses smart recommendation to distinguish between problems with existing classes versus a need for entirely new classes / behaviors. On-device redaction or removal of personal / sensitive data may be deployed. Cloud-based advanced data sampling and deduplication may be used to avoid overloading local hardware. The disclosed solution allows either refinement of an existing detection / segmentation model (model refinement) or personalization of the model with new classes (model personalization) based on user needs.

[0047] In various illustrative embodiments, the disclosure is related to issue reporting and smart recommendations in response, data harvest where logged data is not adequate to determine a recommendation, cloud-based data sampling and deduplication to improve privacy and reduce resource needs, and model refinement or personalization to provide solutions to reported issues.

[0048] Users may submit complaints or new behavior requests through a user interface provided on a computing device, such as a desktop computer, laptop computer, or mobile device (smartphone, tablet, etc.), optionally attaching photos. The system may retrieve relevant frames from the robot’s image logs and compares local model outputs with those of a superlarge teacher model, which may be configured in a cloud computing environment. TheDocket No. FSP2375PCTdisclosed system may decide whether to apply model refinement (improve existing classes) or model personalization (introduce new classes / behaviors).

[0049] If more data is needed, data harvest may be employed to gather RGB, depth, lidar, and odometry data from the robot. The data harvest process may include conducting sampling at 1 frame per second and performing basic quality checks on collected image data. It may also filter personal and sensitive data using recognition-based privacy algorithms, providing a more secure transition to cloud-based data processing of image and sensor data capture by the robot.

[0050] For cloud-based data sampling and deduplication, privacy-cleaned data is uploaded securely from the robot to cloud-based repositories and processing platforms. Advanced central processing unit (CPU) and / or graphics processing unit (GPU) resources may run this data thorough quality checks, near-duplicate removal, and final dataset assembly.

[0051] In model refinement, where underperforming existing classes are detected, newly collected data may be merged with a large global dataset. Multiple candidatedetection / segmentation models may be trained and compared to find the best candidate for deployment. This process may use incremental learning to reduce catastrophic forgetting. The best performing model may be deployed to the robot.

[0052] In model personalization, few-shot or specialized training may be used to integrate a set of user-labeled example sensor frames and potentially global references. For new classes or custom behaviors determined to address a user's reported issue. In this manner, the disclosed solution may produce a new detection head or segmentation label plus user-defined handling rules (e.g., “push shoes” or “treat fragile items gently”) quickly and efficiently.

[0053] Model refinement and model personalization may be configured to not run simultaneously, but rather one or the other may be recommended. Smart recommendations may pick the relevant pipeline based on whether elements indicated in the user issue and related data belong to an existing or a new category.

[0054] The disclosed system may be a complete framework for smart home robots, designed to automate object handling and user-defined behaviors. This system may include issue reporting and smart recommendation, allowing users to report problems or requests through a mobile interface and attach optional photos. The system may then search the robot’s local image logs for matching scenes and compare outputs from the local model with those provided by a super-large teacher model evaluating the same data. It decides whether to Adapt theDocket No. FSP2375PCTexisting model (model refinement) or Personalize it with new classes / behaviors (model personalization). The model may be a detection or semantic segmentation network.

[0055] The disclosed system may further support data harvest. This may include on-device data collection (RGB, depth, LiDAR, odometry), preliminary sampling, and recognition-based privacy filtering. The output from the data harvest may feed a cloud-based data sampling and deduplication algorithm. This may be operated off-device using advanced sampling, deduplication, and dataset curation techniques that may use too many resources to be effectively deployed on-device.

[0056] If the user’s reported issue relates to an existing class performing poorly, the pipeline may refine the model. If the user’s request involves a new class or custom behavior, a few-shot or specialized learning pipeline is invoked to personalize the model. All stages may incorporate strict privacy measures, robust data processing, and user-friendly feedback loops for smart recommendations to decide whether to refine or personalize the robot’s model. The underlying model may support both bounding-box detection and semantic segmentation outputs.

[0057] The disclosed solution may operate upon and deploy multiple trained Al models. Detection logic may be used to generate bounding boxes and class labels for objects represented in image frames captured by robot cameras. In some embodiments, the detection logic may use a You Only Look Once (YOLO) computer vision algorithm or a two-stage deep learning algorithm for region-based convolutional neural networks (R-CNNs), such as Faster R-CNN. A semantic segmentation model may be used to produce pixel-level masks and class labels. In various embodiments, the segmentation model may be DeepLab deep learning model or a two-stage framework such as Mask R-CNN. The disclosed system may also accommodate hybrid, multi-task networks combining bounding boxes and segmentation heads.

[0058] FIG. 1 illustrates a system architecture 100 in accordance with one embodiment. The system architecture 100 may comprise a user interface 120, a robot data log 104, a local robot task control model 102, a cloud-based teacher robot task control model 106, a task control model adaptation controller 108, a recommendation logic 110, a refinement logic 112, a personalization logic 114, a robot data processor 116, a cloud-based data processor 118. The system architecture 100 may further include a robot 700 such as is described in additional detail with respect to FIGs. 7A-9B.

[0059] " Logic" refers to machine memory circuits and non-transitory machine readable media comprising machine-executable instructions (software and firmware), and / or circuitryDocket No. FSP2375PCT(hardware) which by way of its material and / or material-energy configuration comprises control and / or procedural signals, and / or settings and values (such as resistance, impedance, capacitance, inductance, current / voltage ratings, etc.), that may be applied to influence the operation of a device. Magnetic media, electronic circuits, electrical and optical memory (both volatile and nonvolatile), and firmware are examples of logic. Logic specifically excludes pure signals or software per se (however does not exclude machine memories comprising software and thereby forming configurations of matter).

[0060] The local robot task control model 102 may be a machine learning model, artificial intelligence (Al) model, generative Al (genAI) model, or similar computational structure and algorithms designed to support the robot 700 in autonomous operation to complete desired tasks. The local robot task control model 102 may reside, in whole or in part, on processing hardware physically located within the robot's electronic and computational systems. The local robot task control model 102 may reside in whole or in part on a cloud server available through the robot's network interface. The computational capabilities of the robot 700 may be supported by a robotic control system 2900 described in additional detail with respect to FIG. 29.

[0061] The robot data log 104 may be stored in memory structures and hardware such as are well known to those of ordinary skill in the art. These may include file systems, databases, etc., in physical memory on the robot 700, in cloud-based memory, etc. robot data log 104 information may include sensor data such as camera images, video, lidar data, etc., captured by the robot 700 or by a mobile device in communication with the robot. In one embodiment, the robot data log 104 may include computational data developed from the sensor data in support of robot 700 operation, etc. Data from the robot data log 104 may undergo processing, such as filtering, anonymization, and other steps conducive to efficient and secure data handling. This may be performed by the robot data processor 116 either before or after initial storage in the robot data log 104. The data from the robot data log 104 may further undergo processing by the cloud-based data processor 118 in support of improvements provided by the disclosed system. Data processing by the robot data processor 116 and cloud-based data processor 118 is describe in additional detail with respect to the data harvest routine 300 of FIG. 3 and the data sampling and deduplication routine 500 of FIG. 5. In the disclosed embodiments, all sensor frames stored in the stored dataset of the robot data log 104 for retrieval by the similarity search may have undergone the quality-filtering and privacy-cleaning steps of preliminary data sampling routineDocket No. FSP2375PCT400 (FIG. 4), regardless of whether the frames originate from a data harvest session, a training session, or standard robot operations.

[0062] The cloud-based teacher robot task control model 106 may be a machine learning model, an Al model, a GenAI model, or similar computational structure and algorithms designed to support the robot 700 in autonomous operations, similar to the local robot task control model 102. The cloud-based teacher robot task control model 106 may be a larger, more generalized model including more information and able to support a broader range of operations. It may support a superset of robots 700 whose local robot task control models 102 are lighter and more specialized to their specific tasks and environments. The cloud-based teacher robot task control model 106 model may reside on a cloud server. The cloud server may be accessible to elements of the system via a network interface. The cloud server may be part of a system environment 3300 such as is described in additional detail with respect to FIG. 33.

[0063] The task control model adaptation controller 108 may include the recommendation logic 110, refinement logic 112, and personalization logic 114. The task control model adaptation controller 108 may be a machine learning, Al, or GenAI algorithm configured as described herein to support improved operation of the robot 700 in its intended environment, with its intended tasks. The task control model adaptation controller 108 may reside in whole or in part on processing hardware physically integrated within the electronic, computational, and control systems of the robot 700. The task control model adaptation controller 108 may in one embodiment reside in whole or in part in a cloud environment in network communication with the robot 700 and other elements of the system architecture 100.

[0064] The user interface 120 may be configured on the computing device 122 such as a mobile device Such computing devices or apparatuses may include a smartphone or tablet, a laptop computer, a desktop computer, a control interface designed for human interaction integrated within the robot 700 itself, or other hardware accessible to a user 124 and in communication with elements of the disclosed robotic system. The robot 700 may be trained and operated within a robot's working space 126. The robot's working space 126 may be one or more rooms of a private home in one embodiment. In one embodiment, the robot's working space 126 may be an individual or shared commercial space, such as a mall food court, a restaurant, a stadium or arena, a movie theater, etc. The user interface 120 may accept a user report 128 provided by the user 124. The user interface 120 may in one embodiment support manual operation 130 of the robot 700 by the user 124. The user 124 may interact with the userDocket No. FSP2375PCTinterface 120 from a location within the robot's working space 132 in one embodiment. In one embodiment, the user 124 may interact with the user interface 120 from a location remote from the robot's working space 134. This may be accomplished through network connectivity of the computing device 122 with elements of the robot 700, the robotic control system 2900 described with respect to FIG. 29, and other elements of the system architecture 100, as will be readily understood by one of ordinary skill in the art.

[0065] These elements of the system architecture 100 may be configured to perform the report and recommendation routine 200, data harvest routine 300, data sampling and deduplication routine 500, model deployment routine 600, and other routines and actions as described herein.Issue Reporting & Smart Recommendation

[0066] FIG. 2 illustrates a report and recommendation routine 200 in accordance with one embodiment. The objective of the report and recommendation routine 200 may be to provide a direct feedback method allowing users to report issues with or request new behaviors from the robot 700 using the system architecture 100 introduced with respect to FIG. 1. This system architecture 100 may automatically find frames in the robot’s logs that match the user’s reported issue, complaint, or request, then compare local model outputs with teacher model outputs under those use conditions. The system architecture 100 may determine whether the object or behavior under consideration is an existing class, in which case the system architecture 100 may perform model refinement, or a new class, where model personalization may be performed.

[0067] Although a particular sequence of operations is illustrated for report and recommendation routine 200, the sequence may be altered without departing from the scope of the present disclosure. For example, some of the operations depicted may be performed in parallel or in a different sequence that does not materially affect the function of the report and recommendation routine 200. In other examples, different components of an example device or system that implements the report and recommendation routine 200 may perform functions at substantially the same time or in a specific sequence.

[0068] At block 202, the user report may be received by the system architecture 100 via the user interface 120, with which a user may interact from a location within the robot's working space or a location remote from the robot's working space, using a computing device 122 such as a mobile device as previously described. The user report may include text data, i.e., textDocket No. FSP2375PCTtyped into the user interface 120 by the user, image data, and other sensor data, i.e., an optional photo taken and uploaded to the user interface 120 by the user. The text data and image data or sensor data, in the form of sensor frames, provided by the user may indicate use conditions that are of interest to the user. Use conditions may include objects the user wishes the robot to act upon, objects in the vicinity that may impact the robot's actions, starting location indications, and target locations for the objects to be acted upon. The user may enter text describing an issue with the robot's behavior under the reported use conditions, such as a failed robotic task, or a new desired robot behavior the user desires the robot to exhibit under similar use conditions. For example, the user may include an image of an object or objects in an environment to be tidied and may indicate an object the robot fails to handle or request a new action the user wishes to be performed with respect to an indicated object. In one embodiment, the image may be of a pair of shoes the robot has put in a bin instead of upon a shelf beside the bin. The text data may indicate the user's wish that shoes are to be placed on the shelf when the robot encounters them during tidying. In one embodiment, the user report may include data captured during a failed robotic task, as well as data captured as the user manually operates the robot using the user interface through a successful performance of the task.

[0069] The system architecture 100 may log the user report data at block 204. The user report data may be stored as a user report record along with a user identification (ID) and timestamp. Storage may be in robot memory or cloud-based memory, as will be readily understood by one of ordinary skill in the art. In one embodiment, the user report may be stored in the robot data log 104. The user report, particularly when it contains sensor frame, may be processed with a multi-dimensional vision transformer, such as a 512-dimensional vision transformer, to develop vector embeddings associated with and descriptive of each individual sensor frame. These vector embeddings may be indexed and stored alongside the sensor frames in robot memory, cloud-based storage, or a combination thereof. In one embodiment, the user report may be text data alone. In one embodiment, the user report may also include one or more images or sensor frames provided by the user. The user-provided sensor frames may have a multi-dimensional vision transformer applied to extract user-provided sensor frame vector embeddings associated with the user-provided sensor frames. The user-provided sensor frame vector embeddings may be included in the similarity search described with respect to block 206.

[0070] In one embodiment, the user report may include data related to robot actions taken as part of a failed robotic task, as well as data captured as the user uses the user interface toDocket No. FSP2375PCTmanually operate the robot through successful completion of the task. In this embodiment, a report and recommendation routine 200 may include capturing, using at least one camera configured as a sensing system of the robot, failed robotic task actions as failure sensor frames. The user may manually operate the robot through the user interface 120, to complete the failed robotic task successfully as a successful robotic task. The sensing system having at least one camera may capture successful robotic task actions as success sensor frames, wherein the user report further includes the failure sensor frames and the success sensor frames. The sensing system may also capture depth data and lidar data, which may be included along with image data as the sensor frames. A multi-dimensional vision transformer may be used to extract failure sensor frame vector embeddings and success sensor frame vector embeddings. The failure sensor frame vector embeddings and the success sensor frame vector embeddings may be included in the similarity search described with respect to block 206.

[0071] At block 206, the system architecture 100 may perform a robot data log 104 search. The robot data log 104 may store image logs containing past image frames and / or sensor frames saved during previous data harvest sessions robot training sessions, or standard robot operations. The system architecture 100 may retrieve sensor frames and associated vector embeddings from a stored dataset within the robot data log 104 or other robot memory. The data retrieved may be a relevant subset of sensor frames, their relevance based on a similarity index determined through comparison with the use conditions indicated by the text data and image data / sensor data provided in block 202. In one embodiment, the top 15 frames with an embedding score > 0.75 may be returned as similar frames. The robot may keep an on-device database of multi-dimensional embeddings for each stored frame. In one embodiment, the embeddings may be 512-dimensional embeddings. An Approximate Nearest Neighbor (ANN) search (using Facebook Al Similarity Search (FAISS) with an Angular distance metric) may be used to retrieve the top 15 frames with an embedding similarity score above 0.75. These frames may be expected to contain objects and use conditions matching those in the user report. For example, the frames may all include shoes. Some frames may indicate shoes in the bin near the shelf in question.

[0072] Local model processing and analysis may be performed at block 208 by the robot data processor 116 through model inference by the local robot task control model 102. Each retrieved similar frame of the relevant subset of sensor frames may be run through local object detection or segmentation inference by the local robot task control model 102. For objectDocket No. FSP2375PCTdetection, bounding boxes may be generated with associated classes and confidence, comprising first classification outputs and first confidence values. For segmentation, class-labeled masks may be generated as the first classification outputs having first confidence values. Bounding boxes or segmentation masks having a confidence score below 0.30 may be discarded as “unrecognized” in one embodiment. At block 210, this data for detected objects may be extracted by the local robot task control model 102. Detected objects may include objects categorized as shoes, bins, and shelves.

[0073] Cloud-based teacher model processing may also be performed on the similar frames of the relevant subset of sensor frames at block 212 by the cloud-based data processor 118 running a cloud-based teacher robot task control model 106. The cloud-based teacher robot task control model 106 may output bounding boxes / masks with confidence scores as second classification outputs and second confidence values, similar to the analysis made by the local robot task control model 102. The cloud-based teacher robot task control model 106 may be a much larger and more comprehensively trained model than the local robot task control model 102, taking advantage of the greater memory and higher processing speeds that may be provided by cloud computing environments. The cloud-based teacher robot task control model 106 may generate bounding boxes or masks classified with confidence scores, and may discard those having confidence scores below a desired threshold in one embodiment.

[0074] At block 214, comparison between first classification outputs (local model outputs) and second classification outputs (teacher model outputs) along with the respective first and second confidence values may be performed. In one embodiment, this comparison and confidence scoring may be performed by the cloud-based teacher robot task control model 106. The cloud-based teacher robot task control model 106 may take in the data extracted by the local robot task control model 102 at block 210. The local robot task control model 102 detection / segmentation outputs may be compared to the labels or masks generated by the cloudbased teacher robot task control model 106. For example, if the cloud-based teacher robot task control model 106 consistently identifies an object as “shoe,” “toy,” or another known category while the local robot task control model 102 fails, that indicates underperformance by the local robot task control model 102 on an existing class.

[0075] At decision block 216, comparison outputs may be sent to the recommendation logic 110 of the task control model adaptation controller 108. If additional sensor frames are needed to generate at least one of the first classification outputs, the first confidence values, the secondDocket No. FSP2375PCTclassification outputs, and the second confidence values as determined by the recommendation logic 110, the report and recommendation routine 200 may proceed to the data harvest routine 300, to be followed by the preliminary data sampling routine 400 and data sampling and deduplication routine 500. The additional sensor frames may be determined to be needed when a count of sensor frames in the relevant subset is below a coverage frame-count threshold in one embodiment. In one embodiment, the additional sensor frame may be determined to be needed when at least one of the first confidence values and the second confidence values is below a coverage confidence threshold. Additional sensor frames may be determined to be needed when the relevant subset of sensor frames provides insufficient frame diversity for at least one classification category, as measured by an embedding-based diversity metric below a diversity threshold. In addition or alternatively, a data harvest request may be received through the user interface, indicating a need for the additional sensor frames. In one embodiment, a user may manually complete a failed robotic task through the user interface following an unsuccessful attempt by the local robot task control model. This tele-operation may be used as an initiation of the data harvest process.

[0076] If additional sensor frames are determined at decision block 216 to not be needed, the recommendation logic 110 may direct the system architecture 100 to proceed to the model deployment routine 600, which may be performed by the refinement logic 112 and / or personalization logic 114 as deemed appropriate by the recommendation logic 110. The local robot task control model 102 and cloud-based teacher robot task control model 106 may provide their outputs to the task control model adaptation controller 108 and recommendation logic 110 for use in making this determination.

[0077] The system architecture 100 may be configured to inform the user which actions are triggered (data harvest, model refinement, or model personalization). The selected action steps (model refinement or model personalization) may update the robot’s model with the refined configuration of existing classes or to introduce a new class for personalization. If the alignment of the cloud-based teacher robot task control model 106 with a known class is high, but the local robot task control model 102 fails or underperforms for known classes, model refinement actions may be recommended. If the classes indicated are unknown to the cloudbased teacher robot task control model 106 and local robot task control model 102, model personalization actions may be recommended. This is described in additional detail below with respect to the model deployment routine 600 of FIG. 6.Docket No. FSP2375PCTData harvest: Intelligent Data Collection

[0078] FIG. 3 illustrates a data harvest routine 300 in accordance with one embodiment. The data harvest routine 300 may be performed by the robot 700. The objective of the data harvest routine 300 may be to acquire RGB, depth, LiDAR, and odometry data while ensuring coverage throughout the home. Although a particular sequence of operations is illustrated for data harvest routine 300, the sequence may be altered without departing from the scope of the present disclosure. For example, some of the operations depicted may be performed in parallel or in a different sequence that does not materially affect the function of the data harvest routine 300. In other examples, different components of an example device or system that implements the data harvest routine 300 may perform functions at substantially the same time or in a specific sequence.

[0079] A data harvest initialization request may be received by the robot 700 at block 302. This request may be generated by a user through the user interface 120 in one embodiment, or may be indicated by a signal from the recommendation logic 110 of the system architecture 100 if insufficient data is available to complete the report and recommendation routine 200 and transition to the model deployment routine 600 as described above.

[0080] In response to the data harvest initialization request, the robot 700 may confirm adequate resources are available to perform the harvest at decision block 304. For example, the robot 700 may confirm and begin data harvest if the battery is powered at > 50% and > 5 gigabytes of free memory space is available.

[0081] If resources are found inadequate at decision block 304, user intervention may be requested. The data harvest routine 300 may be paused at block 306 while the robot 700 recharges, performs a memory cleanup, or otherwise takes action to acquire adequate resourcing for the data harvest, as one of ordinary skill in the art will appreciate. As will also be clear to one of ordinary skill in the art, battery power and memory space needs may vary from the exemplary values given herein, and other resource parameters may also apply.

[0082] At block 308, the robot 700 may perform local path planning to plan a data harvest navigation path that ensures adequate coverage during data harvest. A simultaneous localization and mapping or SLAM-based coverage path planning algorithm may be used to ensure each accessible area of an environment is visited at least once. The robot 700 may aim to cover > 90% of the desired environment in harvesting data. The robot 700 may confirm that the data harvest navigation path adequately covers the environment of the robot per a threshold,Docket No. FSP2375PCTsuch as the > 90% above. While coverage is determined to be < 90% at decision block 310, The robot 700 may flag incomplete coverage at block 312. In one embodiment, the robot 700 may alert the user as well, and additional coverage checks and planning may be performed either autonomously or with user guidance.

[0083] Data harvest through sensor recording may be performed by the robot 700 at block 314. In one embodiment, harvesting data using at least one camera configured as a sensing system of the robot may be performed. Robot sensors through which data is harvested may include RGB video and / or still cameras, depth sensors, lidar sensors odometry sensors, inertial measurement unit (IMU) sensors, and other sensors with which the robot 700 may be equipped, and which may provide data pertinent to the environment and use conditions within which the robot 700 is intended to operate. Sensing systems are exemplified with respect to the sensing system 706 and sensing system 1002 described below with respect to FIGs. 7A-9B and FIG.10, respectively. In one embodiment, harvested data may include MP4 video data at 30 frames per second (FPS) and / or still data at from 720p RGB cameras. Lidar scanning may be performed at 5 Hz. Depth data may be calculated from video and still frames, or may be determined through additional sensing. Wheel encoders and other robot joint encoders may provide odometry data for robot movement, location, and pose. Harvested data in the form of sensor recordings may be stored at block 316. In one embodiment, MP4 video from RGB cameras may be encoded in H.264 format with embedded robot operating system (ROS) time metadata. Depth frames, lidar scans, odometry, and IMU data may be saved in ROSBags at 5 Hz. Harvested data may be stored in the robot memory as harvested sensor frames.

[0084] A failure check may be performed at block 318. Failure recognition may in one embodiment include determining that the robot 700 has remained stationary for 10 seconds without performing a planned motion or action. Such failures may be logged as stuck events. If a failure is detected at decision block 320, an alert may be sent to the user at block 322 indicating that a failure has occurred and requesting manual correction. The user may then make the needed corrections to correct the failure at block 324. For example, if an obstacle is causing the failure detected at decision block 320, the user may manually operate the robot 700 around the obstacle, or may remove the obstacle from the robot's path and instruct the robot 700 to continue performing the data harvest routine 300. The robot may then resume harvesting data at block 314.Docket No. FSP2375PCT

[0085] Local storage such as robot data logs 104 may be updated with harvested data when data harvesting is complete and data is ready for processing at block 326. The harvested data may be written to an internal directory and synchronized via timestamps. The harvested data may thus be saved for preliminary local data sampling followed by cloud processing and more distributed storage. After updating local storage with the harvested data at block 326, the system architecture 100, outputs from the robot 700 such as on-device recordings and logged MP4 video, ROSBags for LiDAR and odometry, and coverage / failure logs may undergo the preliminary data sampling routine 400 described below.

[0086] FIG. 4 illustrates a preliminary data sampling routine 400 in accordance with one embodiment. At preliminary data sampling routine 400, data may be further sampled, filtered, and otherwise processed locally, such as by the robot data processor 116, as described below. The preliminary data sampling routine 400 may take in the MP4 videos containing RGB frames and the ROSBags for lidar and odometry data from the data harvest routine 300 and may reduce raw data volume on the robot by extracting frames at 1 FPS and discarding poor-quality images.

[0087] The preliminary data sampling routine 400 may also remove or redact personal and sensitive information before uploading data to the cloud for more distributed and persistent storage and use by cloud resources such as training and refinement of the cloud-based teacher robot task control model 106. "Personal information" refers to data that may directly identify or reasonably be used to identify a person (names, phone numbers, addresses, etc.). "Sensitive information" refers to deeply private data related to health, finances, religious or political beliefs, race, sexual orientation, or other intimate details.

[0088] In one embodiment, this preliminary data sampling routine 400 may be performed locally (i.e., using a processor and memory installed in the hardware of the robot 700) by the robotic control system 2900 of the robot 700. Other processing and storage configurations in communication with the robotic control system 2900 of the robot 700 may operate to perform the preliminary data sampling routine 400, as one of ordinary skill in the art will appreciate.

[0089] Although a particular sequence of operations is illustrated for preliminary data sampling routine 400, the sequence may be altered without departing from the scope of the present disclosure. For example, some of the operations depicted may be performed in parallel or in a different sequence that does not materially affect the function of the preliminary data sampling routine 400. In other examples, different components of an example device or systemDocket No. FSP2375PCTthat implements the preliminary data sampling routine 400 may perform functions at substantially the same time or in a specific sequence.

[0090] The preliminary data sampling routine 400 may reduce raw data volume on the robot by downsampling the frame rate of the harvested sensor frames (e.g., a video frame rate for image data) at block 402. In one example, a single FPS is extracted from a 30 FPS stream, and each extracted frame is assigned a matching ROS-time. At block 404, the routine may include removing low-quality sensor frames from the downsampled sensor frames. Frames having an average brightness outside a configured boundary, such as [40, 220], or a Laplacian variance for sharpness below a threshold of 80, may be discarded in one embodiment. At block 406, near-duplicate frames may be removed. This minimal deduplication may rely on a local feature matching step, such as Oriented Features from Accelerated Segments Test (FAST) and Rotated Binary Robust Independent Elementary Features (BRIEF) (known as ORB) and RANdom SAmple Consensus (RANSAC). Any pair of images with greater than 80% inlier matching may be considered redundant, resulting in one image out of the pair being kept. In this manner, a pre-sampled set of sensor frames may be developed. At block 408, the pre-sampled frames are stored locally on the robot to await privacy filtering.

[0091] The preliminary data sampling routine 400 may continue with a series of privacy filtering steps. At block 410, the system may detect sensor frames containing faces and people. A convolutional neural network (CNN) model, such as a MobileNet-based model, may check image frames for faces and entire human silhouettes. Depth and lidar may be analyzed for indications of human presence and attributes. If a person is detected with a confidence level equal to or greater than a threshold such as 0.70, the entire frame may be removed in one embodiment. In one embodiment, data indicating faces and people may be redacted or obscured for privacy. At block 412, the system may detect frames showing personal data in text. An optical character recognition (OCR) engine such as Tesseract OCR may scan for addresses or phone numbers recognized by alphanumeric patterns (e.g., detecting known strings such as “Street,” “Road,” or series of numerical digits. Relevant regions containing such data may be blacked out. At block 414, the system may analyze depth data for human-like silhouettes. The depth sensor data may be fused with the image data to confirm if a detected silhouette matches a physical person, thereby avoiding false positives from mannequins or images displayed on screens. At block 416, a policy may be applied to redact frames showing at least one of faces, people, and personal information. Redacting the sensor frames may include removing,Docket No. FSP2375PCTobscuring, and anonymizing the image data, depth data, and lidar data captured in the sensor frames.

[0092] At block 418, the system may detect sensor frames containing sensitive information in the pre-sampled set of sensor frames. The system may entirely expunge frames showing sensitive information. For highly sensitive information, such as health data or credit card details, the frame may be entirely discarded rather than redacted. This result may be a filtered, privacy-cleaned set of sensor frames.

[0093] Finally, at block 420, the filtered, privacy-cleaned set of sensor frames may be stored as a filtered, privacy-cleaned, anonymized dataset ready for transmission from robot memory to the cloud-based elements of the disclosed system. In one embodiment, storing the pre-sampled set of sensor frames replaces the previously stored harvested sensor frames in the robot memory, and storing the filtered, privacy-cleaned set of sensor frames replaces the previously stored pre-sampled set of sensor frames in the robot memory. In such embodiments, the most-processed set may be retained at each stage, replacing earlier sets and thus reducing on-device storage needed to host the image or sensor frames useful for robot operations.Cloud-Based Data Sampling & Deduplication

[0094] FIG. 5 illustrates a data sampling and deduplication routine 500 in accordance with one embodiment. The objective of the data sampling and deduplication routine 500 is to securely upload the privacy-cleaned dataset to the cloud for advanced sampling and deduplication and to perform advanced data sampling, near-duplicate removal, and thorough quality checks on the data set using powerful cloud hardware. Corrupted or unreadable files may be eliminated, producing a high-quality dataset for use by the system architecture 100 in performing the model deployment routine 600.

[0095] The data sampling and deduplication routine 500 may be performed by the cloud-based data processor 118. Although a particular sequence of operations is illustrated for data sampling and deduplication routine 500, the sequence may be altered without departing from the scope of the present disclosure. For example, some of the operations depicted may be performed in parallel or in a different sequence that does not materially affect the function of the preliminary data sampling routine 400. In other examples, different components of an example device or system that implements the data sampling and deduplication routine 500 may perform functions at substantially the same time or in a specific sequence.Docket No. FSP2375PCT

[0096] In one embodiment, a user may be queried to confirm they approve sending the harvested data, in the form of the filtered, privacy-cleaned set of sensor frames, off-device for cloud processing. If user approval is received at block 502, confirming approval for data upload to the cloud, the filtered, privacy-cleaned set of sensor frames may be compressed and encrypted for efficient and secure transmission at block 504. The compressed archive (e.g., .tar.gz) may in one embodiment be created and encrypted using 256-bit Advanced Encryption Standard (AES) encryption keyed to the server’s public key.

[0097] At block 506, the collection of privacy-cleaned images and optional sensor logs (e.g., the encrypted filtered, privacy-cleaned set of sensor frames) may be securely uploaded.Transmission may occur via HTTPS POST, and the server may compute a SHA-256 checksum to confirm file integrity. Uploaded data may be encrypted under a unique house identification. Once successful, the upload may be logged as completed at block 508. In one embodiment, the robot may log an “upload complete” event and may notify the user. Following a successful upload, cloud ingestion may then occur at block 510. This ingestion may include validating the data and preparing it for inclusion in a structured storage system such as a database, rather than simply writing the raw data to memory.

[0098] At block 512, associated vector embeddings for the filtered, privacy-cleaned set of sensor frames may be extracted by the cloud-based data processor using an embedding extractor, such as a multi-dimensional vision transformer, which may be a 512-dimensional vision transformer, to process the images. At block 514, the extracted associated vector embeddings may be indexed within, for example, a global FAISS cloud database in one embodiment. At block 516, high-fidelity quality checks may be performed on the filtered, privacy-cleaned set of sensor frames. For example, a large CNN may evaluate image clarity, ensuring no glare or significant motion blur is present, and any image scoring below a 0.50 clarity score may be discarded, removing uncertain and low-quality filtered sensor frames from the ensemble. At block 518, uncertainty-based sampling may be conducted on the high-quality, filtered set of sensor frames. An ensemble detection and segmentation system may process the images to measure disagreement. Frames with an ensemble disagreement above an ensemble disagreement threshold, such as 0.20, may be identified as valuable and retained while others may be discarded. In this manner, a differentiated, high-quality, filtered set of sensor frames may be formed. At block 520, near-duplicate frame removal may be executed to detect and remove additional near-duplicate sensor frames from the differentiated, high-quality, filteredDocket No. FSP2375PCTset of sensor frames. Embeddings with a similarity greater than 0.90 may be merged or removed in one embodiment to reduce dataset bloat. Finally, at block 522, the final resulting curated set of sensor frames, which in some embodiments typically represents 25-30% of the original frames, may be stored in cloud-based storage for future training.

[0099] In one embodiment, the system may perform fault detection steps as follows. At block 524, the system may validate that each image’s SHA-256 checksum hash matches the one computed during upload. At block 526, the system may test image decoding. An OpenCV-based script may attempt to load each image. Decode failures may be logged and the failing file may be removed. Dataset integrity may be confirmed at block 528. The system may create a “fault detection complete” log entry. The final, corruption free dataset may be ready for use in model refinement or model personalization training.Machine Learning Model Adaptation

[0100] FIG. 6 illustrates a model deployment routine 600 in accordance with one embodiment. Although a particular sequence of operations is illustrated for model deployment routine 600, the sequence may be altered without departing from the scope of the present disclosure. For example, some of the operations depicted may be performed in parallel or in a different sequence that does not materially affect the function of the model deployment routine 600. In other examples, different components of an example device or system that implements the model deployment routine 600 may perform functions at substantially the same time or in a specific sequence.

[0101] At decision block 602, it may be determined whether model refinement actions or model personalization actions are needed by the recommendation logic 110 of the task control model adaptation controller 108. In one embodiment, on condition the second confidence values for the second classification outputs (from the teacher model) are at least equal to a first threshold and the first confidence values of the first classification outputs (from the local model) are less than a second threshold, as determined by recommendation logic 110 of the task control model adaptation controller 108, the recommendation logic 110 may recommend refining the model, proceeding through block 604 to block 606. On condition the second confidence values from the teacher model are less than the first threshold, as determined by the recommendation logic 110 of the task control model adaptation controller 108, the recommendation logic 110 may recommend personalizing the model, proceeding through block 604 to block 616.Docket No. FSP2375PCT

[0102] For example, if the teacher model label aligns with known classes (e.g., confidence > 0.80) while the local model fails or has low confidence (< 0.30), the recommendation logic may recommend model refinement, which may be performed by the refinement logic 112.Conversely, if the teacher model identifies an object label not present in the local model’s class list, or indicates an uncertain but novel object, the recommendation logic may recommend model personalization, to be performed by the personalization logic 114.

[0103] In one embodiment, a user notification may be performed at block 604 to inform the user of the recommended adaptation path. The system may inform the user that it may adapt or refine an existing class through model refinement, or that it may create a new class through model personalization. In one embodiment, if fewer than 3 matched frames are found, the recommendation logic 110 may recommend that a data harvest be run again to gather more data through data harvest routine 300. A user confirmation may be provided to trigger additional steps, or the task control model adaptation controller 108 may proceed in accordance with the recommendation of the recommendation logic 110 without user interaction.

[0104] If model refinement is triggered at block 606, the refinement training pipeline may be entered at block 608. The goal of model refinement may be to enhance recognition accuracy for existing classes if the robot’s local model fails on them. Such enhancements may involve or be applied to bounding box detection or segmentation masks. The refinement logic 112 may take in the dataset provided to the cloud and processed by data sampling and deduplication routine 500, as well as a global superset data set which may be a large repository of the labeled images from various tidying environments. A global baseline detection or segmentation model may also be provided.

[0105] Candidate models may be trained at block 610. In one embodiment, an FAISS-based embedding search may retrieve a superset of images (e.g., 200 images) from a global repository that match the user’s environment with a similarity of > 0.80, which may help prevent catastrophic forgetting of global classes. In one embodiment, the local robot task control model 102 may be trained using the relevant subset of sensor frames determined in the report and recommendation routine 200 of FIG. 2. The local data and superset images may be merged, and auto-labeling may be applied if the teacher model’s confidence is > 0.85. A plurality of refined candidate models (for instance, a detection-based approach with partial layer freezing, a segmentation-based approach with no layer freezing, and a hybrid approach with detection plus instance segmentation) may be trained. In one embodiment, each candidate model may beDocket No. FSP2375PCTtrained for 10 epochs with a batch size of 16 and an Adam optimizer at le-4 learning rate. In one embodiment, the global dataset used in training the plurality of refined candidate models comprises at least a portion of the curated set of sensor frames stored in cloud-based storage produced by the data sampling and deduplication routine 500 (FIG. 5). The global dataset used in the disclosed similarity search may comprise sensor frames that are captured during data harvest, training, or normal robot operations, and have undergone deduplication and privacy cleaning as described above. In one embodiment, the global dataset may be, in whole or in part, retrieved as a superset of sensor frames from a global repository that matches the user's environment with a similarity above a predetermined threshold, such as > 0.80.

[0106] At block 612, the candidate models trained in block 610 may be evaluated and ranked against local and global test sets. For example, the system may adhere parameters such as that any global performance drop is < 5%. The candidates may be evaluated against local and global test sets, and the best performer may be picked for deployment. In one embodiment, each candidate model may be evaluated on a local test set of 100 images from the user’s environment and a global test set of 200 images from unrelated environments. The refinement logic 112 may calculate local mAP or mloU improvement and may confirm from these calculations that the global performance drop is < 5%. In this manner, the refinement logic 112 may determine a top-ranked model from the plurality of refined candidate models. This ranking may be based on performance of the plurality of refined candidate models on both the relevant subset of sensor frames and the global dataset, as described.

[0107] The top-ranked model determined in block 612 may be selected as the refined robot task control model. This model may then be deployed to the robot at block 614. Refining the local robot task control model may include converting the refined robot task control model into a format optimized for hardware of the robot before deployment. In one embodiment, this topranked model may be converted to TensorRT format and sent back to the robot at block 614. The robot may automatically load it on the next system restart. Following deployment, robot operation may continue at block 626 with the local robot task control model 102 now refined and specialized to reflect the user's environment to be tidied and forgetting of global classes minimized.

[0108] If model personalization is triggered at block 616, the personalization training pipeline may be entered at block 618 by the personalization logic 114. The goal of model personalization may be to introduce new classes or behaviors outside the global model’s scope,Docket No. FSP2375PCTeither detection or segmentation-based. The personalization logic 114 may take in a global repository including additional references for shape or color similarity. A few-shot data collection may be performed. In one embodiment, the user may need to supply at least 10 labeled images or short video segments of the new class using the user interface 120. The system may retrieve partial matches from the global repository having a similarity > 0.70.

[0109] At block 620, candidate models may be trained. These models may include specialized bounding-box classifiers and / or segmentation classifiers. In one embodiment, two specialized classifiers or segmenters may be trained. The bounding box approach may add new class labels associated with bounding boxes. The segmenter approach may add a new mask channel to the segmenter. In either approach, the models may be trained for 8 epochs at a 2e-4 learning rate with batch size 8.

[0110] At block 622, the models trained in block 620 may be ranked by precision or Intersection over Union (loU), and user-defined behavior rules (e.g., "push these items to a corner") may be assigned to the new class. The personalization logic 114 may measure precision / recall or loU. Trained models may be ranked by precision / IoU. In one embodiment, the user may also include custom behavior rules. For example, the user may enter a rule such as “push these items to a corner” or “handle fragile items below 1 N grip force.” The model that performs best against these criteria may be selected for deployment. In this manner, the refinement logic 112 may determine a top-ranked model from the plurality of personalized candidate models. This ranking may be based in part on at least one of precision and intersection over union (loU) of the plurality of refined candidate models, as described.

[0111] The top-ranked model determined in block 622, may be selected as the personalized robot task control model. Personalizing the local robot task control model may include converting the personalized robot task control model into a format optimized for the hardware of the robot before deployment. This model may then be deployed to the robot at block 614. The specialized model and behavior logic may be compiled and deployed to the robot at block 624. This may include a specialized detector or segmenter associated with a new class, along with the user's specified behavior rules. In one embodiment, an updated behavior tree may be installed to handle the newly-included classes. Following deployment, robot operation may continue at block 626 with the local robot task control model 102 now personalized to reflect the user's environment to be tidied and desired custom behaviors.Docket No. FSP2375PCT

[0112] In one embodiment, model personalizations developed to improve the performance of a robot in completing desired tasks by improving the local robot task control model may also be used to augment, expand, and improve the cloud-based teacher robot task control model.Privacy-cleaned datasets may be stored in cloud-based storage and teacher models trained using anonymized, private data. In this manner, individual robot models may be updated dynamically or “on the fly” as part of day-to-day operations, without necessitating a wait for a regular software update or centralized model maintenance. By leveraging known classes and similar sensor frames across multiple robot devices and their environments, the centralized cloud-based teacher robot task control model may also be updated dynamically, leveraging a substantial body of data, obtained on a regular basis from multiple users, the user reports acting as a feedback loop through which the cloud-based teacher robot task control model may continuously improve without having to wait for regular maintenance and updates. In this manner, the disclosed solution represents a significant improvement in model training efficiency for both individual, comparatively light-weight local robot task control models across a number of environments and more centralized, large-scale, cloud-based teacher robot task control models.

[0113] FIGs. 7A-9B illustrate a robot 700 in accordance with one embodiment. FIG. 7A shows a side view, FIG. 7B shows a top view, FIG. 8 shows a bottom view of a chassis 702 of the robot 700, and FIGs. 9A-9B illustrate a simplified side view and top view of a chassis 702, respectively, in order to show in more detail aspects of the mobility system 704, the sensing system 706, and the communications 906, in connection with the robotic control system 2900. The robot 700 may comprise the chassis 702 to support and contain the other components of the robot 700, a mobility system 704, a sensing system 706, a capture and containment system 708, and a robotic control system 2900 providing actuation signals based on sensor signals and user inputs.

[0114] The capture and containment system 708 may comprise two pusher pads 710 with pad multi-pivot joints 712, two pusher pad arms 714 with pad arm multi-pivot joints 716, pad end grippers 718 at the end of each pusher pad 710, each pad end gripper 718 having a pad end gripper multi-pivot joint 720, a scoop 722, a scoop pivot point 724, a scoop arm 728, a scoop arm pivot point 730, an actuated scoop gripper 732, a scoop gripper arm 734 with a scoop gripper arm pivot point 736, a scoop support wheel 738, a scoop support wheel arm 740 with a scoop support wheel arm pivot point 742, and a lifting column 744 to raise and lower theDocket No. FSP2375PCTcapture and containment system 708 to a desired height. In one embodiment, the scoop gripper arm 734 may include features for gripping and / or gripping surfaces in lieu of or in addition to the actuated scoop gripper 732.

[0115] The robot 700 may further include a mop pad 750, and robot vacuum system 752. The robot vacuum system 752 may induce a cleaning airflow 754, and may work in conjunction with a rotating brush 756 and spinning side brushes 758 to clear dirt and debris from a floor. The robot 700 may include a robot charge connector 760, a battery 762, and a number of motors, actuators, sensors, and mobility components as described in additional detail below.

[0116] The mobility system 704 may comprise wheels as indicated, as well as caterpillar tracks, conveyor belts, etc., as is well understood in the art. The mobility system 704 may further comprise motors, servos, or other sources of rotational or kinetic energy to impel the robot 700 along its desired paths. Mobility system 704 components may be mounted on the chassis 702 for the purpose of moving the entire robot without impeding or inhibiting the range of motion needed by the capture and containment system 708. Elements of the mobility system 704 are described in additional detail with respect to FIG. 8. Elements of a sensing system 706, such as cameras, lidar sensors, or other components, may be mounted on the chassis 702 in positions giving the robot 700 clear lines of sight around its environment in at least some configurations of the chassis 702, scoop 722, pusher pad 710, and pusher pad arm 714 with respect to each other. Elements of the sensing system 706 are described in additional detail with respect to FIGs. 9A-9B.

[0117] The chassis 702 may house and protect all or portions of the robotic control system 2900, (portions of which may also be accessed via connection to a cloud server) comprising in some embodiments a processor, memory, and connections to the mobility system 704, sensing system 706, and capture and containment system 708. The chassis 702 may contain other electronic components such as batteries 762, wireless communications 906 devices, etc., as is well understood in the art of robotics. The robotic control system 2900 may function as described in additional detail with respect to FIG. 29. The mobility system 704 and or the robotic control system 2900 may incorporate motor controllers used to control the speed, direction, position, and smooth movement of the motors. Such controllers may also be used to detect force feedback and limit maximum current (provide overcurrent protection) to ensure safety and prevent damage.Docket No. FSP2375PCT

[0118] Pusher pads 710 of the capture and containment system 708 may have pad motors 764a and 764b to rotate them into different positions around the pad multi-pivot joints 712. Pusher pad arms 714 may be associated with pad arm motors 768a-768c that rotate them around pad arm multi-pivot joints 716, as well as pad arm linear actuators 766 to extend and retract the pusher pad arms 714. Pad end grippers 718 may rotate around pad end gripper multi-pivot joints 720 through the action of pad end gripper motors 770a and 770b.

[0119] Pad motor 764a may actuate pitch rotational motion of the pusher pad 710 and pad motor 764b may actuate yaw rotational motion of the pusher pad 710 at the pad multi-pivot joint 712, giving the pad multi -pivot joint 712 two degrees of rotational freedom. Pad arm motor 768a may actuate pitch rotational motion of the pusher pad arm 714, pad arm motor 768b may actuate yaw rotational motion of the pusher pad arm 714, and pad arm motor 768c may actuate roll rotational motion of the pusher pad arm 714 at the pad arm multi-pivot joint 716, giving the pad arm multi -pivot joint 716 three degrees of rotational freedom. Pad end gripper motor 770a may actuate pitch rotational motion of the pad end gripper 718 and pad end gripper motor 770b may actuate roll rotational motion of the pad end gripper 718 at the pad end gripper multi -pivot joint 720, giving the pad end gripper multi -pivot joint 720 two degrees of rotational freedom.

[0120] In this manner, the pusher pads 710, pusher pad arms 714, and pad end grippers 718 may mimic the mobility of human arms with the pad arm multi -pivot joints 716 acting as shoulders, the pad multi-pivot joints 712 acting as elbows, and the pad end gripper multi-pivot joints 720 acting as wrists. In one embodiment, the pad arm linear actuator 766 may include an additional fourth degree of freedom by imparting linear motion along the length of the pusher pad arms 714, acting as the upper arm portions of the robot arms. This linear motion may be actuated at the pad arm multi-pivot joint 716, thus improving upon the motion of humanoid arms by supporting extra arm length when needed while allowing shortened arm length when that is desirable for a particular action.

[0121] Pusher pads 710 of the capture and containment system 708, acting as a forearm portion of the disclosed robot arm, may incorporate sweeping bristles, soft rubber or foam strips, or some other similar material along one edge. This may allow the pusher pads 710 to sweep very small, light objects along a surface as part of the tidying process, without damage to the surface or objects. The pad end grippers 718, acting as the hand of the robot arm may be simple grippers in one embodiment, having a gripping actuator configured to open, close, andDocket No. FSP2375PCThold the gripper in position. In one embodiment, the pad end grippers 718 may be a three- or five-fingered hand-like gripper including individual finger actuators for each finger. The pad multi -pivot joint 712 may be a spherical joint with three degrees of freedom, similar to the pad arm multi-pivot joint 716. It will be readily apprehended by one of ordinary skill in the art that additional actuators may be used in conjunction with those illustrated to impart additional degrees of freedom of motion at each multi -pivot joint.

[0122] The capture and containment system 708 may comprise a scoop 722 with an associated scoop motor 772 to rotate the scoop 722 into different positions at the scoop pivot point 724. The capture and containment system 708 may also include a scoop arm 728 with an associated scoop arm motor 776 to rotate the scoop arm 728 into different positions around the scoop arm pivot point 730, and a scoop arm linear actuator 774 to extend the scoop arm 728. The scoop gripper arm 734 may include a scoop gripper arm motor 780 to move the scoop gripper arm 734 around a scoop gripper arm pivot point 736, as well as a scoop gripper arm linear actuator 778 to extend and retract the scoop gripper arm 734. In this manner the scoop gripper arm 734 may be able to move and position itself and / or the actuated scoop gripper 732 to perform the tasks disclosed herein. The scoop 722 may include a scoop support wheel 738 at the end of a scoop support wheel arm 740 in one embodiment. The scoop support wheel arm 740 may be extended using a scoop support wheel linear actuator 782 and positioned around a scoop support wheel arm pivot point 742 using a scoop support wheel motor 784. This may allow the robot 700 to extend the scoop support wheel 738 to roll along the floor beneath the scoop 722, providing support to the scoop 722 and counteracting the cantilever effect of weight within the scoop 722 when carrying heavy or unstable objects. The scoop 722 may have movable scoop walls 726 on either side. The motion of the movable scoop walls 726 may allow the scoop 722 to support objects wider than the dimensions of the scoop 722 when its walls are in position as shown in FIG. 7A.

[0123] Points of connection shown herein between the scoop arms and pusher pad arms are exemplary positions and are not intended to limit the physical location of such points of connection. Such connections may be made in various locations as appropriate to the construction of the chassis and arms, and the applications of intended use. The pusher pad arms 714 may each attach to either side of the lifting column 744 at a pad arm multi-pivot joint 716, and the scoop arm 728 may attach to the front of the lifting column 744 at a scoop arm pivot point 730, as shown here. In some embodiments, the pusher pad arms 714 may attach to theDocket No. FSP2375PCTscoop 722. In other embodiments, the pusher pad arm 714 and scoop arms 728 may attach to the chassis 702 at a common point, as shown, for example, in FIG. 7A. It will be well understood by one of ordinary skill in the art that the configurations illustrated may be designed to perform the basic motions needed for the processes illustrated elsewhere herein.

[0124] The geometry of the scoop 722 and the disposition of the pusher pads 710 and pusher pad arms 714 with respect to the scoop 722 may describe a containment area in which objects may be securely carried. Servos, direct current (DC) motors, or other actuators at the pad multipivot joints 712, pad arm multi -pivot joints 716, and scoop arm pivot point 730, may be used to adjust the disposition of the scoop 722, pusher pads 710, and pusher pad arms 714 between fully lowered scoop and grabber positions and raised scoop and grabber positions.

[0125] In some embodiments, gripping surfaces may be configured on the sides of the pusher pads 710 facing inward toward objects to be lifted. These gripping surfaces may provide cushion, grit, elasticity, or some other feature that increases friction between the pusher pads 710 and objects to be captured and contained. In some embodiments, the pusher pad 710 may include suction cups in order to better grasp objects having smooth, flat surfaces. In some embodiments, the pusher pads 710 may be configured with sweeping bristles. These sweeping bristles may assist in moving small objects from the floor up onto the scoop 722. In some embodiments, the sweeping bristles may angle down and inward from the pusher pads 710, such that, when the pusher pads 710 sweep objects toward the scoop 722, the sweeping bristles form a ramp, allowing the foremost bristles to slide beneath the object, and direct the object upward toward the pusher pads 710, facilitating capture of the object within the scoop and reducing a tendency of the object to be pressed against the floor, increasing its friction and making it more difficult to move.

[0126] The capture and containment system 708, as well as some portions of the sensing system 706, may be mounted atop a lifting column 744, such that these components may be raised and lowered with respect to the ground to facilitate performance of complex tasks. A lifting column linear actuator 746 may control the elevation of the capture and containment system 708 by extending and retracting the lifting column 744. A lifting column motor 748 may allow the lifting column 744 to rotate so that the capture and containment system 708 may be moved with respect to the robot 700 base or chassis 702 in all three dimensions.

[0127] The robot 700 may include floor cleaning components such as a mop pad 750 and a robot vacuum system 752 producing a cleaning airflow 754 as shown here. The mop pad 750Docket No. FSP2375PCTmay be able to raise and lower with respect to the bottom of the robot 700 chassis 702, so that it may be placed in contact with the floor when desired. The mop pad 750 may include a drying element to dry wet spots detected on the floor. In one embodiment, the robot 700 may include a fluid reservoir, which may be in contact with the mop pad 750 and able to dampen the mop pad 750 for cleaning. In one embodiment, the robot 700 may be able to spray cleaning fluid from a fluid reservoir onto the floor in front of or behind the robot 700, which may then be absorbed by the mop pad 750.

[0128] FIG. 8 illustrates a simplified plan view of the bottom of a chassis 702 of the robot 700, illustrating details of a mobility system 704 in accordance with one embodiment. The mobility system 704 of the robot 700 may be configured with elements that allow it to navigate, tidy, and vacuum on stairs in addition to the other household tasks the robot 700 is capable of, such as moving bins on / off shelves, opening and closing cabinets, tidying objects off of countertops, etc., as described herein. The mobility system 704 may include wheels such as the right front wheel 802a, left front wheel 802b, right middle wheel 802c, left middle wheel 802d, right rear wheel 802e, and left rear wheel 802f. The mobility system 704 may further include a left sliding track 804a and right sliding track 804b upon which the left middle wheel 802d and right middle wheel 802c may travel, respectively. These elements of the mobility system 704 may be actuated to perform the tasks described herein through the operation of drive motors 806, orienting motors 808, lifting actuators 810, and sliding actuators 812.

[0129] In one embodiment, the wheelbase of the robot 700 mobility system 704 may have six actively driven wheels. That is, each of wheels 802a-802f may include a drive motor 806. In one embodiment, the wheelbase may have four motorized wheels, two at the front and two midway along the chassis 702 (right front wheel 802a, left front wheel 802b, right middle wheel 802c, and left middle wheel 802d), and two passive castor wheels at the rear (right rear wheel 802e and left rear wheel 802f). These layouts may allow the robot 700 to climb stairs as well as move to the left and right while on the stairs.

[0130] In one embodiment, the two front wheels (right front wheel 802a and left front wheel 802b) may be actively-driven wheels that spin, as driven by drive motors 806 in order to move the robot 700 from one location or position to another. Right front wheel 802a and left front wheel 802b may further have active orientation control through orienting motors 808. In one embodiment orientation control may be provided through magnets rather than motors. The orienting motors 808 or magnets may allow the right front wheel 802a and left front wheelDocket No. FSP2375PCT802b to be rotated to face in any direction. In particular, they may be rotated into an orientation that allows motion in a forward and backward direction or a side-to-side direction (in addition to degrees between these two). In one embodiment, the right front wheel 802a and left front wheel 802b may further include lifting actuators 810 such that these wheels may be moved away from or towards the chassis 702 in order to lift the robot 700 to a desired height. In another embodiment, the right front wheel 802a and left front wheel 802b may omit lifting actuators 810 and may remain vertically stationary with respect to the chassis 702. While frontwheel drive as described here may be configured for the robot 700, in one embodiment the front wheels may be passive casters capable of spinning and rotating to follow the motion of the robot as induced by actuators associated with other wheels, such as a rear-wheel drive configuration.

[0131] In one embodiment, the two middle wheels (right middle wheel 802c and left middle wheel 802d) may be actively-driven wheels that spin, as driven by drive motor 806, in order to move the robot 700 from one location or position to another. Right middle wheel 802c and left middle wheel 802d may further have active orientation control through orienting motors 808. In one embodiment, orientation control may be provided through magnets rather than motors. The orienting motor 808 or magnets may allow the right middle wheel 802c and left middle wheel 802d to be rotated to face in any direction. In particular, they may be rotated into an orientation that allows motion in a forward and backward direction or a side-to-side direction (in addition to degrees between these two). Each middle wheel may be extended downward away from the chassis 702 or upward toward the chassis 702 using lifting actuators 810 in order to assist in lifting the robot 700 upwards off of a floor or ground plane in conjunction with other points of support. Each middle wheel may further be capable of shifting horizontally toward the front and rear of the chassis 702 of the robot 700 along a track so that they may assist in lifting the robot 700 upward off of the ground from a forward, centered, or rearward position, as needed. The left middle wheel 802d may move along the left sliding track 804a and the right middle wheel 802c may move along the right sliding track 804b. For example, the left sliding track 804a and right sliding track 804b may be ridged or toothed, and the left middle wheel 802d and right middle wheel 802c may be fitted with toothed or geared cogs impelled by sliding actuators 812 such as motors. In one embodiment, the middle wheels may travel along smooth tracks using linear actuators. Other configurations will readily suggest themselves to those of ordinary skill in the art.Docket No. FSP2375PCT

[0132] In one embodiment, the rear wheels (right rear wheel 802e and left rear wheel 802f) may be actively-driven wheels that spin, as driven by drive motors 806, in order to move the robot 700 from one location or position to another. Tn one embodiment, right rear wheel 802e and left rear wheel 802f may be passively spinning castors which support and follow the movement of the robot 700 as induced by drive motors 806 associated with other wheels. The right rear wheel 802e and left rear wheel 802f may include active orientation control, such as orienting motors 808 or magnets that allow the right rear wheel 802e and left rear wheel 802f to be rotated to face in any direction. In particular, they may be rotated into an orientation that allows motion in a forward and backward direction or a side-to-side direction (in addition to degrees between these two). Right rear wheel 802e and left rear wheel 802f may in one embodiment have passive wheel orientation, which may be supported by wheel bearings or a caster configuration, that allows these wheels to be rotated passively to face in any direction needed to follow the movement of the robot induced by actuators at the other wheels. Each of the rear wheels may in one embodiment be extended and retracted vertically, such as through lifting actuators 810, in order to assist in lifting the robot 700 upward off the ground together with other points of support.

[0133] The wheels described here may operate together in pairs, threes, fours, or all six together as needed to complete various tasks and movements. These wheels may also each operate completely independently from each other, providing all-wheel drive and support in three dimensions, in order to complete complex motions such as ascending a spiral staircase.

[0134] FIGs. 9A-9B illustrate a simplified side view and top view of a chassis 702, respectively, in order to show in more detail aspects of the mobility system 704, the sensing system 706, and the communications 906, in connection with the robotic control system 2900. In some embodiments, the communications 906 may include the network interface 2912 described in additional detail with respect to robotic control system 2900.

[0135] In one embodiment, the mobility system 704 may comprise a right front wheel 802a, a left front wheel 802b, a right middle wheel 802c, a left middle wheel 802d, a right rear wheel 802e, and a left rear wheel 802f. The robot 700 may have front-wheel drive, where right front wheel 802a and left front wheel 802b are actively driven by one or more actuators or motors, around either a shared axle or separate individual axles, while the right rear wheel 802e and left rear wheel 802f spin passively on a shared axle or individually axles while supporting the rear portion of the chassis 702. In another embodiment, the robot 700 may have rear-wheel drive,Docket No. FSP2375PCTwhere the right rear wheel 802e and left rear wheel 802f are actuated and the front wheels turn passively. In another embodiment, the robot 700 may have additional motors to provide allwheel drive, may use a different number of wheels, or may use caterpillar tracks or other mobility devices in lieu of wheels. Middle wheels, right middle wheel 802c and left middle wheel 802d, may similarly be configured as actuated or passive, on a shared axle or individual axles, depending on expected robot operations.

[0136] The sensing system 706 may further comprise cameras such as the front left camera 786a, rear left camera 786b, front right camera 786c, rear right camera 786d, and scoop camera 786e, lidar sensors 902, and IMU sensors 904. In some embodiments, there may be a single front camera and a single rear camera.

[0137] FIG. 10 illustrates robotic control system interfaces 1000 in accordance with one embodiment. The robotic control system interfaces 1000 may comprise a sensing system 1002, including the robot's sensors, such as cameras 1004, lidar sensors 1006, IMU sensors 1008, wheel encoders 1010, and other sensors 1012. The robotic control system interfaces 1000 may further comprise robot user interfaces 1014 and robot actuators 1016. A tidying robot such as the robot 700 introduced in FIG. 7A having a robotic control system such as the robotic control system 2900 described with respect to FIG. 29 may include operational components that interface with the sensing system 1002, robot user interfaces 1014, robot actuators 1016, and each other. These components may include a perception logic 1100, a navigation logic 1200, a robot brain 1300, and an action logic 1400, which are described in detail with respect to FIG.11 through FIG. 14.

[0138] The perception logic 1100 may receive image data 1018 from the cameras 1004. In one embodiment, the perception logic 1100 may receive additional data from other elements of the sensing system 1002. For example, in one embodiment the perception logic 1100 may also receive lidar data 1020 from the lidar sensors 1006. Based on the input image data 1018 received from the cameras 1004, the perception logic 1100 may generate scene layout map data 1036 and local occupancy map data 1038 as described with respect to FIG. 11. The scene layout map data 1036 and local occupancy map data 1038 may be transmitted to the navigation logic 1200. In some embodiments, the perception logic 1100 may send the filtered 2D and 3D bounding boxes 1034 it may generate to the navigation logic 1200 as well.

[0139] The navigation logic 1200 may receive lidar data 1020 from the lidar sensors 1006, IMU data 1022 from the IMU sensors 1008, and wheel encoding data 1024 from the wheelDocket No. FSP2375PCTencoders 1010 in addition to the scene layout map data 1036 and local occupancy map data 1038 from the perception logic 1100. The lidar data 1020, IMU data 1022, and wheel encoding data 1024 may be used to determine a position or “pose” of the robot 700 within the environment it intends to navigate. In addition, this data may be used to transform the scene layout map data 1036 and local occupancy map data 1038 into an inflated local map data 1042 and inflated global map data 1044, as is described in additional detail with respect to FIG. 12. The robot pose 1040, inflated local map data 1042, and inflated global map data 1044 may be transmitted to the robot brain 1300.

[0140] The robot brain 1300 may receive other sensor data 1026 from the other sensors 1012 of the sensing system 1002 in addition to the robot pose 1040, inflated local map data 1042, and inflated global map data 1044 from the navigation logic 1200. The other sensor data 1026 may be used along with the data from the navigation logic 1200 to generate ROS messages, such as ROS messaging signals 1046, for transmission to the action logic 1400. This is described in additional detail with respect to FIG. 13 and FIG. 14.

[0141] The action logic 1400 may receive the ROS messaging signals 1046 from the robot brain 1300. The action logic 1400 may generate robot action command signals 1030 based on the ROS messaging signals 1046 from the robot brain 1300, as is described in additional detail with respect to FIG. 14. The robot actuators 1016 may be actuated by these robot action command signals 1030 in a manner that commands the robot 700 to perform actions as directed by the robot brain 1300. During such actuation, the forces applied by the robot actuators 1016 as they operate may be measured and transmitted back to the action logic 1400 as actuator force feedback signals 1032. Such actuator force feedback signals 1032 may indicate the completion of actions, the encountering of obstructions, fault conditions, etc. The action logic 1400 may use ROS messaging signals 1046 to communicate the actuator force feedback signals 1032 back to the robot brain 1300.

[0142] FIG. 11 illustrates a perception logic 1100 in accordance with one embodiment. The perception logic 1100 may include a detection logic 1102, a scene segmentation logic 1110, and a mapping logic 1116. In one embodiment, the detection logic 1102 may include a 2D object detection model 1104, 3D location prediction 1106, and detection filtering 1108. The scene segmentation logic 1110 may include a segmentation model 1112 and edge detection algorithms 1114. In one embodiment, the mapping logic 1116 may include a labeling logic 1118, a top view transformation logic 1120, a scene layout logic 1122, and a local occupancyDocket No. FSP2375PCTlogic 1126. The perception logic 1100 may take in image data 1018 from the sensing system 1002 and may produce scene layout map data 1036 and local occupancy map data 1038, which it may transmit to the navigation logic 1200.

[0143] The perception logic 1100 may take RGB image data 1018 and may create higher-level features such as 2D bounding boxes and class labels for objects detected in the image data, as well as scene layout and local occupancy maps for use in planning and navigation. The perception logic 1100 may use object detection and segmentation models to identify different elements in the scene represented by the image data. These elements may be converted into 3D coordinates and / or maps using post-processing algorithms.

[0144] Detected objects may be classified and labeled in a number of ways. Some classes that may be useful for the purposes of a clutter tidying robot may be “obstacles,” “dynamic objects,” “static objects,” “movable objects,” “tidyable objects,” “toys,” “pets,” “humans,” “children,” “clothing,” “furniture,” “miscellaneous household object,” etc. Detected objects may be identified using one or more class labels, as is most useful for robot planning and operation. "Obstacles" refers to objects that may impede the passage of a robot as it navigates its environment to complete desired tasks. "Dynamic objects" refers to objects that may be expected to move independently of the robot during the time the robot is exploring, and navigating its environment, such as people, pets, other robots, etc. The term "Static object" in this disclosure refers to elements of a scene that are not expected to change over time, typically because they are rigid and immovable. Some composite objects may be split into a movable part and a static part. Examples include door frames, bookshelves, walls, countertops, floors, couches, dining tables, etc. The term "Movable object" in this disclosure refers to elements of the scene that are not desired to be moved by the robot (e.g., because they are decorative, too large, or attached to something), but that may be moved or deformed in the scene due to human influence. Some composite objects may be split into a movable part and a static part. Examples include doors, windows, blankets, rugs, chairs, laundry baskets, storage bins, etc.

[0145] Given image data 1018, such as a raw RGB camera frame representing a detected image of a scene 1600, exemplified by the image shown in FIG. 16, the detection logic 1102 may estimate the shape and positions of objects of interest in the frame with respect to the robot 700, such as the floor 1602, rug 1604, furniture 1606, person 1608, and toys 1610 captured in the detected image of a scene 1600. The detection logic 1102 may use a 2D object detection model 1104 to detect objects in the frame. This may include object classifications andDocket No. FSP2375PCTthe extents of 2D bounding boxes indicating the locations of the objects. Object detection models such as those in the YOLO series or DEtection TRansformer (DETR) series may be used to perform object detection through neural network architectures or using classical methods. These models may produce 2D bounding boxes with pixel coordinates for box center and size, as well as class labels, such as toy, bin, bin-label, book, clothes, home-misc., human, pet, furniture, etc., as exemplified by classified 2D bounding boxes 1700. Exemplary classified 2D bounding boxes 1700, as shown in FIG. 17, may include bounding boxes classified to indicate obstacles 1702, dynamic objects 1704, tidyable objects 1706, etc.

[0146] The detection logic 1102 may then use 3D location prediction 1106 and a detected ground plane to generate 3D bounding boxes for the objects in the classified 2D bounding boxes 1700. The detections may be lifted from 2D to 3D, assuming the objects are on the ground (in contact with the ground plane). This allows the detection logic 1102 to project the detected objects as 3D bounding boxes with respect to the frame of reference of the robot 700. Assuming all detections to be on the ground, the bottom corners of the bounding boxes may be used as contact points with the floor. Where the height of the camera from the floor is known, as may be the case for the 160, the 3D location of each floor pixel in the image plane may be calculated. This may be used to convert the contact points for each 2D bounding box to 3D coordinates with respect to the camera. The height of each bounding box may be assumed to be the same as the width.

[0147] Where the assumption of every object being on the floor is violated, much larger distances may be calculated as a result. These issues may be corrected by post-processing detection filtering 1108, where the 3D outputs may be filtered to remove irrelevant objects such as humans, furniture, and other objects that violate assumptions of being on the ground or are projected inside the scoop 722 of the robot 700. The detection filtering 1108 may take in the 2D and 3D bounding boxes and the class labels and may compute a distance from the 3D bounding boxes to a known center point of the robot 700. This distance may be used to determine whether or not the object is too far from the robot, or whether it lies within the extent of the robot 700 scoop 722. Objects too far from the robot 700 and objects in the scoop 722 may have their classification and bounding box data removed to form a set of filtered 2D and / or 3D bounding boxes and class labels 1130.

[0148] The scene segmentation logic 1110 may take in raw RGB image data 1018, such as the detected image of a scene 1600, and may use a segmentation model 1112 to identify the floorDocket No. FSP2375PCTand any rugs, playmats, and other relevant boundaries as an edge map. The segmentation model 1112 may be a model such as BiSeNet, Seaformer, or a similar model. The segmentation model 1112 may produce a multi-class segmentation map 1800 for the floor or drivable area.

[0149] The multi-class segmentation map may be sent to edge detection algorithms 1114. The edge detection algorithm 1114 may be a 3x3 kernel convolution with thresholding deployed on a GPU or CPU. Canny or a similar edge detector may be used. Techniques such as maskshifting followed by mask subtraction, performed by the GPU or CPU may be used. In this manner, the edge detection algorithms 1114 may identify the relevant boundaries in the segmentation map with their semantic information. The scene segmentation logic 1110 may in this manner output drivable area boundary masks with semantic information 1900, as illustrated in FIG. 19, identifying a floor boundary 1902 for the detected floor 1602 region and other relevant boundaries 1904 for other relevant regions 1906, such as the rug 1604, or playmats, drop cloths, etc., as drivable area boundary masks with semantic information 1900. The drivable area boundary masks with semantic information 1900 from the scene segmentation logic 1110, and the filtered 2D and / or 3D bounding box and class label 1130 from the detection logic 1102, may both be passed to the mapping logic 1116 for further processing.

[0150] The mapping logic 1116 may take the filtered 2D and / or 3D bounding box and class label 1130 and drivable area boundary masks with semantic information 1900 as input and may add additional semantic context. In this manner, the mapping logic 1116 may generate semantically rich pseudo-laserscans that may be used for planning and navigation tasks by the robot 700. A labeling logic 1118 of the mapping logic 1116 may take in the filtered 2D and / or 3D bounding box and class label 1130 and drivable area boundary masks with semantic information 1900 and may provide additional semantic labels 1132 to the boundary map to improve task-specific filtering of generated pseudo-laserscan points. The locations of 2D bounding boxes or the 3D locations of 3D bounding boxes with respect to the robot 700 may be used to determine points in the edge mask that are inside the corresponding bounding boxes. All of these points may be assigned the same label as the bounding box to provide a semantically rich boundary map 2000.

[0151] A top view transformation logic 1120 may receive the semantically rich boundary map 2000 and use it to create a pseudo-laserscan / point cloud with semantic label classes 2100. The top view transformation logic 1120 may wrap the image of the semantically rich boundary map 2000 to real-world location coordinates. In one embodiment, this may be performed using aDocket No. FSP2375PCTlookup table that stores the pixel mappings from image space to the real world. In one embodiment, this may be performed using a homography matrix that maps the pixels to real-world coordinates in real time. The pseudo-laserscan / point cloud with semantic label classes 2100 may be sent to a scene layout logic 1122 and a local occupancy logic 1126.

[0152] The scene layout logic 1122 may accept the pseudo-laserscan / point cloud with semantic label classes 2100 and may pass it through a filter 1124, keeping the semantically relevant points needed for global mapping and discarding points not needed. The filter 1124 may iterate over all of the points in the pseudo-laserscan / point cloud with semantic label classes 2100 and may remove points whose class does not match classes identified in the filter 1124 configuration parameters. The result may be scene layout map data 1036 such as the scene layout map 2200, which may include the elements relevant for global mapping and may be transmitted to the navigation logic 1200.

[0153] The local occupancy logic 1126 may take in the pseudo-laserscan / point cloud with semantic label classes 2100 and may pass it through a filter 1128, keeping the semantically relevant points needed for local path planning and obstacle avoidance. The filter 1128 may iterate over all of the points in the pseudo-laserscan / point cloud with semantic label classes 2100 and may remove those whose class does not match classes identified in the configuration for the filter 1128. The filtered laserscan / point cloud may form local occupancy map data 1038 such as the local occupancy map 2300, which may include elements useful for local path planning and local obstacle avoidance and may be transmitted to the navigation logic 1200. In some embodiments, the filtered 2D and / or 3D bounding box and class label 1130 may be passed to the navigation logic 1200 or other navigation logic embodiment as shown for the filtered 2D and 3D bounding boxes 1034 introduced in FIG. 10.

[0154] FIG. 12 illustrates a navigation logic 1200 in accordance with one embodiment. The navigation logic 1200 may include a simultaneous localization and mapping logic 1202, a global mapper logic 1208, and a fusion and inflation logic 1216. In one embodiment, the simultaneous localization and mapping logic 1202 may include a sensor data filter and fusion logic 1204 and a main pipeline 1206. The global mapper logic 1208 may include a sensor data filter and semantic filter 1210, multi-sensor data registration 1212, and loop closure integration 1214. The fusion and inflation logic 1216 may include map fusion 1218 and map inflation 1220. The navigation logic 1200 may take in lidar data 1020, IMU data 1022, and wheel encoding data 1024 from the sensing system 1002, as well as scene layout map data 1036 andDocket No. FSP2375PCTlocal occupancy map data 1038 from the perception logic 1100. The navigation logic 1200 may use this data to generate a robot pose 1040, an inflated local map data 1042, and an inflated global map data 1044, which it may transmit to the robot brain 1300.

[0155] The simultaneous localization and mapping logic 1202 may process the lidar data 1020, IMU data 1022, and wheel encoding data 1024 as described below to determine a robot 700 pose and location within a SLAM global map. The robot 700 may be capable of taking on a number of poses for the purposes of performing its tasks, these poses comprising the conditions and actions of its moveable parts, such as the scoop 722, scoop arm 728, scoop arm pivot point 730, pusher pads 710, pusher pad arms 714, pad multi-pivot joints 712, and pad arm multipivotjoints 716, described in detail with respect to FIGs. 7A-9B. The main pipeline 1206 of the simultaneous localization and mapping logic 1202 may provide the robot pose 1040 to the robot brain 1300.

[0156] The sensor data filter and fusion logic 1204 of the simultaneous localization and mapping logic 1202 may take in lidar data 1020, which is a form of laser or visual data representing detected objects in a scene 2400, as well as IMU data 1022 and wheel encoding data 1024 as a source of odometry. Additional visual and odometry data may be received as input from other sources depending on the application or on the hardware available. This data may be passed through relevant filters to eliminate noise and data that is potentially unreliable. Lidar data 1020 may be passed through a box filter that may remove data that lies within a predefined footprint around the lidar, in order to cater to objects around the lidar that may move along with it. This may be achieved by checking each point in the lidar data 1020 to determine if the point lies within a provided polygon based on the number of intersections of a line passing through the point and infinity with all sides of the polygon. Other filters may be used at this stage as well, based on the quality of lidar data 1020 input. In this manner, the sensor data filter and fusion logic 1204 may generate filtered lidar data 1222.

[0157] In order to cater to the bias offset created due to the misalignment of the IMU sensor 1008 z-axis with a gravity vector, a correction filter may be used by the sensor data filter and fusion logic 1204 to rectify this offset in the angular velocity output of the sensor, represented in the IMU data 1022. This offset may be configured at a calibration step. The filter stage may subtract the same offset from every data instance. Additional components of the IMU data 1022 may be used and filtered appropriately in the sensor data filter and fusion logic 1204 depending on the input data quality.Docket No. FSP2375PCT

[0158] Owing to the noise in the angular velocity output of wheel encoding data 1024, the same may be filtered by the sensor data filter and fusion logic 1204. The remaining wheel encoding data 1024 may be fused with the filtered IMU data 1022 to generate an encoder-IMU filtered and fused odometry data 1224 source for the simultaneous localization and mapping logic 1202. An extended Kalman filter (EKF) may be used for this fusion. The filtered lidar data 1222 and filtered and fused odometry data 1224 created by the sensor data filter and fusion logic 1204 may be passed to the simultaneous localization and mapping logic 1202 main pipeline 1206.

[0159] The main pipeline 1206 may receive the filtered lidar data 1222 and filtered and fused odometry data 1224, as well as other relevant visual and odometry information depending on the application or available hardware. The sensing system 1002 may provide a new laser measurement, in the form of lidar data 1020 at particular intervals governed by robot motion. The lidar data 1020 may correspond to a given odometry reading at these intervals, which may also be determined using data from the sensing system 1002, as described above. For each new laser measurement provided in the filtered lidar data 1222, a new 2D point registration may be created by the main pipeline 1206. The filtered lidar data 1222 for a single time instance may be provided in the form of consecutive angular range measurements that give a Euclidean distance between an obstacle detected at a point located at a particular angle from the lidar center. This angular range may be provided in polar coordinate form. This polar data may be converted to 2D cartesian coordinates using standard formulas for 2D geometry, and then transformed from the lidar frame of reference (center of lidar) to the frame of reference for the odometry reading at that point in time, provided in the filtered and fused odometry data 1224. The 2D point registrations may be added to a pose-graph, which may be a graphical representation of the odometry poses at each lidar measurement and their relationships, where nodes represent the poses and edges represent the spatial constraints (i.e., 2D transformations between the poses of corresponding nodes). The pose-graph may be initiated by the main pipeline 1206 when it is provided the filtered lidar data 1222 from the robot's initial lidar scan.

[0160] The main pipeline 1206 may use consecutive such filtered lidar data 1222 scans and filtered and fused odometry data 1224 readings for scan-to-scan matching to estimate and correct any odometry slippages that may have occurred between lidar scans. An odometry slippage may be detected when the filtered and fused odometry data 1224 is inconsistent with the location of the robot 700 indicated by the filtered lidar data 1222. As subsequent scans andDocket No. FSP2375PCTreadings arrive at the main pipeline 1206, a number “N” of the poses (nodes) in the pose-graph may be used together in a chain to look for loop closure. The number N may be tuned based on factors such as expected odometry and lidar noise and compute availability. Loop closure may be detected by determining if the current location being observed through the sensors has been previously visited by the robot or not. This is achieved by comparing current sensor data instances with previously stored instances corresponding to the area near the current estimated location of the robot. If a loop closure is detected, a correction constraint may be added to the pose-graph. The odometry poses for the N poses in the chain may be corrected based on the results from a pose-graph optimization. The optimization may be an error minimization technique incorporating the correction of the newly observed loop closure constraint into the N previous poses. As a result, the main pipeline 1206 may effectively achieve better global consistency in the SLAM global map it generates as output by reducing distortions and noise in mapped regions that arise from odometry slippages.

[0161] The main pipeline 1206 may perform a final spatial registration that may include probabilistically updating the 2D point registrations using methods such as Bayesian belief update and log odds update, which are based on counters of hits and misses of lidar beams. The final spatial registration may use all of the 2D point registrations it has collected from all data instances in a common grid for each particular location in the occupancy grid (cell location) defined by its spatial location in the real world, the starting location of the pipeline, and the map resolution. This grid is initialized at an origin point of the occupancy grid map based on the location where the pipeline is started. The local occupancy grid may drift due to odometry slippage but may be corrected in this manner. Grid sizing depends on the resolution of the map to be created and updated in the main pipeline 1206, as well as the extents of the coordinates in the 2D point registrations generated above, which may increase as the robot explores more areas. This workflow encompasses the approach based on the sensor suite used by us. However, for a different set of sensors, providing similar types of sources may involve additional or alternative steps, but overall may eventually lead to a similar occupancy grid-based representation of the environment.

[0162] Through these steps, the main pipeline 1206 may produce a SLAM global map 2500, which may be a 2D occupancy grid representation 2502 of the environment with obstacle information at lidar height and real-time location information for the robot. An exemplary SLAM global map 2500 is shown in FIG. 25, along with lidar data 2504, scene map data 2506,Docket No. FSP2375PCTa global plan 2508, and real-time location information of the robot 2510. The main pipeline 1206 may provide the SLAM global map 2500 to the loop closure integration 1214 logic of the global mapper logic 1208, which is described in additional detail below.

[0163] The global mapper logic 1208 may include a sensor data filter and semantic filter 1210. The sensor data filter and semantic filter 1210 may receive as input the lidar data 1020 from the sensing system 1002, as well as other sources of relevant visual information, and the scene layout map data 1036 from the perception logic 1100. Similar to the sensor data filter and fusion logic 1204 of the simultaneous localization and mapping logic 1202, this filter may eliminate noise and potentially unreliable data in the context of mapping without regard to localization (the global mapper logic 1208 may rely on the simultaneous localization and mapping logic 1202 for accurate localization). The lidar data 1020 and other visual data may be processed as described for the sensor data filter and fusion logic 1204. The scene layout map data 1036 may be passed through a semantic filter that caters to the filtering of points from the scene layout map data 1036 based on semantic information provided. This may act as an additional step of filtering control to eliminate unreliable / irrelevant semantic labels during mapping. The filtered lidar data 1222 and filtered scene layout map 1226 provided by the sensor data filter and semantic filter 1210 may be input to the multi-sensor data registration 1212.

[0164] Similar to the SLAM main pipeline 1206, the multi-sensor data registration 1212 may create 2D point registrations, but in this case, it may be for the visual sensors of the sensing system 1002. These registrations may created using the localization information from the simultaneous localization and mapping logic 1202 rather than odometry data. The multi-sensor data registration 1212 may transform the visual sensor data from the sensor frame of reference to the frame of reference in which the SLAM logic main pipeline 1206 provides real-time localization information, i.e., the location where the main pipeline 1206 begins mapping. All of the 2D point registrations for the visual sensor data may then be probabilistically updated together in a common 2D occupancy grid based on predetermined sensor confidence values using similar approaches to those described for the main pipeline 1206 above. Additional optimization techniques such as those used for the pose-graph optimization as well as particle filter-based updates may be introduced to provide additional accuracy improvements based on the use case. The multi-sensor data registration 1212 may thus create 2D point registrationsDocket No. FSP2375PCT1228 forming a 2D occupancy grid representation of the environment with obstacle information at lidar height and information from the scene layout map data 1036.

[0165] The 2D point registrations 1228 may be provided to the loop closure integration 1214 of the global mapper logic 1208, along with the SLAM global map 2500 from the simultaneous localization and mapping logic 1202, including odometry pose corrections for the N data points at their N timestamps. The loop closure integration 1214 may use these inputs to create a navigation global map 2600 that is an improved 2D occupancy grid representation 2602, as compared with the 2D occupancy grid representation 2502 of the SLAM global map 2500. The loop closure integration 1214 may leverage the loop closure performed in the simultaneous localization and mapping logic 1202 to improve already -registered points that may be affected due to slippage in odometry that has been fixed by the simultaneous localization and mapping logic 1202 but may need to be updated in the global mapper logic 1208. The loop closure integration 1214 may reiterate over the 2D point registrations 1228 that are temporally near the loop closure pose corrections previously generated. In this manner, slippage may not need to be recomputed and recorrected during global mapper registration at the multi-sensor data registration 1212.

[0166] Using these map pose corrections, the loop closure integration 1214 may update each data instance, and may then update the corresponding 2D point registrations 1228 using the corrected poses. These updated registrations may then be used to re-update the probabilities of respective cells in the grid of the navigation global map 2600 in a manner similar to that used by the main pipeline 1206 to update the SLAM global map 2500. The navigation global map 2600 may thus be an improved 2D occupancy grid representation 2602 of the environment (as compared to the SLAM global map 2500). An exemplary navigation global map 2600 is shown in FIG. 26, along with lidar data 2604, scene map data 2606, and real-time location information of the robot 2608.

[0167] The navigation global map 2600 may be sent to the fusion and inflation logic 1216. The fusion and inflation logic 1216 may also receive lidar data 1020 from the sensing system 1002 and local occupancy map data 1038 from the perception logic 1100. The map fusion 1218 of the fusion and inflation logic 1216 may fuse the multiple input maps and / or sensor data into a fused local occupancy map 1230. The local occupancy map data 1038 and the lidar data 1020 may be used to register the data points in a single-shot manner onto the navigation global map 2600 from the global mapper logic 1208. Any new point in the local occupancy map data 1038Docket No. FSP2375PCTor lidar data 1020 may be updated immediately in the output fused local occupancy map 1230, not probabilistically. The fused local occupancy map 1230 may then be cropped into a smaller square map around the robot 700 (acting as its center). This may provide a local map that is useful for obstacle avoidance in the environment. The immediate updates of laser data may aid with the inclusion of objects at lidar height, such as humans walking, furniture, etc., within the lidar field of vision, as well as those from the local occupancy map data 1038 that may be below the lidar field of vision, such as pets, babies, toys, etc. The fused local occupancy map 1230 may be passed to map inflation 1220.

[0168] The map inflation 1220 may take in the navigation global map 2600 from the global mapper logic 1208 and the fused local occupancy map 1230 from map fusion 1218. In applications where other map information is available, the map inflation 1220 may also take in this data. Map inflation 1220 may perform the inflation of input occupancy grid maps to include a factor of safety around obstacles, in essence creating a buffer for the robot 700. The size of this buffer region may be governed by the robot’s size in order to allow it to avoid obstacles. In one embodiment map inflation 1220 may be used to make inflated versions of global and local maps that may be used for navigation. An expanded occupied cost region may be created around every occupied cell in the input occupancy grid, based on the radius of inflation. This may aid the robot 700 in planning its movements to avoid these regions of buffer around obstacles. This inflation may also help mask narrow spaces in the environment that may appear to be free space but may not have dimensions that would allow the robot 700 to pass through. In this manner, the map inflation 1220 may produce an inflated local map 2700 that is a 2D occupancy grid representation 2702 of the robot's environment with a buffer region 2704. An exemplary inflated local map 2700 is shown in FIG. 27, including the lidar data 2706 and scene map data 2708. A local plan with an integrated global mapper logic 2710, a global plan with an integrated global mapper logic 2712, and real-time location information of the robot 2714 are also shown. The inflated local map 2700 may be provided as inflated local map data 1042 to the robot brain 1300.

[0169] Similarly, the map inflation 1220 may produce an inflated global map 2800 that may be provided as inflated global map data 1044 to the robot brain 1300. The inflated global map 2800 may be a 2D occupancy grid representation 2802 including a buffer region 2804. An exemplary inflated global map 2800 is shown in FIG. 28, including lidar data 2806 and scene map data 2808. A global plan 2810 and real-time location information of the robot 2812 areDocket No. FSP2375PCTalso shown. The inflated global map 2800 may be provided as inflated global map data 1044 to the robot brain 1300.

[0170] FIG. 13 illustrates a robot brain 1300 in accordance with one embodiment. The robot brain 1300 may include a state logic 1302 and a decision logic 1308. The state logic 1302 may include a robot state 1304 and an environment state 1306. The decision logic 1308 may include service endpoints 1310, an event queue 1312, status check 1314 algorithms, event check 1316 algorithms, and behaviors 1318.

[0171] The robot brain 1300 may take the robot pose 1040, inflated local map data 1042, and inflated global map data 1044 data from the navigation logic 1200, as well as data from other sensors 1012 among the sensing system 1002 in addition to those previously mentioned. A state logic 1302 in the robot brain 1300 may use this information to determine a robot state 1304 and an environment state 1306, which the robot brain 1300 may communicate, via a ROS message queue 1402, to an action logic 1400. The robot state 1304 may include conditions and parameters such as the robot pose 1040, as determined by the navigation logic 1200 in one embodiment, a power charging status, the robot's location within its environment, a current task, a previous task, and / or next task the robot has to perform, a task status such as success or failure, and other state conditions, as are described in more detail with respect to FIG. 15A and FIG. 15B. The environment state may reflect conditions of an environment as detected by sensors and preserved as observations and maps. The environment state 1306 may include locations of objects as detected and located by the perception logic 1100, the boundaries of a navigable area, a progress state toward a goal such as all detected objects handled as determined by their categories, etc. A decision logic 1308 may interact with a robot user interface 1014, such as a mobile computing device application, web page, and / or hardware buttons, which provides a user the ability to communicate with and control the robot. The decision logic 1308 may take interface signals 1028 from the robot user interfaces 1014 and determine service endpoints 1310, inform an event handler or event queue 1312, and control robot behaviors 1318. The data from the decision logic 1308 likewise may be communicated to the action logic 1400 via the ROS message queue 1402.

[0172] FIG. 14 illustrates an action logic 1400 in accordance with one embodiment. The action logic 1400 may include a ROS message queue 1402, an ROS universal asynchronous receiver / transmitter (UART) shown as ROS UART 1404, a controller 1406, a local planner 1408, and a global planner 1410.Docket No. FSP2375PCT

[0173] The action logic 1400 may take information from and send information to the robot brain 1300 using the ROS message queue 1402. The action logic 1400 may also take information from and send information to robot actuators 1016, so as to effect action by the robot 700 within its environment, as will be well understood by one of skill in the art. In one embodiment, the action logic 1400 may be a portion of the robotic control system 2900, and may include the motion controller and motor and servo controller that controls actuators such as motors and servos which may impel the robot 700 to move throughout its environment and adjust its pose in order to perform the tasks it is instructed to perform.

[0174] FIG. 15A and FIG. 15B illustrate a robot brain flow diagram 1500 in accordance with one embodiment. The actions of the flow diagram may be performed by a robot brain 1300 in conjunction with the perception logic 1100, the navigation logic 1200, and the action logic 1400, as introduced in FIG. 14.

[0175] The state logic 1302 of the robot brain 1300 may store data on a robot state, such as the robot's position, orientation, velocity, accelerationjoint states, and battery status. The state logic 1302 may also store data on an environment state, such as the global map, the local map, detected objects, and destination bins for different classes of objects. This state data may be communicated via the ROS message queue 1402 to the action logic 1400.

[0176] The robot brain 1300 may also include a decision logic 1308 that takes in data from interfaces (which may include a mobile app, a web application programming interface, hardware buttons, the robot charging port, etc.). The interfaces may communicate with service endpoints, which may inform an event queue 1312 or priority message queue. The event queue 1312 may send poll messages to a status check 1314 algorithm, which may determine an appropriate behavior 1318 from among the behaviors 1318 available. The selection of a behavior 1318 results in a push message back to the event queue 1312, as well as communication to the ROS message queue 1402 in order to effect an action by the robot using the action logic 1400.

[0177] Beginning with the status check 1314, if a need for an emergency stop is detected at check 1502, the HALT 1540 behavior 1318 may be selected. Otherwise, if an ERROR is detected at check 1504, a recovery 1542 behavior 1318 may be selected. Otherwise, if a low battery status is detected at check 1506, an approach charging station 1558 behavior 1318 may be selected. Otherwise, if an upgrade running state is detected at check 1508, a sleep 1560 behavior 1318 may be selected. Otherwise, if a robot start status is detected at check 1510, aDocket No. FSP2375PCTstart action 1544 behavior 1318 may be selected. Otherwise, if a robot shutdown status is detected at check 1512, a shutdown action 1546 behavior 1318 may be selected. After the status checks 1314 have been performed, they may be followed by event checks 1316.

[0178] The event check 1316 algorithm may begin by determining if a dump failed event is detected at check 1514, and if so, a recovery 1542 behavior 1318 may be selected. Otherwise, if a dump success event is detected at check 1516, a search 1548 behavior 1318 may be selected. Otherwise, if a pickup failed event is detected at check 1518, a recovery 1542 behavior 1318 may be selected. Otherwise, if a pickup success event is detected at check 1520, a search 1548 behavior 1318 may be selected. Otherwise, if a bin dock failed event is detected at check 1522, a recovery 1542 behavior 1318 may be selected. Otherwise, if a bin dock success event is detected check 1524, a dump 1556 behavior 1318 may be selected. Otherwise, if a charging dock failed event is detected at check 1526, a recovery 1542 behavior 1318 may be selected. Otherwise, if a charging doc success event is detected at check 1528, a shutdown action 1546 behavior 1318 may be selected. Otherwise, if a shovel full event is detected at check 1530, a search 1548 behavior 1318 may be selected. Otherwise, if a bin available event is detected at check 1532, an approach bin 1554 behavior 1318 may be selected. Otherwise, if a tidyable object available event is detected at check 1534, an approach tidyable object 1550 behavior 1318 may be selected. In one embodiment, the approach tidyable object 1550 behavior 1318 may be immediately followed by a pickup 1552 behavior 1318. Otherwise, if a cleanup complete event is detected at check 1536, an approach charging station 1558 behavior 1318 may be selected. Otherwise, if a cleanup start event is detected at check 1538, a search 1548 behavior 1318 may be selected. Otherwise, a sleep 1560 behavior 1318 may be selected.

[0179] This description is intended to illustrate and not limit the operation of the robot brain 1300 within a robot as disclosed herein. One of skill in the art will readily apprehend that the checks listed may be performed in some other order and that more or fewer checks than are illustrated here may be performed, resulting in the selection of more, fewer, or other behaviors than are included in behaviors 1318.

[0180] FIGs. 16-28 provide exemplary visual representations of data input to, processed in, and output by the perception logic 1100 and navigation logic 1200 embodiments herein. FIG.16 illustrates a detected image of a scene 1600 in accordance with one embodiment. The detected image of a scene 1600 comprises a floor 1602, a rug 1604, a furniture 1606, a person 1608, and toys 1610, described in additional detail with reference to FIG. 11.Docket No. FSP2375PCT

[0181] FIG. 17 illustrates classified 2D bounding boxes 1700 in accordance with one embodiment. The classified 2D bounding boxes 1700 comprises an obstacle 1702, a dynamic object 1704, and a tidyable object 1706, described in additional detail with reference to FIG.11.

[0182] FIG. 18 illustrates a multi-class segmentation map 1800 in accordance with one embodiment. The multi-class segmentation map 1800 comprises a floor 1602 and a rug 1604, described in additional detail with reference to FIG. 11.

[0183] FIG. 19 illustrates drivable area boundary masks with semantic information 1900 in accordance with one embodiment. The drivable area boundary masks with semantic information 1900 comprises a floor 1602, a rug 1604, a floor boundary 1902, an other relevant boundary 1904, and an other relevant region 1906, described in additional detail with reference to FIG. 11.

[0184] FIG. 20 illustrates a semantically rich boundary map 2000 in accordance with one embodiment, described in additional detail with reference to FIG. 11.

[0185] FIG. 21 illustrates pseudo-laserscan / point cloud with semantic label classes 2100 in accordance with one embodiment, described in additional detail with reference to FIG. 11.

[0186] FIG. 22 illustrates scene layout map 2200 in accordance with one embodiment, described in additional detail with reference to FIG. 11.

[0187] FIG. 23 illustrates local occupancy map 2300 in accordance with one embodiment, described in additional detail with reference to FIG. 11.

[0188] FIG. 24 illustrates detected objects in a scene 2400 in accordance with one embodiment, described in additional detail with reference to FIG. 12.

[0189] FIG. 25 illustrates a SLAM global map 2500 in accordance with one embodiment. The SLAM global map 2500 comprises a 2D occupancy grid representation 2502, a lidar data 2504, a scene map data 2506, a global plan 2508, and a real-time location information of the robot 2510, described in additional detail with reference to FIG. 12.

[0190] FIG. 26 illustrates a navigation global map 2600 in accordance with one embodiment. The navigation global map 2600 comprises an improved 2D occupancy grid representation 2602, a lidar data 2604, a scene map data 2606, and a real-time location information of the robot 2608, described in additional detail with reference to FIG. 12.Docket No. FSP2375PCT

[0191] FIG. 27 illustrates an inflated local map 2700 in accordance with one embodiment. The inflated local map 2700 comprises a 2D occupancy grid representation 2702, a buffer region 2704, a lidar data 2706, a scene map data 2708, a local plan with an integrated global mapper logic 2710, a global plan with an integrated global mapper logic 2712, and a real-time location information of the robot 2714, described in additional detail with reference to FIG. 12.

[0192] FIG. 28 illustrates an inflated global map 2800 in accordance with one embodiment. The inflated global map 2800 comprises a 2D occupancy grid representation 2802, a buffer region 2804, a lidar data 2806, a scene map data 2808, a global plan 2810, and a real-time location information of the robot 2812, described in additional detail with reference to FIG. 12.

[0193] FIG. 29 depicts an embodiment of a robotic control system 2900 to implement components and process steps of the systems described herein. Some or all portions of the robotic control system 2900 and its operational logic may be contained within the physical components of a robot and / or within a cloud server in communication with the robot and / or within the physical components of a user's mobile computing device, such as a smartphone, tablet, laptop, personal digital assistant, or other such mobile computing devices. In one embodiment, aspects of the robotic control system 2900 on a cloud server and / or user's mobile computing device may control more than one robot at a time, allowing multiple robots to work in concert within a working space.

[0194] Input devices 2904 (e.g., of a robot or companion device such as a mobile phone or personal computer) comprise transducers that convert physical phenomena into machine internal signals, typically electrical, optical, or magnetic signals. Signals may also be wireless in the form of electromagnetic radiation in the radio frequency (RF) range but also potentially in the infrared or optical range. Examples of input devices 2904 are contact sensors which respond to touch or physical pressure from an object or proximity of an object to a surface, mice which respond to motion through space or across a plane, microphones which convert vibrations in the medium (typically air) into device signals, scanners which convert optical patterns on two or three-dimensional objects into device signals. The signals from the input devices 2904 are provided via various machine signal conductors (e.g., buses or network interfaces) and circuits to memory 2906.

[0195] The memory 2906 is typically what is known as a first- or second-level memory device, providing for storage (via configuration of matter or states of matter) of signals received from the input devices 2904, instructions and information for controlling operation ofDocket No. FSP2375PCTthe central processing unit or processor 2902, and signals from storage devices 2910. The memory 2906 and / or the storage devices 2910 may store computer-executable instructions and thus forming logic 2914 that when applied to and executed by the processor 2902 implement embodiments of the processes disclosed herein. "Logic" refers to machine memory circuits and non-transitory machine readable media comprising machine-executable instructions (software and firmware), and / or circuitry (hardware) which by way of its material and / or material-energy configuration comprises control and / or procedural signals, and / or settings and values (such as resistance, impedance, capacitance, inductance, current / voltage ratings, etc.), that may be applied to influence the operation of a device. Magnetic media, electronic circuits, electrical and optical memory (both volatile and nonvolatile), and firmware are examples of logic. Logic specifically excludes pure signals or software per se (however does not exclude machine memories comprising software and thereby forming configurations of matter). Logic 2914 may include portions of a computer program, along with configuration data, that are run by the processor 2902 or another processor. Logic 2914 may include one or more machine learning models 2916 used to perform the disclosed actions. In one embodiment, portions of the logic 2914 may also reside on a mobile or desktop computing device accessible by a user to facilitate direct user control of the robot.

[0196] Information stored in the memory 2906 is typically directly accessible to the processor 2902 of the device. Signals input to the device cause the reconfiguration of the internal material / energy state of the memory 2906, creating in essence a new machine configuration, influencing the behavior of the robotic control system 2900 by configuring the processor 2902 with control signals (instructions) and data provided in conjunction with the control signals.

[0197] Second- or third-level storage devices 2910 may provide a slower but higher capacity machine memory capability. Examples of storage devices 2910 are hard disks, optical disks, large-capacity flash memories or other non-volatile memory technologies, and magnetic memories.

[0198] In one embodiment, memory 2906 may include virtual storage accessible through a connection with a cloud server using the network interface 2912, as described below. In such embodiments, some or all of the logic 2914 may be stored and processed remotely.

[0199] The processor 2902 may cause the configuration of the memory 2906 to be altered by signals in storage devices 2910. In other words, the processor 2902 may cause data andDocket No. FSP2375PCTinstructions to be read from storage devices 2910 in the memory 2906 which may then influence the operations of processor 2902 as instructions and data signals, and which may also be provided to the output devices 2908. The processor 2902 may alter the content of the memory 2906 by signaling to a machine interface of memory 2906 to alter the internal configuration and then converted signals to the storage devices 2910 alter its material internal configuration. In other words, data and instructions may be backed up from memory 2906, which is often volatile, to storage devices 2910, which are often non-volatile.

[0200] Output devices 2908 are transducers that convert signals received from the memory 2906 into physical phenomena such as vibrations in the air, patterns of light on a machine display, vibrations (i.e., haptic devices), or patterns of ink or other materials (i.e., printers and 3-D printers).

[0201] The network interface 2912 receives signals from the memory 2906 and converts them into electrical, optical, or wireless signals to other machines, typically via a machine network. The network interface 2912 also receives signals from the machine network and converts them into electrical, optical, or wireless signals to the memory 2906. The network interface 2912 may allow a robot to communicate with a cloud server 2922 containing logic 2914 and hosting or in communication with the cloud-based teacher robot task control model 106, in communication with the computing device 122, other robots, and other network-supported devices, and providing cloud-based data processor 118.

[0202] In one embodiment, a global database 2918 may provide data storage available across the devices that comprise or are supported by the robotic control system 2900. The global database 2918 may include maps, robotic instruction algorithms, robot state information, static, movable, and tidyable object reidentification fingerprints, labels, and other data associated with known static, movable, and tidyable object reidentification fingerprints, or other data supporting the implementation of the disclosed solution. The global database 2918 may be a single data structure or may be distributed across more than one data structure and storage platform, as may best suit an implementation of the disclosed solution. In one embodiment, the global database 2918 is coupled to other components of the robotic control system 2900 through a wired or wireless network, and in communication with the network interface 2912.

[0203] In one embodiment, a robot instruction database 2920 may provide data storage available across the devices that comprise or are supported by the robotic control system 2900. The robot instruction database 2920 may include the programmatic routines that direct specificDocket No. FSP2375PCTactuators of the tidying robot, such as are described previously, to actuate and cease actuation in sequences that allow the tidying robot to perform individual and aggregate motions to complete tasks.

[0204] FIG. 30 illustrates sensor input analysis 3000 in accordance with one embodiment. Sensor input analysis 3000 may inform the robot 700 of the dimensions of its immediate environment 3002 and the location of itself and other objects within that environment 3002.

[0205] The robot 700 as previously described includes a sensing system 706. This sensing system 706 may include at least one of cameras 3004, IMU sensors 3006, lidar sensor 3008, odometry 3010, and actuator force feedback sensor 3012. These sensors may capture data describing the environment 3002 around the robot 700.

[0206] Image data 3014 from the cameras 3004 may be used for object detection and classification 3016. Object detection and classification 3016 may be performed by algorithms and models configured within the robotic control system 2900 of the robot 700. In this manner, the characteristics and types of objects in the environment 3002 may be determined.

[0207] Image data 3014, object detection and classification 3016 data, and other sensor data 3018 may be used for a global / local map update 3020. The global and / or local map may be stored by the robot 700 and may represent its knowledge of the dimensions and objects within its decluttering environment 3002. This map may be used in navigation and strategy determination associated with decluttering tasks.

[0208] The robot 700 may use a combination of camera 3004, lidar sensor 3008 and the other sensors to maintain a global or local area map of the environment and to localize itself within that. Additionally, the robot may perform object detection and object classification and may generate visual re-identification fingerprints for each object. The robot may utilize stereo cameras along with a machine learning / neural network software architecture (e.g., semisupervised or supervised convolutional neural network) to efficiently classify the type, size and location of different objects on a map of the environment.

[0209] The robot may determine the relative distance and angle to each object. The distance and angle may then be used to localize objects on the global or local area map. The robot may utilize both forward and backward facing cameras to scan both to the front and to the rear of the robot.

[0210] Image data 3014, object detection and classification 3016 data, other sensor data 3018, and global / local map update 3020 data may be stored as observations, current robot state,Docket No. FSP2375PCTcurrent object state, and sensor data 3022. The observations, current robot state, current object state, and sensor data 3022 may be used by the robotic control system 2900 of the robot 700 in determining navigation paths and task strategies.

[0211] FIG. 31 illustrates an image processing routine 3100 in accordance with one embodiment. Detected images of a scene 1600 captured by the robot sensing system 1002 may undergo segmentation, such that areas of the segmented image 5602 may be identified as different objects, and those objects may be classified. Classified objects may then undergo perspective transform 5604, such that a map, as shown by the top-down view at the bottom, may be updated with objects detected through segmentation of the image.

[0212] The robot 700 illustrated with respect to FIGs. 7A-9B may include one or more processors as part of a robotic control system such as that illustrated in FIG. 29, or may be in communication with processing utilities available on other local devices in communication with the robot or through cloud computing. The robot may include sensors as described above. The sensors may provide sensor data to the robotic control system. In one embodiment, this may be image data, such as the detected image of a scene 1600 of this figure.

[0213] The processors of the robotic control system may perform a segmentation step to determine separate objects or entities in its surroundings, as well as the floor those entities may rest upon, which the robot may need to traverse. This may result in a segmented image 3102. This view may also include classifications of the objects detected, as is indicated here by the different colors. The robot may determine that portions of the image represent itself (red, orange, teal, and blue), toys (yellow and beige), furniture (black), and the floor (gray). This segmentation may be performed by a perception logic, such as the perception logic 1100 described in additional detail with respect to FIG. 11. In one embodiment, a detection logic 1102 in the perception logic 1100 may detect and apply bounding boxes and categorizations to objects detected in the detected image of a scene 1600. A scene segmentation logic 1110 of the perception logic 1100 may perform floor segmentation to identify a ground plane from the detected image of a scene 1600.

[0214] The segmented image 3102, such as one delineating detected objects and ground plane, and in some embodiments the classifications contained therein, may undergo a perspective transformation. Such a transform may be performed on an edge detection from a floor boundary mask, using mathematical algorithms applied to the segmented detections based on the geometry of the robot. These may be performed in some embodiments by a top viewDocket No. FSP2375PCTtransformation logic 1120 that resides in a mapping logic 1116 of a perception logic 1100, as described with regard to FIG. 11. In this manner, the robot may create or update a perspective transform 3104 of the objects, entities, and structures in the robot's working space. The perspective transform 3104 may be instrumental in instructing additional control engines in a robotic control system, such as a navigation logic 1200, a robot brain 1300, and an action logic 1400, as is described in additional detail with respect to previously described figures.

[0215] FIG. 32 illustrates a video-feed segmentation routine 3200 in accordance with one embodiment. Although the example video-feed segmentation routine 3200 depicts a particular sequence of operations, the sequence may be altered without departing from the scope of the present disclosure. For example, some of the operations depicted may be performed in parallel or in a different sequence that does not materially affect the function of the video-feed segmentation routine 3200. In other examples, different components of an example device or system that implements the video-feed segmentation routine 3200 may perform functions at substantially the same time or in a specific sequence.

[0216] According to some examples, the method includes receiving and processing live video with depth at block 3202. The live video feed may capture an environment to be tidied. For example, a mobile computing device 3208 may be configured to receive and process live video with depth using a camera 1004 configured as part of the mobile computing device 3208 in conjunction with the robotic control system 2900. Mobile computing devices 3208 may include smartphones, tablets, personal digital assistants, laptops, and the robot 700 disclosed herein and described in detail with respect to FIGs. 7A-9B above.

[0217] This live video may be used to begin mapping the environment to be tidied, and to support the configuration and display of an augmented reality (AR) user interface.Alternatively, the tidying robot previously disclosed may be configured to receive and process live video with depth using their cameras camera 1004 in conjunction with the robotic control system 2900. This may support the robot's initialization, configuration, and operation as disclosed herein. The live video feed may include detected images of a scene 1600 across the environment to be tidied. These may be processed to display an augmented reality view to a user on a global map of the environment to be tidied.

[0218] According to some examples, the method includes running a panoptic segmentation model 3210 to assign labels at block 3204. For example, the panoptic segmentation model 3210 illustrated in FIG. 32 may run a model to assign labels. The model may assign a semantic labelDocket No. FSP2375PCT(such as an object type), an instance identifier, and a movability attribute (such as static, movable, and tidyable) for each pixel in a detected image of a scene 1600 (such as is displayed in a frame of captured video). The panoptic segmentation model 3210 may be configured as part of the logic 2914 of the robotic control system 2900 in one embodiment. The panoptic segmentation model 3210 may in this manner produce a segmented image 3212 for each detected image of a scene 1600. Elements detected in the segmented image 3212 may in one embodiment be labeled as shown:a. floorb. rugc. bedframed. nightstande. drawerf. bedspreadg. boxh. lampi. booksj. picturek. walll. curtainsm. headboardn. pillowo. stuffed animalp. painting

[0219] According to some examples, the method includes separating the segmented image into static objects 3216, movable objects 3218, and tidyable objects 3220 at block 3206. For example, the robotic control system 2900 illustrated in FIG. 29 may separate static, movable, and tidyable objects. Using the segmented image 3212 and assigned labels, static structures in the represented scene, such as floors, walls, and large furniture, may be separated out as static objects 3216 from movable objects 3218 like chairs, doors, and rugs, and tidyable objects 3220 such as toys, books, and clothing. Upon completion of the video-feed segmentation routine 3200, the mobile computing device and robotic control system may act to identify objects separated into static objects, movable objects, and tidyable objects 3214.Docket No. FSP2375PCT

[0220] The following figures set forth, without limitation, exemplary cloud-based systems that may be used to implement at least one embodiment.

[0221] In at least one embodiment, cloud computing is a style of computing in which dynamically scalable and often virtualized resources are provided as a service over the Internet. In at least one embodiment, users need not have knowledge of, expertise in, or control over technology infrastructure, which may be referred to as “in the cloud,” that supports them. In at least one embodiment, cloud computing incorporates infrastructure as a service, platform as a service, software as a service, and other variations that have a common theme of reliance on the Internet for satisfying the computing needs of users. In at least one embodiment, a typical cloud deployment, such as in a private cloud (e.g., enterprise network), or a data center in a public cloud (e.g., Internet) may consist of thousands of servers (or alternatively, virtual machines (VMs)), hundreds of Ethernet, Fiber Channel or Fiber Channel over Ethernet (FCoE) ports, switching and storage infrastructure, etc. In at least one embodiment, cloud may also consist of network services infrastructure like IPsec virtual private network (VPN) hubs, firewalls, load balancers, wide area network (WAN) optimizers etc. In at least one embodiment, remote subscribers may access cloud applications and services securely by connecting via a VPN tunnel, such as an IPsec VPN tunnel.

[0222] In at least one embodiment, cloud computing is a model for enabling convenient, on-demand network access to a shared pool of configurable computing resources (e g., networks, servers, storage, applications, and services) that may be rapidly provisioned and released with minimal management effort or service provider interaction.

[0223] In at least one embodiment, cloud computing is characterized by on-demand self-service, in which a consumer may unilaterally provision computing capabilities, such as server time and network storage, as needed automatically without need for human interaction with each service's provider. In at least one embodiment, cloud computing is characterized by broad network access, in which capabilities are available over a network and accessed through standard mechanisms that promote use by heterogeneous thin or thick client platforms (e.g., mobile phones, laptops, and personal digital assistants (PDAs)). In at least one embodiment, cloud computing is characterized by resource pooling, in which a provider's computing resources are pooled to serve multiple consumers using a multi-tenant model, with different physical and virtual resources dynamically assigned and reassigned according to consumer demand. In at least one embodiment, there is a sense of location independence in that aDocket No. FSP2375PCTcustomer generally has no control or knowledge over an exact location of provided resources, but may be able to specify location at a higher level of abstraction (e.g., country, state, or datacenter). In at least one embodiment, examples of resources include storage, processing, memory, network bandwidth, and virtual machines. In at least one embodiment, cloud computing is characterized by rapid elasticity, in which capabilities may be rapidly and elastically provisioned, in some cases automatically, to quickly scale out and rapidly released to quickly scale in. In at least one embodiment, to a consumer, capabilities available for provisioning often appear to be unlimited and may be purchased in any quantity at any time. In at least one embodiment, cloud computing is characterized by measured service, in which cloud systems automatically control and optimize resource use by leveraging a metering capability at some level of abstraction appropriate to a type of service (e.g., storage, processing, bandwidth, and active user accounts). In at least one embodiment, resource usage may be monitored, controlled, and reported providing transparency for both a provider and consumer of a utilized service.

[0224] In at least one embodiment, cloud computing may be associated with various services. In at least one embodiment, cloud Software as a Service (SaaS) may refer to a service in which a capability provided to a consumer is to use a provider's applications running on a cloud infrastructure. In at least one embodiment, applications are accessible from various client devices through a thin client interface such as a web browser (e.g., web-based email). In at least one embodiment, the consumer does not manage or control underlying cloud infrastructure including network, servers, operating systems, storage, or even individual application capabilities, with a possible exception of limited user-specific application configuration settings.

[0225] In at least one embodiment, cloud Platform as a Service (PaaS) may refer to a service in which capability is provided to a consumer to deploy onto the cloud infrastructure consumer-created or acquired applications created using programming languages and tools supported by a provider. In at least one embodiment, a consumer does not manage or control underlying cloud infrastructure including networks, servers, operating systems, or storage, but has control over deployed applications and possibly application hosting environment configurations.

[0226] In at least one embodiment, cloud Infrastructure as a Service (laaS) may refer to a service in which a capability provided to a consumer is to provision processing, storage, networks, and other fundamental computing resources where a consumer is able to deploy andDocket No. FSP2375PCTrun arbitrary software, which may include operating systems and applications. In at least one embodiment, a consumer does not manage or control underlying cloud infrastructure, but has control over operating systems, storage, deployed applications, and possibly limited control of select networking components (e.g., host firewalls).

[0227] In at least one embodiment, cloud computing may be deployed in various ways. In at least one embodiment, a private cloud may refer to a cloud infrastructure that is operated solely for an organization. In at least one embodiment, a private cloud may be managed by an organization or a third party and may exist on-premises or off-premises. In at least one embodiment, a community cloud may refer to a cloud infrastructure that is shared by several organizations and supports a specific community that has shared concerns (e.g., mission, security, policy, and compliance considerations). In at least one embodiment, a community cloud may be managed by organizations or a third party and may exist on-premises or off-premises. In at least one embodiment, a public cloud may refer to a cloud infrastructure that is made available to the general public or a large industry group and is owned by an organization providing cloud services. In at least one embodiment, a hybrid cloud may refer to a cloud infrastructure that is a composition of two or more clouds (private, community, or public) that remain unique entities but are bound together by standardized or proprietary technology that supports data and application portability (e.g., cloud bursting for load-balancing between clouds). In at least one embodiment, a cloud computing environment is service-oriented with a focus on statelessness, low coupling, modularity, and semantic interoperability.

[0228] FIG. 33 illustrates one or more components of a system environment 3300 in which services may be offered as third-party network services, in accordance with at least one embodiment. In at least one embodiment, a third-party network may be referred to as a cloud, cloud network, cloud computing network, and / or variations thereof. In at least one embodiment, system environment 3300 includes one or more client computing devices 3304, 3306, and 3308 that may be used by users to interact with a third-party network infrastructure system 3302 that provides third-party network services, which may be referred to as cloud computing services. In at least one embodiment, third-party network infrastructure system 3302 may comprise one or more computers and / or servers.

[0229] It may be appreciated that third-party network infrastructure system 3302 depicted in FIG. 33 may have other components than those depicted. Further, FIG. 33 depicts an embodiment of a third-party network infrastructure system. In at least one embodiment, third-Docket No. FSP2375PCTparty network infrastructure system 3302 may have more or fewer components than depicted in FIG. 33, may combine two or more components, or may have a different configuration or arrangement of components.

[0230] In at least one embodiment, client computing devices 3304, 3306, and 3308 may be configured to operate a standard client application such as a web browser, a proprietary client application, or some other application, which may be used by a user of a client computing device to interact with third-party network infrastructure system 3302 to use services provided by third-party network infrastructure system 3302. Although exemplary system environment 3300 is shown with three client computing devices, any number of client computing devices may be supported. In at least one embodiment, other devices such as devices with sensors, etc. may interact with third-party network infrastructure system 3302. In at least one embodiment, network 3310 may facilitate communications and exchange of data between client computing devices 3304, 3306, and 3308 and third-party network infrastructure system 3302.

[0231] In at least one embodiment, services provided by third-party network infrastructure system 3302 may include a host of services that are made available to users of a third-party network infrastructure system on demand. In at least one embodiment, various services may also be offered including, without limitation, online data storage and backup solutions, Webbased e-mail services, hosted office suites and document collaboration services, database management and processing, managed technical support services, and / or variations thereof. In at least one embodiment, services provided by a third-party network infrastructure system may dynamically scale to meet the needs of its users.

[0232] In at least one embodiment, a specific instantiation of a service provided by third-party network infrastructure system 3302 may be referred to as a “service instance.” In at least one embodiment, in general, any service made available to a user via a communication network, such as the Internet, from a third-party network service provider's system is referred to as a “third-party network service.” In at least one embodiment, in a public third-party network environment, servers and systems that make up a third-party network service provider's system are different from a customer's own on-premises servers and systems. In at least one embodiment, a third-party network service provider's system may host an application, and a user may, via a communication network such as the Internet, on demand, order and use an application.Docket No. FSP2375PCT

[0233] In at least one embodiment, a service in a computer network third-party network infrastructure may include protected computer network access to storage, a hosted database, a hosted web server, a software application, or other service provided by a third-party network vendor to a user. In at least one embodiment, a service may include password-protected access to remote storage on a third-party network through the Internet. In at least one embodiment, a service may include a web service-based hosted relational database and a script-language middleware engine for private use by a networked developer. In at least one embodiment, a service may include access to an email software application hosted on a third-party network vendor's website.

[0234] In at least one embodiment, third-party network infrastructure system 3302 may include a suite of applications, middleware, and database service offerings that are delivered to a customer in a self-service, subscription-based, elastically scalable, reliable, highly available, and secure manner. In at least one embodiment, third-party network infrastructure system 3302 may also provide “big data” related computation and analysis services. In at least one embodiment, the term “big data” is generally used to refer to extremely large data sets that may be stored and manipulated by analysts and researchers to visualize large amounts of data, detect trends, and / or otherwise interact with data. In at least one embodiment, big data and related applications may be hosted and / or manipulated by an infrastructure system on many levels and at different scales. In at least one embodiment, tens, hundreds, or thousands of processors linked in parallel may act upon such data in order to present it or simulate external forces on data or what it represents. In at least one embodiment, these data sets may involve structured data, such as that organized in a database or otherwise according to a structured model, and / or unstructured data (e.g., emails, images, data blobs (binary large objects), web pages, complex event processing). In at least one embodiment, by leveraging the ability of an embodiment to relatively quickly focus more (or fewer) computing resources upon an objective, a third-party network infrastructure system may be better available to carry out tasks on large data sets based on demand from a business, government agency, research organization, private individual, group of like-minded individuals or organizations, or other entity.

[0235] In at least one embodiment, third-party network infrastructure system 3302 may be adapted to automatically provision, manage and track a customer's subscription to services offered by third-party network infrastructure system 3302. In at least one embodiment, third-party network infrastructure system 3302 may provide third-party network services viaDocket No. FSP2375PCTdifferent deployment models. In at least one embodiment, services may be provided under a public third-party network model in which third-party network infrastructure system 3302 is owned by an organization selling third-party network services, and services are made available to the general public or different industry enterprises. In at least one embodiment, services may be provided under a private third-party network model in which third-party network infrastructure system 3302 is operated solely for a single organization and may provide services for one or more entities within an organization. In at least one embodiment, third-party network services may also be provided under a community third-party network model in which third-party network infrastructure system 3302 and services provided by third-party network infrastructure system 3302 are shared by several organizations in a related community. In at least one embodiment, third-party network services may also be provided under a hybrid third-party network model, which is a combination of two or more different models.

[0236] In at least one embodiment, services provided by third-party network infrastructure system 3302 may include one or more services provided under Software as a Service (SaaS) category, Platform as a Service (PaaS) category, Infrastructure as a Service (laaS) category, or other categories of services including hybrid services. In at least one embodiment, a customer, via a subscription order, may order one or more services provided by third-party network infrastructure system 3302. In at least one embodiment, third-party network infrastructure system 3302 then performs processing to provide services in a customer's subscription order.

[0237] In at least one embodiment, services provided by third-party network infrastructure system 3302 may include, without limitation, application services, platform services, and infrastructure services. In at least one embodiment, application services may be provided by a third-party network infrastructure system via a SaaS platform. In at least one embodiment, the SaaS platform may be configured to provide third-party network services that fall under the SaaS category. In at least one embodiment, the SaaS platform may provide capabilities to build and deliver a suite of on-demand applications on an integrated development and deployment platform. In at least one embodiment, the SaaS platform may manage and control underlying software and infrastructure for providing SaaS services. In at least one embodiment, by utilizing services provided by a SaaS platform, customers may utilize applications executing on a third-party network infrastructure system. In at least one embodiment, customers may acquire application services without a need for customers to purchase separate licenses and support. In at least one embodiment, various different SaaS services may be provided. In at least oneDocket No. FSP2375PCTembodiment, examples include, without limitation, services that provide solutions for sales performance management, enterprise integration, and business flexibility for large organizations.

[0238] In at least one embodiment, platform services may be provided by third-party network infrastructure system 3302 via a PaaS platform. In at least one embodiment, the PaaS platform may be configured to provide third-party network services that fall under the PaaS category. In at least one embodiment, examples of platform services may include without limitation services that allow organizations to consolidate existing applications on a shared, common architecture, as well as an ability to build new applications that leverage shared services provided by a platform. In at least one embodiment, the PaaS platform may manage and control underlying software and infrastructure for providing PaaS services. In at least one embodiment, customers may acquire PaaS services provided by third-party network infrastructure system 3302 without a need for customers to purchase separate licenses and support.

[0239] In at least one embodiment, by utilizing services provided by a PaaS platform, customers may employ programming languages and tools supported by a third-party network infrastructure system and also control deployed services. In at least one embodiment, platform services provided by a third-party network infrastructure system may include database third-party network services, middleware third-party network services, and third-party network services. In at least one embodiment, database third-party network services may support shared service deployment models that allow organizations to pool database resources and offer customers a Database as a Service in the form of a database third-party network. In at least one embodiment, middleware third-party network services may provide a platform for customers to develop and deploy various business applications, and third-party network services may provide a platform for customers to deploy applications, in a third-party network infrastructure system.

[0240] In at least one embodiment, various different infrastructure services may be provided by an laaS platform in a third-party network infrastructure system. In at least one embodiment, infrastructure services facilitate management and control of underlying computing resources, such as storage, networks, and other fundamental computing resources for customers utilizing services provided by a SaaS platform and a PaaS platform.

[0241] In at least one embodiment, third-party network infrastructure system 3302 may also include infrastructure resources 3330 for providing resources used to provide various servicesDocket No. FSP2375PCTto customers of a third-party network infrastructure system. In at least one embodiment, infrastructure resources 3330 may include pre-integrated and optimized combinations of hardware, such as servers, storage, and networking resources to execute services provided by a Paas platform and a Saas platform, and other resources.

[0242] In at least one embodiment, resources in third-party network infrastructure system 3302 may be shared by multiple users and dynamically re-allocated per demand. In at least one embodiment, resources may be allocated to users in different time zones. In at least one embodiment, third-party network infrastructure system 3302 may allow a first set of users in a first time zone to utilize resources of a third-party network infrastructure system for a specified number of hours and then allow a re-allocation of the same resources to another set of users located in a different time zone, thereby maximizing utilization of resources.

[0243] In at least one embodiment, a number of internal shared services 3332 may be provided that are shared by different components or logic modules of third-party network infrastructure system 3302 to support the provision of services by third-party network infrastructure system 3302. In at least one embodiment, these internal shared services may include, without limitation, a security and identity service, an integration service, an enterprise repository service, an enterprise manager service, a virus scanning and white list service, a high availability, backup and recovery service, service for enabling third party network support, an email service, a notification service, a file transfer service, and / or variations thereof.

[0244] In at least one embodiment, third-party network infrastructure system 3302 may provide comprehensive management of third-party network services (e.g., SaaS, PaaS, and laaS services) in a third-party network infrastructure system. In at least one embodiment, third-party network management functionality may include capabilities for provisioning, managing, and tracking a customer's subscription received by third-party network infrastructure system 3302, and / or variations thereof.

[0245] In at least one embodiment, as depicted in FIG. 33, third-party network management functionality may be provided by one or more logic modules, such as an order management logic 3320, an order orchestration logic 3322, an order provisioning logic 3324, an order management and monitoring logic 3326, and an identity management logic 3328. In at least one embodiment, these logic modules may include or be provided using one or more computers and / or servers, which may be general-purpose computers, specialized server computers, server farms, server clusters, or any other appropriate arrangement and / or combination.Docket No. FSP2375PCT

[0246] In at least one embodiment, at a service request step 3334, a customer using a client device, such as client computing devices 3304, 3306, or 3308, may interact with third-party network infrastructure system 3302 by requesting one or more services provided by third-party network infrastructure system 3302 and placing an order for a subscription for one or more services offered by third-party network infrastructure system 3302. In at least one embodiment, a customer may access a third-party network User Interface (UI) such as third-party network UI 3312, third-party network UI 3314, and / or third-party network UI 3316 and place a subscription order via these UIs. In at least one embodiment, order information received by third-party network infrastructure system 3302 in response to a customer placing an order may include information identifying a customer and one or more services offered by a third-party network infrastructure system 3302 that a customer intends to subscribe to.

[0247] In at least one embodiment, at a storing information step 3336, order information received from a customer may be stored in an order database 3318. In at least one embodiment, if this is a new order, a new record may be created for an order. In at least one embodiment, order database 3318 may be one of several databases operated by third-party network infrastructure system 3302 and operated in conjunction with other system elements.

[0248] In at least one embodiment, at a forwarding information step 3338, order information may be forwarded to an order management logic 3320 that may be configured to perform billing and accounting functions related to an order, such as verifying an order, and upon verification, booking an order.

[0249] In at least one embodiment, at a communicating information step 3340, information regarding an order may be communicated to an order orchestration logic 3322 that is configured to orchestrate the provisioning of services and resources for an order placed by a customer. In at least one embodiment, order orchestration logic 3322 may use services of order provisioning logic 3324 for provisioning. In at least one embodiment, order orchestration logic 3322 supports the management of business processes associated with each order and applies business logic to determine whether an order may proceed to provisioning.

[0250] In at least one embodiment, at receiving a new order step 3342, upon receiving an order for a new subscription, order orchestration logic 3322 sends a request to order provisioning logic 3324 to allocate resources and configure resources needed to fulfill a subscription order. In at least one embodiment, an order provisioning logic 3324 supports an allocation of resources for services ordered by a customer. In at least one embodiment, an orderDocket No. FSP2375PCTprovisioning logic 3324 provides a level of abstraction between third-party network services provided by third-party network infrastructure system 3302 and a physical implementation layer that is used to provision resources for providing requested services. Tn at least one embodiment, this allows order orchestration logic 3322 to be isolated from implementation details, such as whether or not services and resources are actually provisioned in real-time or pre-provisioned and allocated / assigned upon request.

[0251] In at least one embodiment, at a service provided step 3344, once services and resources are provisioned, a notification may be sent to subscribing customers indicating that a requested service is now ready for use. In at least one embodiment, information (e.g., a link) may be sent to a customer that allows a customer to start using the requested services.

[0252] In at least one embodiment, at a notification step 3346, a customer's subscription order may be managed and tracked by an order management and monitoring logic 3326. Tn at least one embodiment, order management and monitoring logic 3326 may be configured to collect usage statistics regarding a customer's use of subscribed services. In at least one embodiment, statistics may be collected for the amount of storage used, the amount of data transferred, the number of users, the amount of system up time and system down time, and / or variations thereof.

[0253] In at least one embodiment, third-party network infrastructure system 3302 may include an identity management logic 3328 that is configured to provide identity services, such as access management and authorization services in third-party network infrastructure system 3302. In at least one embodiment, identity management logic 3328 may control information about customers who wish to utilize services provided by third-party network infrastructure system 3302. In at least one embodiment, such information may include information that authenticates the identities of such customers and information that describes which actions those customers are authorized to perform relative to various system resources (e.g., files, directories, applications, communication ports, memory segments, etc.). In at least one embodiment, identity management logic 3328 may also include management of descriptive information about each customer and about how and by whom that descriptive information may be accessed and modified.

[0254] FIG. 34 illustrates a computing environment 3400 including cloud computing environment 3402, in accordance with at least one embodiment. In at least one embodiment, cloud computing environment 3402 comprises one or more computer system s / servers 3404Docket No. FSP2375PCTwith which computing devices such as, personal digital assistant (PDA) or computing device 3406a, computing device 3406b, computing device 3406c, and / or computing device 3406d communicate. In at least one embodiment, this allows for infrastructure, platforms, and / or software to be offered as services from cloud computing environment 3402, so as to not require each client to separately maintain such resources. It is understood that the types of computing devices 3406a-3406d shown in FIG. 34 (a mobile or handheld device, a desktop computer, a laptop computer, and an automobile computer system) are intended to be illustrative, and that cloud computing environment 3402 may communicate with any type of computerized device over any type of network and / or network / addressable connection (e.g., using a web browser).

[0255] In at least one embodiment, a computer system / server 3404, which may be denoted as a cloud computing node, is operational with numerous other general purpose or special purpose computing system environments or configurations. In at least one embodiment, examples of computing systems, environments, and / or configurations that may be suitable for use with computer system / server 3404 include, but are not limited to, personal computer systems, server computer systems, thin clients, thick clients, hand-held or laptop devices, multiprocessor systems, microprocessor-based systems, set-top boxes, programmable consumer electronics, network personal computers (PCs), minicomputer systems, mainframe computer systems, and distributed cloud computing environments that include any of the above systems or devices, and / or variations thereof.

[0256] In at least one embodiment, computer system / server 3404 may be described in a general context of computer system-executable instructions, such as program modules, being executed by a computer system. In at least one embodiment, program modules include routines, programs, objects, components, logic, data structures, and so on, that perform particular tasks or implement particular abstract data types. In at least one embodiment, an exemplary computer system / server 3404 may be practiced in distributed cloud computing environments where tasks are performed by remote processing devices that are linked through a communications network. In at least one embodiment, in a distributed cloud computing environment, program modules may be located in both local and remote computer system storage media including memory storage devices.

[0257] FIG. 35 illustrates a set of functional abstraction layers 3500 provided by cloud computing environment 3402 (FIG. 34), in accordance with at least one embodiment. It may beDocket No. FSP2375PCTunderstood in advance that the components, layers, and functions shown in FIG. 35 are intended to be illustrative, and components, layers, and functions may vary.

[0258] In at least one embodiment, hardware and software layer 3502 includes hardware and software components. In at least one embodiment, examples of hardware components include mainframes, various RISC (Reduced Instruction Set Computer) architecture-based servers, various computing systems, supercomputing systems, storage devices, networks, networking components, and / or variations thereof. In at least one embodiment, examples of software components include network application server software, various application server software, various database software, and / or variations thereof.

[0259] In at least one embodiment, virtualization layer 3504 provides an abstraction layer from which the following exemplary virtual entities may be provided: virtual servers, virtual storage, virtual networks, including virtual private networks, virtual applications, virtual clients, and / or variations thereof.

[0260] In at least one embodiment, management layer 3506 provides various functions. In at least one embodiment, resource provisioning provides dynamic procurement of computing resources and other resources that are utilized to perform tasks within a cloud computing environment. In at least one embodiment, metering provides usage tracking as resources are utilized within a cloud computing environment, and billing or invoicing for consumption of these resources. In at least one embodiment, resources may comprise application software licenses. In at least one embodiment, security provides identity verification for users and tasks, as well as protection for data and other resources. In at least one embodiment, a user interface provides access to a cloud computing environment for both users and system administrators. In at least one embodiment, service level management provides cloud computing resource allocation and management such that the needed service levels are met. In at least one embodiment, Service Level Agreement (SLA) management provides pre-arrangement for, and procurement of, cloud computing resources for which a future need is anticipated in accordance with an SLA.

[0261] In at least one embodiment, workloads layer 3508 provides functionality for which a cloud computing environment is utilized. In at least one embodiment, examples of workloads and functions which may be provided from this layer include mapping and navigation, software development and management, educational services, data analytics and processing, transaction processing, and service delivery.Docket No. FSP2375PCT

[0262] FIG. 36 illustrates an Al system 3600 in accordance with one embodiment. The Al system 3600 may comprise a sensor data 3602, a text data 3604, a signal classifier 3610, a tokenizer 3618, a prompt composer 3622, an Al model 3624, an Al output 3628, a database 3612, a user 3606, and a model selector 3626. The Al system 3600 may in one embodiment be implemented on a single computing device such as the computer system / server 3404 described in additional detail below. In other embodiments, the elements of the Al system 3600 may be distributed across computer systems / servers 3404 interconnected in a cloud computing environment 3402 as described with respect to FIG. 34.

[0263] Input to the Al system 3600 may be in the form of sensor data 3602, text data 3604, user 3606, and other forms of data, as will be readily understood by one of ordinary skill in the art. The sensor data 3602 may be provided by sensors providing digital or analog readings. These sensors may be connected in a computing environment and may thus provide output sensor data 3602 for storage in various forms of computer memory and for processing by computer processors. Text data 3604 may be provided as input from various sources, including human users 3606 operating a text entry peripheral attached to a computational device, such as a keyboard, a microphone with its audio output processes by speech-to-text algorithms, etc. Text data 3604 may also include historical data stored in various memory structures. Users 3606 may also provide input to the Al system 3600 in the form of user selection data 3608 through a computational input / output (I / O) interface such as a mouse, keyboard, microphone, touchscreen, etc. This description is not intended to be limiting, and additional sources of sensor data 3602, text data 3604, and user selection data 3608 may readily occur to one of ordinary skill in the art.

[0264] Sensor data 3602 may be sent to a signal classifier 3610 in order to analyze and interpret the data provided by sensors. In one embodiment, the signal classifier 3610 may act as a specialized tokenizer trained or otherwise programed to recognize and interpret sensor signals and tokenize sensor data 3602 such that the phenomena detected by the sensors and recorded in the sensor data 3602 may be communicated in a manner interpretable by other elements of a computational system, such as the components of the Al system 3600 described here. A database 3612 may include stored data 3614 which may be made available to the signal classifier 3610 in order to classify current text sensor data 3602 quickly based on known classification so similar signals based on prior knowledge, programmed relationships, or Al system 3600 learnings. In one embodiment, the signal classifier 3610 may provide such signalDocket No. FSP2375PCTclassifications 3616 directly to the prompt composer 3622. In another embodiment, signal classifications 3616 may be sent for further tokenization by the tokenizer 3618. text data 3604 and user selection data 3608 from the user 3606 may also be sent to the tokenizer 3618. The tokenizer 3618 may develop input tokens 3620 from the various streams of input data as is described for the exemplary tokenizer 3700 illustrated in FIG. 37. The input tokens 3620 may be provided as input to the prompt composer 3622.

[0265] The prompt composer 3622 may receive the input tokens 3620 developed from the sensor data 3602, the text data 3604, the user selection data 3608, and other sources of data available to the Al system 3600 not pictured here. The prompt composer 3622 may also receive input in the form of stored data 3614 from one or more databases 3612. Using these inputs, the prompt composer 3622 may construct a single token, a series of tokens, a body of text, and / or a series of conditional or unconditional commands suitable to use as a prompt 3630 to one or more connected Al models 3624. For example, a series such as “conditional on command A success, send command B, else send command C” may be built and sent all at once given a specific data precondition, rather than being built and sent separately.

[0266] The prompt composer 3622 may generate tokens or other prompt elements that identify a requested or desired output modality (e.g., text, audio / visual, computer / robotic commands, etc.). The prompt composer 3622 may generate an embedding which may be provided separately from the prompt 3630 for use in an intermediate layer of the Al model 3624. The prompt composer 3622 may generate multiple tokenized sequences at once that constitute a series of conditional commands. In some embodiments, the prompt composer 3622 may utilize a formal prompt composition language such as Microsoft Guidance. In such a case, the composition language may utilize one or more formal structures that facilitate deterministic prompt composition as a function of mixed modality inputs. For example, the prompt composer 3622 may contain subroutines that process raw signal data, such as may be included in sensor data 3602 and user selection data 3608, and utilize this data to modify prompts 3630 in order to ensure specific types of Al outputs 3628.

[0267] In one embodiment, the prompt composer 3622 may be configured to identify among multiple available Al models 3624 the one best suited (specifically trained, historically successful, etc.) for addressing the need represented by the data in the prompt 3630. In this case, the prompt composer 3622 may provide a model identifier 3632 interpretable by a model selector 3626. The model selector 3626 may be configured to route the prompt 3630 to theDocket No. FSP2375PCTspecific Al model 3624 indicated by the model identifier 3632. In this manner, the Al system 3600 may be able to optimize model usage for the most quickly generated and most accurate Al output 3628.

[0268] Al models 3624 may include large language models (LLMs), a Generative Pre-trained Transformer (GPT) model, a generalist agent such as Gato, and other models such as may be readily anticipated by one of ordinary skill in the art. These Al model 3624 may be configured to supply Al outputs 3628 based on the prompt 3630. The Al output 3628 may be multimodal in some embodiments, and may take the form of text output, audio output, visual output, programmatic output such as computer or robotic commands, etc. Al output 3628 may be sent to users 3606, which may be the person providing the text data 3604 or user selection data 3608 inputs or other users similarly in contact with the Al system 3600 through a computational interface. The Al output 3628 may be sent to interconnected computing devices or systems 3636 such as these users 3606 may utilize. The Al output 3628 may be sent to robotic systems 3634 configured to perform automated tasks based on the Al output 3628.

[0269] FIG. 37 illustrates an exemplary tokenizer 3700 in accordance with one embodiment, data to be tokenized 3702 may be provided to the exemplary tokenizer 3700 in the form of a text string typed by a user. In one embodiment, the text string may be generated by performing voice-to-text conversion on an audio stream. The exemplary tokenizer 3700 may detect tokenizable elements 3704 within the data to be tokenized 3702. Each tokenizable element 3704 may be converted into a token 3706. The set of tokens 3706 created from the tokenizable elements 3704 of the data to be tokenized 3702 may be sent from the exemplary tokenizer 3700 as tokenized output 3708. The set of tokens 3706 may be such as are used to create the prompt FIG. 36.

[0270] For structured historical data such as plaintext, database, or web-based textual content, tokens may consist of the numerical indexes in an embedded or vectorized (e.g., word2vec or similar) representation of the text content such as are shown here. In some embodiments, a machine learning technique called an autoencoder may be utilized to transform plaintext inputs into high dimensional vectors that are suitable for indexing and tokenization ingestion by the prompt composer 3622 introduced with respect to FIG. 36.

[0271] In some embodiments, data to be tokenized may include audio, visual, or other multimodal data. For images, video, and similar visual data, tokenization may be performed using a convolution-based tokenizer such as a vision transformer. In some alternateDocket No. FSP2375PCTembodiments, multimodal data may be quantized and converted into tokens 3706 using a codebook. In yet other alternate embodiments, multimodal data may be directly encoded and for presentation to a language model as a vector space encoding. An exemplary system that utilizes this tokenizer strategy is Gato, a generalist agent capable of ingesting a mixture of discrete and continuous inputs, images, and text as tokens.

[0272] FIG. 38 illustrates the training and deployment of a deep neural network 3800, such as the basic deep neural network 4600 illustrated in FIG. 46, according to at least one embodiment. In at least one embodiment, untrained neural network 3806 is trained using a training dataset 3802. In at least one embodiment, training framework 3804 is a PyTorch framework, whereas in other embodiments, training framework 3804 is a TensorFlow, Boost, Caffe, Microsoft Cognitive Toolkit / CNTK, MXNet, Chainer, Keras, Deeplearning4j, or another training framework. In at least one embodiment, training framework 3804 trains an untrained neural network 3806 and allows it to be trained using processing resources described herein to generate a trained neural network 3808. In at least one embodiment, weights may be chosen randomly or by pre-training using a deep belief network. In at least one embodiment, training may be performed in either a supervised, partially supervised, or unsupervised manner.

[0273] In at least one embodiment, untrained neural network 3806 is trained using supervised learning, wherein training dataset 3802 includes an input paired with a desired output for the input, or where training dataset 3802 includes input having a known output and an output of untrained neural network 3806 is manually graded. In at least one embodiment, untrained neural network 3806 is trained in a supervised manner, processes inputs from training dataset 3802, and compares resulting outputs against a set of expected or desired outputs. In at least one embodiment, errors are then propagated back through untrained neural network 3806. In at least one embodiment, training framework 3804 adjusts weights that control untrained neural network 3806. In at least one embodiment, training framework 3804 includes tools to monitor how well untrained neural network 3806 is converging towards a model, such as trained neural network 3808, suitable to generating correct answers, such as in result 3812, based on input data such as a new dataset 3810. In at least one embodiment, training framework 3804 trains untrained neural network 3806 repeatedly while adjusting weights to refine an output of untrained neural network 3806 using a loss function and adjustment algorithm, such as stochastic gradient descent. In at least one embodiment, training framework 3804 trains untrained neural network 3806 until untrained neural network 3806 achieves the desiredDocket No. FSP2375PCTaccuracy. In at least one embodiment, trained neural network 3808 may then be deployed to implement any number of machine learning operations.

[0274] In at least one embodiment, untrained neural network 3806 is trained using unsupervised learning, wherein untrained neural network 3806 attempts to train itself using unlabeled data. In at least one embodiment, an unsupervised learning training dataset 3802 will include input data without any associated output data or “ground truth” data. In at least one embodiment, untrained neural network 3806 may learn groupings within training dataset 3802 and may determine how individual inputs are related to other data in the training dataset 3802. In at least one embodiment, unsupervised training may be used to generate a self-organizing map in a trained neural network 3808 capable of performing operations useful in reducing the dimensionality of the new dataset 3810. In at least one embodiment, unsupervised training may also be used to perform anomaly detection, which allows the identification of data points in new dataset 3810 that deviate from normal patterns of new dataset 3810.

[0275] In at least one embodiment, semi-supervised learning may be used, which is a technique in which training dataset 3802 includes a mix of labeled and unlabeled data. In at least one embodiment, training framework 3804 may be used to perform incremental learning, such as through transferred learning techniques. In at least one embodiment, incremental learning allows trained neural network 3808 to adapt to new dataset 3810 without forgetting knowledge instilled within trained neural network 3808 during initial training. A trained neural network 3808 such as the one described may be used as the basis for Al models and machine learning models such as may be used in computational systems to analyze complex data and provide results based on that analysis productive toward the improved knowledge or task action performance of people and computational and robotic systems.

[0276] The following figures set forth, without limitation, exemplary artificial intelligencebased systems that may be used to implement at least one embodiment.

[0277] FIG. 39A illustrates inference and / or training logic 3900a used to perform inferencing and / or training operations associated with one or more embodiments. Details regarding training logic / hardware structure 3910 are provided below in conjunction with FIG. 39A and / or FIG.39B.

[0278] In at least one embodiment, training logic / hardware structure 3910 may include, without limitation, code and / or data storage 3902 to store forward and / or output weight and / or input / output data, and / or other parameters to configure neurons or layers of a neural networkDocket No. FSP2375PCTtrained and / or used for inferencing in aspects of one or more embodiments. In at least one embodiment, training logic / hardware structure 3910 may include or be coupled to code and / or data storage 3902 to store graph code or other software to control the timing and / or order in which weight and / or other parameter information is to be loaded to configure logic, including integer and / or floating point units (collectively, arithmetic logic units (ALUs)). In at least one embodiment, code, such as graph code, loads weight or other parameter information into processor ALUs based on an architecture of a neural network to which such code corresponds. In at least one embodiment code and / or data storage 3902 stores weight parameters and / or input / output data of each layer of a neural network trained or used in conjunction with one or more embodiments during forward propagation of input / output data and / or weight parameters during training and / or inferencing using aspects of one or more embodiments. In at least one embodiment, any portion of code and / or data storage 3902 may be included with other on-chip or off-chip data storage, including a processor’s LI, L2, or L3 cache or system memory.

[0279] In at least one embodiment, any portion of code and / or data storage 3902 may be internal or external to one or more processors or other hardware logic devices or circuits. In at least one embodiment, code and / or data storage 3902 may be cache memory, dynamic randomly addressable memory (DRAM), static randomly addressable memory (SRAM), nonvolatile memory (e.g., flash memory), or other storage. In at least one embodiment, a choice of whether code and / or data storage 3902 is internal or external to a processor, for example, or comprising DRAM, SRAM, flash, or some other storage type may depend on available storage on-chip versus off-chip, latency needs of training and / or inferencing functions being performed, batch size of data used in inferencing and / or training of a neural network, or some combination of these factors.

[0280] In at least one embodiment, training logic / hardware structure 3910 may include, without limitation, a code and / or data storage 3906 to store backward and / or output weight and / or input / output data corresponding to neurons or layers of a neural network trained and / or used for inferencing in aspects of one or more embodiments. In at least one embodiment, code and / or data storage 3906 stores weight parameters and / or input / output data of each layer of a neural network trained or used in conjunction with one or more embodiments during backward propagation of input / output data and / or weight parameters during training and / or inferencing using aspects of one or more embodiments. In at least one embodiment, training logic / hardware structure 3910 may include or be coupled to code and / or data storage 3906 to store graph codeDocket No. FSP2375PCTor other software to control the timing and / or order in which weight and / or other parameter information is to be loaded to configure logic, including integer and / or floating point units (collectively, arithmetic logic units (ALUs)).

[0281] In at least one embodiment, code, such as graph code, causes loading of weight or other parameter information into processor ALUs based on an architecture of a neural network to which such code corresponds. In at least one embodiment, any portion of code and / or data storage 3906 may be included with other on-chip or off-chip data storage, including a processor’s LI, L2, or L3 cache or system memory. In at least one embodiment, any portion of code and / or data storage 3906 may be internal or external to one or more processors or other hardware logic devices or circuits. In at least one embodiment, code and / or data storage 3906 may be cache memory, DRAM, SRAM, non-volatile memory (e.g., flash memory), or other storage. In at least one embodiment, a choice of whether code and / or data storage 3906 is internal or external to a processor, for example, or comprising DRAM, SRAM, flash memory or some other storage type may depend on available storage on-chip versus off-chip, latency needs of training and / or inferencing functions being performed, batch size of data used in inferencing and / or training of a neural network, or some combination of these factors.

[0282] In at least one embodiment, code and / or data storage 3902 and code and / or data storage 3906 may be separate storage structures. In at least one embodiment, code and / or data storage 3902 and code and / or data storage 3906 may be a combined storage structure. In at least one embodiment, code and / or data storage 3902 and code and / or data storage 3906 may be partially combined and partially separate. In at least one embodiment, any portion of code and / or data storage 3902 and code and / or data storage 3906 may be included with other on-chip or off-chip data storage, including a processor’s LI, L2, or L3 cache or system memory.

[0283] In at least one embodiment, training logic / hardware structure 3910 may include, without limitation, one or more arithmetic logic units 3912, including integer and / or floating point units, to perform logical and / or mathematical operations based, at least in part on, or indicated by, training and / or inference code (e.g., graph code), a result of which may produce activations (e.g., output values from layers or neurons within a neural network) stored in an activation storage 3914 that are functions of input / output and / or weight parameter data stored in code and / or data storage 3902 and / or code and / or data storage 3906. In at least one embodiment, activations stored in activation storage 3914 are generated according to linear algebraic and or matrix-based mathematics performed by arithmetic logic units 3912 inDocket No. FSP2375PCTresponse to performing instructions or other code, wherein weight values stored in code and / or data storage 3906 and / or code and / or data storage 3902 are used as operands along with other values, such as bias values, gradient information, momentum values, or other parameters or hyperparameters, any or all of which may be stored in code and / or data storage 3906 or code and / or data storage 3902 or another storage on or off-chip.

[0284] In at least one embodiment, arithmetic logic units 3912 are included within one or more processors or other hardware logic devices or circuits, whereas in another embodiment, arithmetic logic units 3912 may be external to a processor or other hardware logic device or circuit that uses them (e.g., a co-processor). In at least one embodiment, arithmetic logic units 3912 may be included within a processor’s execution units or otherwise within a bank of ALUs accessible by a processor’s execution units either within the same processor or distributed between different processors of different types (e.g., central processing units, graphics processing units, fixed function units, etc ). In at least one embodiment, code and / or code and / or data storage 3902, code and / or data storage 3906, and activation storage 3914 may share a processor or other hardware logic device or circuit, whereas, in another embodiment, they may be in different processors or other hardware logic devices or circuits, or some combination of same and different processors or other hardware logic devices or circuits. In at least one embodiment, any portion of activation storage 3914 may be included with other on-chip or off-chip data storage, including a processor’s LI, L2, or L3 cache or system memory. Furthermore, inferencing and / or training code may be stored with other code accessible to a processor or other hardware logic or circuit and fetched and / or processed using a processor’s fetch, decode, scheduling, execution, retirement, and / or other logic circuits.

[0285] In at least one embodiment, activation storage 3914 may be cache memory, DRAM, SRAM, non-volatile memory (e.g., flash memory), or other storage. In at least one embodiment, activation storage 3914 may be completely or partially within or external to one or more processors or other logic circuits. In at least one embodiment, a choice of whether activation storage 3914 is internal or external to a processor, for example, or comprising DRAM, SRAM, flash memory or some other storage type may depend on available storage on-chip versus off-chip, latency needs of training and / or inferencing functions being performed, batch size of data used in inferencing and / or training of a neural network, or some combination of these factors.Docket No. FSP2375PCT

[0286] In at least one embodiment, the training logic / hardware structure 3910 illustrated in FIG. 39A may be used in conjunction with an application-specific integrated circuit (ASIC), such as a TensorFlow® Processing Unit from Google, an inference processing unit (FPU) from Graphcore™, or a Nervana® (e.g., “Lake Crest) processor from Intel Corp. In at least one embodiment, the training logic / hardware structure 3910 illustrated in FIG. 39A may be used in conjunction with CPU hardware, GPU hardware, or other hardware, such as field programmable gate arrays (FPGAs).

[0287] FIG. 39B illustrates inference and / or training logic 3900b, according to at least one embodiment. In at least one embodiment, training logic / hardware structure 3910 may include, without limitation, hardware logic in which computational resources are dedicated or otherwise exclusively used in conjunction with weight values or other information corresponding to one or more layers of neurons within a neural network. In at least one embodiment, the training logic / hardware structure 3910 illustrated in FIG. 39B may be used in conjunction with an application-specific integrated circuit (ASIC), such as TensorFlow® Processing Unit from Google, an inference processing unit (IPU) from Graphcore™, or a Nervana® (e.g., “Lake Crest”) processor from Intel Corp. In at least one embodiment, the training logic / hardware structure 3910 illustrated in FIG. 39B may be used in conjunction with CPU hardware, GPU hardware, or other hardware, such as field programmable gate arrays (FPGAs). In at least one embodiment, training logic / hardware structure 3910 includes, without limitation, code and / or data storage 3902 and code and / or data storage 3906, which may be used to store code (e g., graph code), weight values and / or other information, including bias values, gradient information, momentum values, and / or other parameter or hyperparameter information. In at least one embodiment illustrated in FIG. 39B, each of code and / or data storage 3902 and code and / or data storage 3906 is associated with a dedicated computational resource, such as computational hardware 3904 and computational hardware 3908, respectively. In at least one embodiment, each of computational hardware 3904 and computational hardware 3908 comprises one or more ALUs that perform mathematical functions, such as linear algebraic functions, on information stored in code and / or data storage 3902 and code and / or data storage 3906, respectively, the result of which is stored in activation storage 3914.

[0288] In at least one embodiment, each of code and / or data storage 3902 and 3906 and corresponding computational hardware 3904 and 3908, respectively, correspond to different layers of a neural network, such that resulting activation from one storage / computational pairDocket No. FSP2375PCT3902 / 3904 of code and / or data storage 3902 and computational hardware 3904 is provided as an input to a next storage / computational pair 3906 / 3908 of code and / or data storage 3906 and computational hardware 3908, in order to mirror a conceptual organization of a neural network. In at least one embodiment, each of the storage / computational pairs 3902 / 3904 and 3906 / 3908 may correspond to more than one neural network layer. In at least one embodiment, additional storage / computation pairs (not shown) subsequent to or in parallel with storage / computation pairs 3902 / 3904 and 3906 / 3908 may be included in training logic / hardware structure 3910.

[0289] FIG. 40 depicts an exemplary convolutional neural network 4000. The convolutional neural network 4000 comprises a three dimensional neuron configuration (width, height, depth), as depicted in convolutional layer 4002. Layers of the convolutional neural network 4000 transform a 3D volume of inputs to a 3D output volume of neuron activations. In this example, the input layer 4004 encodes the image, therefore its width and height are configured to the dimensions of the image, and the depth of the 104 is configured to three (e g., for Red, Green, and Blue channels). The convolutional layer 4002 further transforms the outputs of the input layer 4004. The output layer 4006 transforms the outputs of the convolutional layer 4002 into one or more classifications of the image content.

[0290] Convolutional neural network (CNN) is a deep learning model which outperforms many other models in learning global features of an image and classifying those features into several categories. Unlike conventional computer vision, a CNN-like network model may be gradually tuned to store the global features of a given image to mimic how the human visual perception works.

[0291] Classification is the task of taking an input, such as text or an image and outputting a class (a question, a complaint, a cat, dog, etc.) or a probability of classes that best summarizes or describes the input. CNNs are particularly well suited to classifying features in data sets modelled in two or three dimensions. This makes CNNs popular for image classification, because images may be represented in computer memories in three dimensions (two dimensions for width and height, and a third dimension for pixel features like color components and intensity). For example a color JPEG image of size 480 x 480 pixels may be modelled in computer memory using an array that is 480 x 480 x 3, where each of the values of the third dimension is a red, green, or blue color component intensity for the pixel ranging from 0 to 255. Inputting this array of numbers to a trained CNN will generate outputs that describe the probability of the image being a certain class (.80 for cat, .15 for dog, .05 for bird, etc).Docket No. FSP2375PCTFundamentally, CNNs input the data set, pass it through a series of convolutional transformations, nonlinear activation functions, such as Rectified Linear Units (RELU), and pooling operations (downsampling, e.g., maxpool), and an output layer (e.g., softmax) to generate the classifications.

[0292] Classification deep learning models with a CNN architecture, such as VGG, Inception, Resnet or XceptionNet may be a suitable base architecture for classifiers in one embodiment of the present disclosure. These classification models perform well to detect overall noise features of the image and classify the image into an identified classification category.

[0293] A supervised learning scheme may be used for training in one embodiment, because supervised learning based on its greater efficiency compared to unsupervised learning methods. A CNN architecture may be chosen for the classifiers utilized herein. A cross-validation scheme may be used to train and validate the model. Thus, the dataset may be split into two groups, and the training dataset may be batched and fed to the model for training. Later, the validation dataset may be fed to validate the model’s accuracy. Model weights may be tuned via a b ackpropagation training scheme.

[0294] FIG. 41 depicts an exemplary convolutional neural network layers 4100 in more detail. An example subregion of the input layer region 4104, from a tile 4102 of the input layer 4004 corresponding to a tile of an image, is transformed by a convolutional layer subregion 4108 in the convolutional layer 4106. The tile 4102 in this example is 32x32 neurons (e.g., corresponding to a 32x32 tile 4102), and three neurons deep (e.g., three color channels per pixel of the input region input to the tile 4102). Each neuron in the convolutional layer 4106 is coupled to a local region in the tile 4102 spatially (e.g., in height and width), but to the full depth (i.e., to all color channels if the input is an image). There are multiple neurons (five in this example) along the depth of the convolutional layer subregion 4108 that analyze the subregion of the input layer region 4104 of the tile 4102, in which each neuron of the convolutional layer subregion 4108 may receive inputs from every neuron of the subregion of the input layer region 4104.

[0295] FIG. 42 depicts a VGG network architecture known as VGG16 in one embodiment. The model achieves 92.7% top-5 test accuracy on ImageNet, a dataset of millions images belonging to thousands of classes. VGG16 utilizes multiple 3><3 kernel-sized filters in a series of convolutional layers.Docket No. FSP2375PCT

[0296] The input in this example is a 224 x 224 RGB image. The image is passed through a stack of convolutional (conv) layers, each with filters of a 3x3 receptive field. In one configuration, the model also utilizes 1 xl convolution filters to provide a linear transformation of the input channels (followed by a non-linearity layer). The convolution stride is fixed to 1 pixel; the spatial padding is set such that the spatial resolution is preserved after convolution, i.e. the padding is 1 -pixel for the 3*3 conv layers. Spatial pooling is carried out by five maxpooling layers, which follow some of the conv layers (not all the conv layers are followed by max-pooling). Max-pooling is performed over a 2x2 pixel window, with stride 2.

[0297] Three fully connected (FC) layers follow a stack of convolutional layers (which has a different depth in different configurations of the model). The first two FC layers comprise 4096 channels each. The third performs 1000-way ImageNet Large Scale Visual Recognition Challenge (ILSVRC) classification and thus comprises one channel per class. The final layer is a Softmax layer.

[0298] Hidden layers are equipped with rectification (ReLU) non-linearity. Most VGG16 configurations do not utilize Local Response Normalisation (LRN), as such normalization does not improve the performance but incurs increased memory consumption and computation time.

[0299] FIG. 43 depicts a form of a CNN known as a VGG net 4300. The initial convolution layer 4302a stores the raw image pixels and the final pooling layer 4306b determines the class scores. The intermediate convolution layers (convolution layer 4302b, convolution layer 4302c, and convolution layer 4302d) and rectifier activations (RELU layer 4304a, RELUlayer 4304b, RELUlayer 4304c, and RELUlayer 4304d) and intermediate pooling layers (pooling layer 4306a, pooling layer 4306b) along the processing path are also depicted.

[0300] The VGG net 4300 replaces the (often large) single-layer filters of basic CNNs with multiple smaller-sized (e.g., 3x3) filters in series. With a given receptive field (the effective area size of the input image), multiple stacked smaller-size filters may perform better at image feature classification than a single layer with a larger filter size, because multiple non-linear layers increase the depth of the network which allows it to learn more complex features. In a VGG net 4300 each pooling layer may be small, e.g., 2x2.

[0301] FIGs. 44A-44B illustrate convolution layer filtering 4400 that connects the outputs from groups of neurons in a convolution layer 4402 to neurons in a next layer 4406 using a filter 4404. As shown in FIG. 44A, a receptive field is defined for the convolution layer 4402, in this example sets of 5x5 neurons. The collective outputs of each neuron in the receptive fieldDocket No. FSP2375PCTare weighted and mapped to a single neuron in the next layer 4406. This weighted mapping is referred to as the filter 4404 for the convolution layer 4402 (or sometimes referred to as the kernel of the convolution layer 4402). The filter 4404 depth is not illustrated in this example (i.e., the filter 4404 is actually a cubic volume of neurons in the convolution layer 4402, not a square as illustrated). Thus what is shown is a "slice" of the full filter 4404.

[0302] The filter 4404 is slid, or convolved, around the input image, each time mapping to a different neuron in the next layer 4406. For example, FIG. 44B shows how the filter 4404 is stepped 4408 to the right by 1 unit (i.e., the stride 4410), creating a slightly offset receptive field from the top one, and mapping its output to the next neuron in the next layer 4406. The stride may be and often is other numbers besides one, with larger strides reducing the overlaps in the receptive fields, and hence further reducing the size of the next layer 4406. Every unique receptive field in the convolution layer 4402 that may be defined in this stepwise manner maps to a different neuron in the next layer 4406. Thus, if the convolution layer 4402 is 32x32x3 neurons per slice, the next layer 4406 may be as small as 28x28x1 neurons to cover all the receptive fields of the convolution layer 4402. This is referred to as an activationmap or feature map. There is thus a reduction in layer complexity from the filtering. There are 784 different ways that a 5 x 5 filter may uniquely fit on a 32 x 32 convolution layer 4402, so the next layer 4406 may be 28 x 28. The depth of the convolution layer 4402 is also reduced from 3 to 1 in the next layer 4406.

[0303] The number of total layers to use in a CNN, the number of convolution layers, the filter sizes, and the values for strides at each layer are examples of "hyperparameters" of the CNN.

[0304] FIG. 45 illustrates a pooling layer function 4500 with a 2x2 receptive field and a stride of two. The pooling layer function 4500 is an example of the maxpool pooling technique. The outputs of all the neurons in a particular receptive field of the input layer 4502 are replaced by the maximum valued one of those outputs in the pooling layer 4504. Other options for pooling layers are average pooling and L2-norm pooling. The reason to use a pooling layer is that once a specific feature is recognized in the original input volume (there will be a high activation value), its exact location is not as important as its relative location to the other features.Pooling layers may drastically reduce the spatial dimension of the input layer 4502 from that point forward in the neural network (the length and the width change but not the depth). This serves two main purposes. The first is that the amount of parameters or weights is greatlyDocket No. FSP2375PCTreduced thus lessening the computation cost. The second is that it will control overfitting. Overfitting refers to when a model is so tuned to the training examples that it is not able to generalize well when applied to live data sets.

[0305] FIG. 46 illustrates a basic deep neural network 4600 in accordance with one embodiment. A basic deep neural network 4600 is based on a collection of connected units or nodes called artificial neurons which loosely model the neurons in a biological brain. Each connection, like the synapses in a biological brain, may transmit a signal from one artificial neuron to another. An artificial neuron that receives a signal may process it and then signal additional artificial neurons connected to it.

[0306] In common implementations, the signal at a connection between artificial neurons is a real number, and the output of each artificial neuron is computed by some non-linear function (the activation function) of the sum of its inputs. The connections between artificial neurons are called 'edges' or axons. Artificial neurons and edges typically have a weight that adjusts as learning proceeds. The weight increases or decreases the strength of the signal at a connection. Artificial neurons may have a threshold (trigger threshold) such that the signal is sent if the aggregate signal crosses that threshold. Typically, artificial neurons are aggregated into layers. Different layers may perform different kinds of transformations on their inputs. Signals travel from the first layer (the input layer 4602), to the last layer (the output layer 4606), possibly after traversing one or more intermediate layers, called hidden layers 4604.

[0307] Referring to FIG. 47, an artificial neuron 4700 receiving inputs from predecessor neurons consists of the following components:• inputs Xi;• weights wi applied to the inputs;• an optional threshold (b), which stays fixed unless changed by a learning function; and • an activation function 4702 that computes the output from the previous neuron inputs and threshold, if any.

[0308] An input neuron has no predecessor but serves as input interface for the whole network of artificial neurons 4700, such as may form a basic deep neural network 4600. Similarly an output neuron has no successor and thus serves as output interface of the whole network.

[0309] The network includes connections, each connection transferring the output of a neuron in one layer to the input of a neuron in a next layer. Each connection carries an input x and is assigned a weight w.Docket No. FSP2375PCT

[0310] The activation function 4702 often has the form of a sum of products of the weighted values of the inputs of the predecessor neurons.

[0311] The learning rule is a rule or an algorithm which modifies the parameters of the neural network, in order for a given input to the network to produce a favored output. This learning process typically involves modifying the weights and thresholds of the neurons and connections within the network.

[0312] Various functional operations described herein may be implemented in logic that is referred to using a noun or noun phrase reflecting said operation or function. For example, an association operation may be carried out by an "associator" or "correlator". Likewise, switching may be carried out by a "switch", selection by a "selector", and so on. "Logic" refers to machine memory circuits and non-transitory machine readable media comprising machineexecutable instructions (software and firmware), and / or circuitry (hardware) which by way of its material and / or material-energy configuration comprises control and / or procedural signals, and / or settings and values (such as resistance, impedance, capacitance, inductance, current / voltage ratings, etc.), that may be applied to influence the operation of a device.Magnetic media, electronic circuits, electrical and optical memory (both volatile and nonvolatile), and firmware are examples of logic. Logic specifically excludes pure signals or software per se (however does not exclude machine memories comprising software and thereby forming configurations of matter).

[0313] Within this disclosure, different entities (which may variously be referred to as "units," "circuits," other components, etc.) may be described or claimed as "configured" to perform one or more tasks or operations. This formulation — [entity] configured to [perform one or more tasks] — is used herein to refer to structure (i.e., something physical, such as an electronic circuit). More specifically, this formulation is used to indicate that this structure is arranged to perform the one or more tasks during operation. A structure may be said to be "configured to" perform some task even if the structure is not currently being operated. A "credit distribution circuit configured to distribute credits to a plurality of processor cores" is intended to cover, for example, an integrated circuit that has circuitry that performs this function during operation, even if the integrated circuit in question is not currently being used (e g., a power supply is not connected to it). Thus, an entity described or recited as "configured to" perform some task refers to something physical, such as a device, circuit, memory storing program instructionsDocket No. FSP2375PCTexecutable to implement the task, etc. This phrase is not used herein to refer to something intangible.

[0314] The term "configured to" is not intended to mean "configurable to." An unprogrammed field programmable gate array (FPGA), for example, would not be considered to be "configured to" perform some specific function, although it may be "configurable to" perform that function after programming.

[0315] Reciting in the appended claims that a structure is "configured to" perform one or more tasks is expressly intended not to invoke 35 U.S.C. § 112(f) for that claim element.Accordingly, claims in this application that do not otherwise include the "means for" [performing a function] construct should not be interpreted under 35 U.S.C § 112(f).

[0316] As used herein, the term "based on" is used to describe one or more factors that affect a determination. This term does not foreclose the possibility that additional factors may affect the determination. That is, a determination may be solely based on specified factors or based on the specified factors as well as other, unspecified factors. Consider the phrase "determine A based on B." This phrase specifies that B is a factor that is used to determine A or that affects the determination of A. This phrase does not foreclose that the determination of A may also be based on some other factor, such as C. This phrase is also intended to cover an embodiment in which A is determined based solely on B. As used herein, the phrase "based on" is synonymous with the phrase "based at least i n part on."

[0317] As used herein, the phrase "in response to" describes one or more factors that trigger an effect. This phrase does not foreclose the possibility that additional factors may affect or otherwise trigger the effect. That is, an effect may be solely in response to those factors, or may be in response to the specified factors as well as other, unspecified factors. Con sider the phrase "perform A in response to B." This phrase specifies that B is a factor that triggers the performance of A. This phrase does not foreclose that performing A may also be in response to some other factor, such as C. This phrase is also intended to cover an embodiment in which A is performed solely in response to B.

[0318] As used herein, the terms "first," "second," etc. are used as labels for nouns that they precede, and do not imply any type of ordering (e.g., spatial, temporal, logical, etc.), unless stated otherwise. For example, in a register file having eight registers, the terms "first register" and "second register" may be used to refer to any two of the eight registers, and not, for example, just logical registers 0 and 1.Docket No. FSP2375PCT

[0319] When used in the claims, the term "or" is used as an inclusive or and not as an exclusive or. For example, the phrase "at least one of x, y, or z" means any one of x, y, and z, as well as any combination thereof.

[0320] As used herein, a recitation of “and / or” with respect to two or more elements should be interpreted to mean only one element or a combination of elements. For example, “element A, element B, and / or element C” may include only element A, only element B, only element C, element A and element B, element A and element C, element B and element C, or elements A, B, and C. In addition, “at least one of element A or element B” may include at least one of element A, at least one of element B, or at least one of element A and at least one of element B. Further, “at least one of element A and element B” may include at least one of element A, at least one of element B, or at least one of element A and at least one of element B.

[0321] The subject matter of the present disclosure is described with specificity herein to meet statutory requirements. However, the description itself is not intended to limit the scope of this disclosure. Rather, the inventors have contemplated that the claimed subject matter might also be embodied in other ways, to include different steps or combinations of steps similar to the ones described in this document, in conjunction with other present or futuretechnologies. Moreover, although the terms “step” and / or “block” may be used herein to connote different elements of methods employed, the terms should not be interpreted as implying any particular order among or between various steps herein disclosed unless and except when the order of individual steps is explicitly described.

[0322] Having thus described illustrative embodiments in detail, it will be apparent that modifications and variations are possible without departing from the scope of the disclosure as claimed. The scope of inventive subject matter is not limited to the depicted embodiments but is rather set forth in the following Claims.

Claims

Docket No. FSP2375PCTCLAIMSWhat is claimed is:

1. A computer-implemented method for adapting a robot task control model, comprising:receiving, at a task control model adaptation controller, through a user interface, a user report from a user, the user report including text data describing at least one of a failed robotic task and a desired robot behavior for a robot;retrieving, from a stored dataset of sensor frames and associated vector embeddings in a robot memory, a relevant subset of sensor frames, by performing a similarity search over the user report and the associated vector embeddings;processing, by a robot data processor, the relevant subset of sensor frames using a local robot task control model to generate first classification outputs including first confidence values;processing, by a cloud-based data processor, the relevant subset of sensor frames using a cloud-based teacher robot task control model to generate second classification outputs including second confidence values;on condition the second confidence values are at least equal to a first threshold and the first confidence values are less than a second threshold, as determined by recommendation logic of the task control model adaptation controller:refining, using refinement logic of the task control model adaptation controller, the local robot task control model, including:training a plurality of refined candidate models using the relevant subset of sensor frames and a global dataset;selecting a refined robot task control model from the plurality of refined candidate models based on performance of the plurality of refined candidate models on both the relevant subset of sensor frames and the global dataset; and deploying the refined robot task control model to the robot; andon condition the second confidence values are less than the first threshold, as determined by the recommendation logic of the task control model adaptation controller:personalizing, using personalization logic of the task control model adaptation controller, the local robot task control model, including:Docket No. FSP2375PCTtraining a plurality of personalized candidate models on at least one new class using a set of user-labeled example sensor frames;selecting a personalized robot task control model from the plurality of personalized candidate models based in part on at least one of precision and intersection over union (loU) of the plurality of personalized candidate models; anddeploying the personalized robot task control model to the robot.

2. The computer-implemented method of claim 1, wherein the user interface supports manual operation of the robot by the user from at least one of a location within the robot's working space and a location remote from the robot's working space.

3. The computer-implemented method of claim 2, further comprising:capturing, using at least one camera configured as a sensing system of the robot, failed robotic task actions as failure sensor frames;manually operating the robot, by the user, through the user interface, to complete the failed robotic task successfully as a successful robotic task; andcapturing, using the at least one camera, successful robotic task actions as success sensor frames, wherein the user report further includes the failure sensor frames and the success sensor frames;extracting, using a multi-dimensional vision transformer, failure sensor frame vector embeddings and success sensor frame vector embeddings; andincluding the failure sensor frame vector embeddings and the success sensor frame vector embeddings in the similarity search.

4. The computer-implemented method of claim 1, wherein the user report further includes at least one user-provided sensor frame.

5. The computer-implemented method of claim 4, further comprising:extracting, using a multi-dimensional vision transformer, a user-provided sensor frame vector embedding associated with the user-provided sensor frame; andincluding the user-provided sensor frame vector embedding in the similarity search.

6. The computer-implemented method of claim 1, further comprising:Docket No. FSP2375PCTrefining the local robot task control model including converting the refined robot task control model into a format optimized for hardware of the robot before deployment; and personalizing the local robot task control model including converting the personalized robot task control model into the format optimized for the hardware of the robot before deployment.

7. The computer-implemented method of claim 1, further comprising:on condition additional sensor frames are needed to generate at least one of the first classification outputs, the first confidence values, the second classification outputs, and the second confidence values:performing, by the robot, local path planning, to determine a data harvest navigation path;confirming, by the robot, that the data harvest navigation path adequately covers an environment of the robot;harvesting data using at least one camera configured as a sensing system of the robot; andstoring harvested data in the stored dataset in the robot memory as harvested sensor frames.

8. The computer-implemented method of claim 7, wherein the additional sensor frames are determined to be needed when at least one of:a count of the sensor frames in the relevant subset of sensor frames is below a coverage frame-count threshold;at least one of the first confidence values and the second confidence values is below a coverage confidence threshold;the relevant subset of sensor frames provides insufficient frame diversity for at least one classification category, as measured by an embedding-based diversity metric that falls below a diversity threshold;a data harvest request is received through the user interface; andthe user manually completes the failed robotic task through the user interface following an unsuccessful attempt by the local robot task control model.

9. The computer-implemented method of claim 7, further comprising:Docket No. FSP2375PCTdownsampling a frame rate of the harvested sensor frames to form downsampled sensor frames;removing low-quality sensor frames and near-duplicate sensor frames from the downsampled sensor frames to form a pre-sampled set of sensor frames;storing the pre-sampled set of sensor frames in the stored dataset;detecting sensor frames containing at least one of faces, people, and personal information in the pre-sampled set of sensor frames;redacting the sensor frames containing the at least one of faces, people, and personal information;detecting sensor frames containing sensitive information in the pre-sampled set of sensor frames;expunging the sensor frames containing the sensitive information from the pre-sampled set of sensor frames to form a filtered, privacy-cleaned set of sensor frames; andstoring the filtered, privacy-cleaned set of sensor frames in the stored dataset in the robot memory.

10. The computer-implemented method of claim 9, wherein storing the pre-sampled set of sensor frames replaces the harvested sensor frames previously stored in the stored dataset, and storing the filtered, privacy-cleaned set of sensor frames replaces the pre-sampled set of sensor frames previously stored in the stored dataset.

11. The computer-implemented method of claim 9, wherein redacting the sensor frames includes at least one of removing, obscuring, and anonymizing at least one of image data, depth data, and lidar data captured in the sensor frames.

12. The computer-implemented method of claim 9, further comprising:compressing and encrypting the filtered, privacy-cleaned set of sensor frames; uploading the compressed and encrypted filtered, privacy-cleaned set of sensor frames to the cloud-based data processor;extracting, by the cloud-based data processor, associated vector embeddings for the filtered, privacy-cleaned set of sensor frames;indexing the extracted associated vector embeddings;Docket No. FSP2375PCTperforming high-fidelity quality checks on the filtered, privacy-cleaned set of sensor frames and discarding low-quality filtered sensor frames to form a high-quality, filtered set of sensor frames;performing uncertainty-based sampling on the high-quality, filtered set of sensor frames and discarding sensor frames that fail to exceed an ensemble disagreement threshold to form a differentiated, high-quality, filtered set of sensor frames;detecting and removing additional near-duplicate sensor frames from the differentiated, high-quality, filtered set of sensor frames to form a curated set of sensor frames; and storing the curated set of sensor frames in cloud-based storage.

13. The computer-implemented method of claim 12, wherein the global dataset used in training the plurality of refined candidate models comprises at least a portion of the curated set of sensor frames stored in the cloud-based storage.

14. The computer-implemented method of claim 1, wherein the task control model adaptation controller resides in at least one of the robot and a cloud-based server in network communication with the robot.

15. A robotic system comprising:a user interface;a robot equipped with:a sensing system including at least one camera;a robot memory storing:sensor frames captured by the sensing system;associated vector embeddings extracted from the sensor frames; and a local robot task control model;a robot data processor configured to execute the local robot task control model; a cloud-based data processor configured to receive data from the robot data processor; a cloud-based teacher robot task control model;a robotic control system in at least one of the robot and a cloud server; anda task control model adaptation controller configured to receive input from the user interface, the robot memory, the local robot task control model, the robot data processor, the cloud-based teacher robot task control model, and the cloud-based data processor, to supportDocket No. FSP2375PCTimproved operation of the robot in an intended environment, with intended tasks, the task control model adaptation controller including logic configuring the robotic system, to:receive, at the task control model adaptation controller, through the user interface, a user report from a user, the user report including text data describing at least one of a failed robotic task and a desired robot behavior for the robot;retrieve, from a stored dataset of sensor frames and associated vector embeddings in the robot memory, a relevant subset of sensor frames, by performing a similarity search over the user report and the associated vector embeddings;process, with the robot data processor, the relevant subset of sensor frames using the local robot task control model to generate first classification outputs including first confidence values;process, with the cloud-based data processor, the relevant subset of sensor frames using the cloud-based teacher robot task control model to generate second classification outputs including second confidence values;on condition the second confidence values are at least equal to a first threshold and the first confidence values are less than a second threshold, as determined by recommendation logic of the task control model adaptation controller:refine, using refinement logic of the task control model adaptation controller, the local robot task control model, including:train a plurality of refined candidate models using the relevant subset of sensor frames and a global dataset;select a refined robot task control model from the plurality of refined candidate models based on performance of the plurality of refined candidate models on both the relevant subset of sensor frames and the global dataset; anddeploy the refined robot task control model to the robot; andon condition the second confidence values are less than the first threshold, as determined by the recommendation logic of the task control model adaptation controller:personalize, using personalization logic of the task control model adaptation controller, the local robot task control model, including:train a plurality of personalized candidate models on at least one new class using a set of user-labeled example sensor frames;Docket No. FSP2375PCTselect a personalized robot task control model from the plurality of personalized candidate models based in part on at least one of precision and intersection over union (loU) of the plurality of personalized candidate models; anddeploy the personalized robot task control model to the robot.

16. The robotic system of claim 15, wherein the user interface supports manual operation of the robot by the user from at least one of a location within the robot's working space and a location remote from the robot's working space.

17. The robotic system of claim 16, wherein the logic further configures the robotic system to:capture, using the at least one camera configured in the sensing system of the robot, failed robotic task actions as failure sensor frames;manually operate the robot, by the user, through the user interface, to complete the failed robotic task successfully as a successful robotic task; andcapture, using the at least one camera, successful robotic task actions as success sensor frames, wherein the user report further includes the failure sensor frames and the success sensor frames;extract, using a multi-dimensional vision transformer, failure sensor frame vector embeddings and success sensor frame vector embeddings; andinclude the failure sensor frame vector embeddings and the success sensor frame vector embeddings in the similarity search.

18. The robotic system of claim 15, wherein the user report further includes at least one user-provided sensor frame.

19. The robotic system of claim 18, wherein the logic further configures the robotic system to:extract, using a multi-dimensional vision transformer, a user-provided sensor frame vector embedding associated with the user-provided sensor frame; andinclude the user-provided sensor frame vector embedding in the similarity search.

20. The robotic system of claim 15, wherein the logic further configures the robotic system to:refine the local robot task control model including converting the refined robot task control model into a format optimized for hardware of the robot before deployment; andDocket No. FSP2375PCTpersonalize the local robot task control model including converting the personalized robot task control model into the format optimized for the hardware of the robot before deployment.

21. The robotic system of claim 15, wherein the logic further configures the robotic system to:on condition additional sensor frames are needed to generate at least one of the first classification outputs, the first confidence values, the second classification outputs, and the second confidence values:perform, by the robot, local path planning, to determine a data harvest navigation path; confirm, by the robot, that the data harvest navigation path adequately covers an environment of the robot;harvest data using the at least one camera configured in the sensing system of the robot; andstore harvested data in the stored dataset in the robot memory as harvested sensor frames.

22. The robotic system of claim 21, wherein the additional sensor frames are determined to be needed when at least one of:a count of the sensor frames in the relevant subset of sensor frames is below a coverage frame-count threshold;at least one of the first confidence values and the second confidence values is below a coverage confidence threshold;the relevant subset of sensor frames provides insufficient frame diversity for at least one classification category, as measured by an embedding-based diversity metric that falls below a diversity threshold;a data harvest request is received through the user interface; andthe user manually completes the failed robotic task through the user interface following an unsuccessful attempt by the local robot task control model.

23. The robotic system of claim 21, wherein the logic further configures the robotic system to:downsampling a frame rate of the harvested sensor frames to form downsampled sensor frames;remove low-quality sensor frames and near-duplicate sensor frames from the downsampled sensor frames to form a pre-sampled set of sensor frames;Docket No. FSP2375PCTstore the pre-sampled set of sensor frames in the stored dataset;detect sensor frames containing at least one of faces, people, and personal information in the pre-sampled set of sensor frames;redact the sensor frames containing the at least one of faces, people, and personal information;detect sensor frames containing sensitive information in the pre-sampled set of sensor frames;expunge the sensor frames containing the sensitive information from the pre-sampled set of sensor frames to form a filtered, privacy-cleaned set of sensor frames; andstore the filtered, privacy-cleaned set of sensor frames in the stored dataset in the robot memory.

24. The robotic system of claim 23, wherein storing the pre-sampled set of sensor frames replaces the harvested sensor frames previously stored in the stored dataset, and storing the filtered, privacy-cleaned set of sensor frames replaces the pre-sampled set of sensor frames previously stored in the stored dataset.

25. The robotic system of claim 23, wherein redacting the sensor frames includes at least one of removing, obscure, and anonymizing at least one of image data, depth data, and lidar data captured in the sensor frames.

26. The robotic system of claim 23, wherein the logic further configure the robotic system to:compress and encrypt the filtered, privacy-cleaned set of sensor frames;upload the compressed and encrypted filtered, privacy-cleaned set of sensor frames to the cloud-based data processor;extract, by the cloud-based data processor, associated vector embeddings for the filtered, privacy-cleaned set of sensor frames;index the extracted associated vector embeddings;perform high-fidelity quality checks on the filtered, privacy-cleaned set of sensor frames and discarding low-quality filtered sensor frames to form a high-quality, filtered set of sensor frames;perform uncertainty-based sampling on the high-quality, filtered set of sensor frames and discarding sensor frames that fail to exceed an ensemble disagreement threshold to form a differentiated, high-quality, filtered set of sensor frames;Docket No. FSP2375PCTdetect and remove additional near-duplicate sensor frames from the differentiated, high-quality, filtered set of sensor frames to form a curated set of sensor frames; andstore the curated set of sensor frames in cloud-based storage.

27. The robotic system of claim 26, wherein the global dataset used in training the plurality of refined candidate models comprises at least a portion of the curated set of sensor frames stored in the cloud-based storage.

28. The robotic system of claim 15, wherein the task control model adaptation controller resides in at least one of the robot and a cloud-based server in network communication with the robot.