Smart data buffer systems and methods

Abstract

Techniques are provided for storing data in a data buffer pertaining to anomalies detected by various components, including an artificial intelligence system. The embodiments are directed to receiving image frames captured by an imaging system. As one or more components in an artificial intelligence (AI) system process the image frames, the one or more components in the AI system also detect anomalies in processing of the image frames. A filtering module determines whether to store data or information, including a subset of image frames, corresponding to the anomalies in a data buffer. The data and / or information corresponding to the anomalies is stored in the data buffer based on the determination.

Description

[0001] P230033-724-W001 70052.2050W001

[0002] SMART DATA BUFFER SYSTEMS AND METHODS

[0003] Stefan Schulte

[0004] CROSS REFERENCE TO RELATED APPLICATIONS

[0005] This application claims priority to and the benefit of U.S. Provisional Patent Application No. 63 / 753,374 filed February 3, 2025 and entitled “SMART DATA BUFFER SYSTEMS AND METHODS,” which is incorporated herein by reference in its entirety.

[0006] TECHNICAL FIELD

[0007] The present invention relates generally to identifying training data for artificial intelligence systems, and more specifically to a data buffer that captures data that indicate anomalies in processing data in the artificial intelligence system.

[0008] BACKGROUND

[0009] Artificial intelligence (Al) systems are increasingly being integrated into cloud-based and edge systems. Al systems may also be used to monitor other systems to detect various issues, such as possible gas leaks and the size of the gas leaks or unauthorized access to secure or restricted areas. Al systems may also be used to detect animals, track, monitor, and / or control vehicles and unmanned aircraft systems (UASs), and the like.

[0010] Al systems rely on vast amounts of annotated (labeled) data to create generalized models capable of functioning in various environments and situations. On occasion, Al systems may achieve lower accuracy than expected due to certain objects not being well represented by the data, Al systems confusing different objects, and Al systems issuing false alarms due to objects that look similar to objects that the Al systems learned to associate with the alarms.

[0011] Dedicated data may improve Al systems. This data may be collected using cameras and sensors that exist in the field. Dedicated data may include data that caused the Al systems to achieve low accuracy while processing the data, or the errors in processing the output of the Al systems in the downstream tasks. Dedicated data, however, may be difficult to collect or replicate, and may involve significant time and resources.

[0012] Accordingly, the embodiments are directed to obtaining dedicated data.

[0013] SUMMARYP230033-724-W001 70052.2050W001

[0014] Methods and systems are provided for storing data in a data buffer pertaining to anomalies detected by various components, including an artificial intelligence system.

[0015] In some embodiments, a method comprises receiving data captured by an imaging system, determining, using one or more components in an artificial intelligence (Al) system, an anomaly in processing of the data by the one or more components in the AT system, determining to store the data corresponding to the anomaly in a data buffer, and storing the data corresponding to the anomaly in the data buffer.

[0016] In some embodiments, the data includes an image frame or a video.

[0017] In some embodiments, the Al system is included in the imaging system or is communicatively coupled to the imaging system.

[0018] In some embodiments, the method further comprises generating a training dataset from the data in the data buffer and retraining the Al system using the training dataset.

[0019] In some embodiments, determining to store the data corresponding to the anomaly in the data buffer further comprises configuring a filter module with one or more rules, and determining using the filtering module to store the data in the data buffer based on the one or more rules.

[0020] In some embodiments, the method further comprises configuring the data buffer to store different types of data.

[0021] In some embodiments, a network system comprising logic device configured to receive data captured by a plurality of processing systems, determine, using one or more components in an artificial intelligence (Al) system, a plurality of anomalies in processing of the data, determine to store a subset of the data corresponding to the plurality of anomalies in a data buffer, and store the subset of the data corresponding to the plurality of anomalies in the data buffer.

[0022] In some embodiments, a non-transitory computer readable medium having instructions stored thereon, that when executed by a processor cause the processor to perform operations, the operations comprising receiving an image frame captured by an imaging system, determining, using one or more components, an anomaly in processing of the image frame by the one or more components, determining to store the image frame and data corresponding to the anomaly in a data buffer, and storing the image frame and the data corresponding to the anomaly in the data buffer.P230033-724-W001 70052.2050W001

[0023] The scope of the invention is defined by the claims, which are incorporated into this section by reference. A more complete understanding of embodiments of the present invention will be afforded to those skilled in the art, as well as a realization of additional advantages thereof, by a consideration of the following detailed description of one or more embodiments. Reference will be made to the appended sheets of drawings that will first be described briefly.

[0024] BRIEF DESCRIPTION OF THE DRAWINGS

[0025] Fig. 1 illustrates a block diagram of a standalone imaging system, in accordance with an embodiment of the disclosure.

[0026] Fig. 2 illustrates a block diagram of an imaging system coupled to a network system, in accordance with an embodiment of the disclosure.

[0027] Figs. 3-4 illustrate block diagrams of artificial intelligence systems storing data into a data buffer, in accordance with an embodiment of the disclosure.

[0028] Fig 5 illustrates a block diagram of training Al systems with a training dataset from the data buffer, in accordance with an embodiment of the disclosure.

[0029] Fig. 6 illustrates a process for selectively storing data in the data buffer, in accordance wi th an embodiment of the disclosure.

[0030] Fig. 7 illustrates a neural network model in accordance with an embodiment of the disclosure.

[0031] Embodiments of the present invention and their advantages are best understood by referring to the detailed description that follows. It should be appreciated that like reference numerals are used to identify like elements illustrated in one or more of the figures.

[0032] DETAILED DESCRIPTION

[0033] The embodiments are directed to a processing system that analyzes its behavior and detects when anomalies occur. Generally, anomalies occur when the processing system, such as an imaging system, fails to perform as intended. Some examples of anomalies may include the processing system failing to identify objects in images, failing to track objects that it has designated for tracking, confusing different objects in the images, generating false alerts, and the like.P230033-724-W001 70052.2050W001

[0034] To capture data and / or information (e.g., images) that cause the processing system to generate anomalies, the processing system may monitor the output of its various components. Upon detecting an anomaly, the processing system may determine whether to store the underlying data and / or information that caused the anomaly in a data buffer. The criteria for storing the data and / or information in the data buffer may be configurable. The criteria may include rules that ensure duplicative data and / or information is not stored in the buffer.

[0035] Example rules may also ensure that the data and / or information to be stored in the buffer is more relevant or of higher quality than the data and / or information already in the data buffer with respect to a particular anomaly.

[0036] In some instances, the data from the data buffer may be combined into a training dataset that may be used to retrain or repair the components in the processing system. In other instances, the data in the data buffer may be used to recreate the anomalous behavior in the processing system to modify the processing system and remove the behavior.

[0037] Further description of the embodiments is discussed below.

[0038] Turning now to the drawings, Fig. 1 illustrates a block diagram of an imaging system 100, in accordance with an embodiment of the disclosure. Imaging system 100 may be used to capture and process image frames in accordance with various techniques described herein. Although the embodiments are directed to imaging system 100, the embodiments may also apply to other types of data processing systems. In one embodiment, various components of imaging system 100 may be provided in a housing 101, such as a housing of a camera, a personal electronic device (e.g., a mobile phone), or other system. In another embodiment, one or more components of imaging system 100 may be implemented remotely from each other in a distributed fashion (e.g., networked or otherwise).

[0039] In one embodiment, imaging system 100 includes a logic device 110, a memory component 120, an image capture component 130, optical components 132 (e.g., one or more lenses configured to receive electromagnetic radiation through an aperture 134 in housing 101 and pass the electromagnetic radiation to image capture component 130), a display component 140, a control component 150, a communication component 152, a mode sensing component 160, and a sensing component 162.

[0040] In various embodiments, imaging system 100 may implemented as an imaging device, such as a camera, to capture image frames, for example, of a scene 170 (e.g., a field of view) in an external environment (e.g.. external to imaging system 100). Imaging system 100 mayP230033-724-W001 70052.2050W001

[0041] represent any type of camera system which, for example, detects electromagnetic radiation (e.g., irradiance) and provides representative data (e.g., one or more still image frames or video image frames). For example, imaging system 100 may represent a camera that is directed to detect one or more ranges (e.g., wavebands) of electromagnetic radiation and provide associated image data. In some embodiments, imaging system 100 may include a portable device. In some embodiments, imaging system 100 may be implemented as a handheld device, including a thermal imager handheld device. In some embodiments, imaging system 100 may be a non-portable and / or non-handheld device. In some embodiments, imaging system 100 may be attached to a gimbal and / or other mechanism, device, or structure. In some embodiments, imaging system 100 may be coupled to various tvpes of vehicles (e.g., a land-based vehicle, a watercraft, an aircraft, a spacecraft, or other vehicle) or to various types of fixed locations (e.g., a home security mount, a campsite or outdoors mount, or other location) via one or more types of mounts. In still another embodiment, imaging system 100 may be integrated as part of a non-mobile installation to provide image frames to be stored and / or displayed.

[0042] Logic device 110 may include, for example, a microprocessor, a single-core processor, a multi-core processor, a microcontroller, a programmable logic device (e.g., a field programmable logic device (FPGA)), and / or other device configured to perform processing operations, a digital signal processing (DSP) device, one or more memories for storing executable instructions (e.g., software, firmware, or other instructions), and / or or any other appropriate combination of processing device and / or memory to execute instructions to perform any of the various operations described herein. Logic device 110 is adapted to interface and communicate with components 120, 130, 140, 150, 160, 162, and 164 to perform method and processing steps as described herein. Logic device 110 may include one or more mode modules 112A-112N for operating in one or more modes of operation (e.g., to operate in accordance with any of the various embodiments disclosed herein). In one embodiment, mode modules 112A-112N are adapted to define processing and / or display operations that may be embedded in logic device 110 or stored on memory component 120 for access and execution by logic device 110. In another aspect, logic device 110 may be adapted to perform various types of image processing techniques as described herein.

[0043] In various embodiments, it should be appreciated that each mode module 112A-112N may be integrated in software and / or hardware as part of logic device 110, or code (e.g., software or configuration data) for each mode of operation associated with each modeP230033-724-W001 70052.2050W001

[0044] module 112A-112N, which may be stored in memory component 120. Embodiments of mode modules 112A-112N (i.e.. modes of operation) disclosed herein may be stored by a machine readable medium 113 in a non-transitory manner (e.g., a memory, a hard drive, a compact disk, a digital video disk, or a flash memory) to be executed by a computer (e.g., logic or processor-based system) to perform various methods disclosed herein.

[0045] In various embodiments, the machine readable medium 113 may be included as part of imaging system 100 and / or separate from imaging system 100, with stored mode modules 112A-112N provided to imaging system 100 by coupling the machine readable medium 113 to imaging system 100 and / or by imaging system 100 downloading (e.g., via a wired or wireless link) the mode modules 112A-112N from the machine readable medium (e.g., containing the non-transitory information). In various embodiments, as described herein, mode modules 112A-112N provide for improved camera processing techniques for real time applications, wherein a user or operator may change the mode of operation depending on a particular application, such as an off-road application, a maritime application, an aircraft application, a space application, or other application.

[0046] Memory component 120 includes, in one embodiment, one or more memory devices (e.g., one or more memories) to store data and information. The one or more memory7devices may include various types of memory including volatile and non-volatile memory devices, such as RAM (Random Access Memory), ROM (Read-Only Memory), EEPROM (Electrically-Erasable Read-Only Memory ), flash memory, or other types of memory. In one embodiment, logic device 110 is adapted to execute software stored in memory7component 120 and / or machine readable medium 113 to perform various methods, processes, and modes of operations in manner as described herein.

[0047] In some embodiments, image capture component 130 includes one or more sensors (e.g., any type of thermal infrared, near infrared, short wave infrared, mid wave infrared, long wave infrared, visible light, and / or other type of detector, including a detector implemented as part of a focal plane array) responsive to radiation received from scene 170. For example, the sensors of image capture component 130 may store voltages in response to radiation received from scene 170 (e.g., by integrating currents responsive to the radiation) and convert the voltages (e.g., via an analog-to-digital converter and / or other circuitry7included as part of the sensor or separate from the sensor as part of imaging system 100) to pixel counts associated with pixels of the image frames.P230033-724-W001 70052.2050W001

[0048] Logic device 110 may be adapted to receive image frames from image capture component 130, process image frames, store image frames in memory component 120. and / or retrieve stored image frames from memory component 120. Logic device 110 may be adapted to process image frames stored in memory component 120 to provide image frames to display component 140 for viewing by a user.

[0049] Display component 140 includes, in one embodiment, an image display device (e.g., a liquid cry stal display (LCD)) or various other types of generally known video displays or monitors. Logic device 110 may be adapted to display image data and information on display component 140. Logic device 110 may be adapted to retrieve image data and information from memory component 120 and display any retrieved image data and information on display component 140. Display component 140 may include display electronics, which may be utilized by logic device 110 to display image data and information. Display component 140 may receive image data and information directly from image capture component 130 via logic device 110, or the image data and information may be transferred from memory component 120 via logic device 110.

[0050] In one embodiment, logic device 110 may initially process a captured thermal image frame and present a processed image frame in one mode, corresponding to mode modules 112A-112N, and then upon user input to control component 150, logic device 110 may switch the current mode to a different mode for viewing the processed image frame on display component 140 in the different mode. This switching may be referred to as applying the camera processing techniques of mode modules 112A-112N for real time applications, wherein a user or operator may change the mode while viewing an image frame on display component 140 based on user input to control component 150. In various aspects, display component 140 may be remotely positioned, and logic device 110 may be adapted to remotely display image data and information on display component 140 via wired or wireless communication with display component 140, as described herein.

[0051] Control component 150 includes, in one embodiment, a user input and / or interface device having one or more user actuated components, such as one or more push buttons, slide bars, rotatable knobs or a keyboard, that are adapted to generate one or more user actuated input control signals. Control component 150 may be adapted to be integrated as part of display component 140 to operate as both a user input device and a display device, such as, for example, a touch screen device adapted to receive input signals from a user touching different parts of the display screen. Logic device 110 may be adapted to sense control inputP230033-724-W001 70052.2050W001

[0052] signals from control component 150 and respond to any sensed control input signals received therefrom.

[0053] Control component 150 may include, in one embodiment, a control panel unit (e.g., a wired or wireless handheld control unit) having one or more user-activated mechanisms (e.g., buttons, knobs, sliders, or others) adapted to interface with a user and receive user input control signals. In various embodiments, the one or more user-activated mechanisms of the control panel unit may be utilized to select between the various modes of operation, as described herein in reference to mode modules 112A-112N. In other embodiments, it should be appreciated that the control panel unit may be adapted to include one or more other user-activated mechanisms to provide various other control operations of imaging system 100, such as auto-focus, menu enable and selection, field of view' (FoV), brightness, contrast, gain, offset, spatial, temporal, and / or various other features and / or parameters. In still other embodiments, a variable gain signal may be adjusted by the user or operator based on a selected mode of operation.

[0054] In another embodiment, control component 150 may include a graphical user interface (GUI), which may be integrated as part of display component 140 (e.g., a user actuated touch screen), having one or more images of the user-activated mechanisms (e.g., buttons, knobs, sliders, or others), which are adapted to interface with a user and receive user input control signals via the display component 140. As an example for one or more embodiments as discussed further herein, display component 140 and control component 150 may represent appropriate portions of a smart phone, a tablet, a personal digital assistant (e.g., a wireless, mobile device), a laptop computer, a desktop computer, or other type of device.

[0055] Mode sensing component 160 includes, in one embodiment, an application sensor adapted to automatically sense a mode of operation, depending on the sensed application (e.g., intended use or implementation), and provide related information to logic device 110. In various embodiments, the application sensor may include a mechanical triggering mechanism (e.g., a clamp, clip, hook, switch, push-button, or others), an electronic triggering mechanism (e.g., an electronic switch, push-button, electrical signal, electrical connection, or others), an electro-mechanical triggering mechanism, an electro-magnetic triggering mechanism, or some combination thereof. For example, for one or more embodiments, mode sensing component 160 senses a mode of operation corresponding to the intended application of imaging system 100 based on the type of mount (e.g., accessory or fixture) to which a user has coupled the imaging system 100 (e.g., image capture component 130). Alternatively, theP230033-724-W001 70052.2050W001

[0056] mode of operation may be provided via control component 150 by a user of imaging system 100 (e.g., wirelessly via display component 140 having a touch screen or other user input representing control component 150).

[0057] Furthermore, in accordance with one or more embodiments, a default mode of operation may be provided, such as for example when mode sensing component 160 does not sense a particular mode of operation (e.g., no mount sensed or user selection provided). For example, imaging system 100 may be used in a freeform mode (e.g., handheld with no mount) and the default mode of operation may be set to handheld operation, with the image frames provided wirelessly to a wireless display (e.g., another handheld device with a display, such as a smart phone, or to a vehicle’s display).

[0058] Mode sensing component 160, in one embodiment, may include a mechanical locking mechanism adapted to secure the imaging system 100 to a vehicle or part thereof and may include a sensor adapted to provide a sensing signal to logic device 110 when the imaging system 100 is mounted and / or secured to the vehicle. Mode sensing component 160. in one embodiment, may be adapted to receive an electrical signal and / or sense an electrical connection type and / or mechanical mount type and provide a sensing signal to logic device 110. Alternatively or in addition, as discussed herein for one or more embodiments, a user may provide a user input via control component 150 (e.g., a wireless touch screen of display component 140) to designate the desired mode (e.g., application) of imaging system 100.

[0059] Logic device 110 may be adapted to communicate with mode sensing component 160 (e.g., by receiving sensor information from mode sensing component 160) and image capture component 130 (e.g., by receiving data and information from image capture component 130 and providing and / or receiving command, control, and / or other information to and / or from other components of imaging system 100).

[0060] In various embodiments, mode sensing component 160 may be adapted to provide data and information relating to system applications including a handheld implementation and / or coupling implementation associated with various types of vehicles (e.g., a land-based vehicle, a watercraft, an aircraft, a spacecraft, or other vehicle) or stationary applications (e.g., a fixed location, such as on a structure). In one embodiment, mode sensing component 160 may include communication devices that relay information to logic device 110 via wireless communication. For example, mode sensing component 160 may be adapted to receive and / or provide information through a satellite, through a local broadcast transmissionP230033-724-W001 70052.2050W001

[0061] (e.g., radio frequency), through a mobile or cellular network and / or through information beacons in an infrastructure (e.g., a transportation or highway information beacon infrastructure) or various other wired or wireless techniques (e.g., using various local area or wide area wireless standards).

[0062] In another embodiment, imaging system 100 may include one or more other types of sensing components 162, including environmental and / or operational sensors, depending on the sensed application or implementation, which provide information to logic device 110 (e.g., by receiving sensor information from each sensing component 162). In various embodiments, other sensing components 162 may be adapted to provide data and information related to environmental conditions, such as internal and / or external temperature conditions, lighting conditions (e.g., day, night, dusk, and / or dawn), humidity levels, specific weather conditions (e.g., sun. rain, and / or snow), distance (e.g., laser rangefinder), and / or whether a tunnel, a covered parking garage, or that some type of enclosure has been entered or exited. Accordingly, other sensing components 162 may include one or more conventional sensors as would be known by those skilled in the art for monitoring various conditions (e.g., environmental conditions) that may have an effect (e.g., on the image appearance) on the data provided by image capture component 130.

[0063] In some embodiments, other sensing components 162 may include devices that relay information to logic device 110 via wireless communication. For example, each sensing component 162 may be adapted to receive information from a satellite, through a local broadcast (e.g., radio frequency) transmission, through a mobile or cellular network and / or through information beacons in an infrastructure (e.g., a transportation or highway information beacon infrastructure) or various other wired or wireless techniques. In some embodiments, other sensing components 162 may include one or more motion and / or location sensors (e.g., accelerometers, gyroscopes, micro-electromechanical system (MEMS) devices, and / or others as appropriate).

[0064] In various embodiments, components of imaging system 100 may be combined and / or implemented or not, as desired or depending on application requirements, with imaging system 100 representing various operational blocks of a system. For example, logic device 110 may be combined with memory component 120, image capture component 130, display component 140, and / or mode sensing component 160. In another example, logic device 110 may be combined with image capture component 130 with only certain operations of logic device 110 performed by circuitry (e.g., a processor, a microprocessor, a microcontroller, aP230033-724-W001 70052.2050W001

[0065] logic device, or other circuitry) within image capture component 130. In still another example, control component 150 may be combined with one or more other components or be remotely connected to at least one other component, such as logic device 110, via a wired or wireless control device so as to provide control signals thereto.

[0066] In some embodiments, communication component 152 may be implemented as a network interface component (NIC) adapted for communication with a network including other devices in the network. In various embodiments, communication component 152 may include a wireless communication component, such as a wireless local area network (WLAN) component based on the IEEE 802.11 standards, a wireless broadband component, mobile cellular component, a wireless satellite component, or various other types of wireless communication components including radio frequency (RF), microwave frequency (MWF), and / or infrared frequency (IRF) components adapted for communication with a network. As such, communication component 152 may include an antenna coupled thereto for wireless communication purposes. In other embodiments, the communication component 152 may be adapted to interface with a DSL (e.g., Digital Subscriber Line) modem, a PSTN (Public Switched Telephone Network) modem, an Ethernet device, and / or various other types of wired and / or wireless network communication devices adapted for communication with a network.

[0067] In some embodiments, imaging system 100 may include Al system 164. Al system 1 4 may receive image frames from image capture component 130. Al system 164 alone or with other components may be configured to perform various tasks, such as real-time monitoring of other devices, machines, components included or coupled to the devices and / or machines in an environment surrounding imaging system 100. For example, Al system 164 may perform environmental monitoring and analyze images from sensors to monitor air pollution levels, detect harmful pollutants, etc. Al system 164 may also analyze images from sensors to detect contaminants in water sources and monitor for environmental hazards. In another example, in a health sector, Al system 164 may continuously monitor vital signs of patients in hospitals or at home by analyzing images from medical devices. Wearable health devices can also use Al system 164 to analyze data from images to track physical activity, heart rate, sleep patterns, and other health metrics. In yet another example, Al system 164 may analyze images of equipment to predict when machinery or equipment is likely to fail, allowing for timely maintenance and reducing downtime. In yet another example, Al system 164 may analyze images from drones or satellites to assess crop health, detect diseases, andP230033-724-W001 70052.2050W001

[0068] optimize irrigation. Al system 164 may also analyze images to monitor soil conditions, including moisture levels and nutrient content, to improve crop yields. In yet another example, Al system 164 may analyze images to monitor traffic and transportation issues by analyzing images of buses, trains, and other vehicles, track the performance and punctuality of public transport systems and provide real-time updates to passengers. Al system 164 may also monitor pedestrian traffic in urban areas by analyzing images from cameras to improve safety and optimize traffic flow. In yet another example, Al system 164 may track inventory levels in real-time by analyzing images of stock, optimizing stock levels, and reducing waste. Al system 164 may also analyze customer behavior in stores by examining images from security cameras to improve product placement and marketing strategies.

[0069] Al system 164 may include one or more components, such as neural network models (including convolutional neural network models), algorithms, decision trees, agents, and the like. The components within Al system 164 may be configurable as suited for specific tasks to be performed by Al system 164. Moreover, the components of Al system 164 may be trained on datasets until they leam to interact with a real-w orld environment. More details on the neural netw ork model implementation of Al system 164 are discussed in Fig. 7.

[0070] In some instances, imaging system 100 may also include other components 166. Other components 166 may be software components, applications, etc., that execute in network system 200 and may be designed to perform downstream tasks based on the output of Al system 164. For example, suppose Al system 164 identified an object in one of the image frames. An application included in the other components 166 may be designed to perform a downstream task based on the object, such as track the object using the objects identified in the subsequent image frames.

[0071] As Al system 164 and / or other components 166 perform various tasks, Al system 164 and / or other components 166 may also detect anomalies in their execution. These anomalies may cause Al system 164 and / or other components 166 to function improperly, generate incorrect results, and the like. In some instances, the data, e.g., images in the training dataset that was used to train Al system 164 may have been incomplete or did not account for various hazards, anomalies, etc., that may occur in the real-world environment. For example, the data in the training dataset failed to include special, unusual, or idiosyncratic objects in images, which may cause Al system 164 to confuse, misidentify, or fail to identify objects inP230033-724-W001 70052.2050W001

[0072] the images when placed in a real-world environment. Moreover, these types of problems are ty pically not solved by collecting more training data to train Al system 164. Rather, the training dataset should include the data with the special, unusual, or idiosyncratic objects to further train or finetune the Al system 1 4.

[0073] Similarly, other components 166 may also generate anomalies. Suppose Al system 164 has properly identified an object and invoked a tracking application included in the other components 166. The tracking application, however, fails to engage for this particular object, while it engages for other objects. In this case, although the anomaly occurs in a downstream task (e.g., the tracking application), the anomaly may be due to an object or data corresponding to the object that may be generated by Al system 164.

[0074] Fig. 2 illustrates a block diagram of an imaging system 100 in a network environment, in accordance with an embodiment of the disclosure. Unlike the imaging system 100 in Fig 1, Al system 164 in Fig. 2 may operate externally to imaging system 100. For example, Al system 164 may operate in a network system 200 that is coupled to imaging system 100 by¬ network 210. An example of network 210 may be the Internet or one or more intranets, landline networks, wireless networks, local area networks, wide area networks, and / or other appropriate ty pes of networks. In this embodiment, data, such as image frames, may be captured by imaging system 100 and transmitted to network system 200 for processing using Al system 164. Although a single imaging system 100 is shown, there may be multiple imaging systems 100 connected to network system 200.

[0075] Network system 200 may be or host a cloud server or another server conducive to receiving and processing data from multiple processing systems, such as image processing systems 100. Network system 200 may include a logic device 202, a communication component 204, and memory- component 206. Logic device 202 may be similar to logic device 110, and may include, for example, a microprocessor, a single-core processor, a multicore processor, a microcontroller, a programmable logic device (e.g., a field programmable logic device (FPGA)), and / or other device configured to perform processing operations, a digital signal processing (DSP) device, one or more memories for storing executable instructions (e.g., software, firmware, or other instructions), and / or or any other appropriate combination of processing device and / or memory to execute instructions to perform any of the various operations described herein. Logic device 202 may also be adopted for fast and memory- intensive processing from multiple imaging systems 100.P230033-724-W001 70052.2050W001

[0076] Communication component 204 may be similar to communication component 152 discussed in Fig. 1. Communication component 204 may be implemented as a network interface component (NIC) adapted for communication with a network including other devices in the network. In various embodiments, communication component 204 may include a wireless communication component, such as a wireless local area network (WLAN) component based on the IEEE 802.11 standards, a wireless broadband component, mobile cellular component, a wireless satellite component, or various other types of wireless communication components including radio frequency (RF), microwave frequency (MWF), and / or infrared frequency (IRF) components adapted for communication with a network. As such, communication component 204 may include an antenna coupled thereto for wireless communication purposes. In other embodiments, the communication component 204 may be adapted to interface with a DSL (e.g., Digital Subscriber Line) modem, a PSTN (Public Switched Telephone Network) modem, an Ethernet device, and / or various other types of wired and / or wireless network communication devices adapted for communication with a network. In some instances, communication component 204 may receive image frames from imaging system 100 for processing by Al system 164.

[0077] Memory component 206 includes, in one embodiment, one or more memory devices (e.g., one or more memories) to store data and information. The one or more memory devices may include various ty pes of memory including volatile and non-volatile memory devices, such as RAM (Random Access Memory ), ROM (Read-Only Memory ), EEPROM (Electrically-Erasable Read-Only Memory’), flash memory, or other types of memory.

[0078] Memory component 176 may also be adopted for large scale storage and may enable fast data processing by logic device 172. In some instances, memory’ component 206 may store and perform memory' management for Al system 164.

[0079] Al system 164 may use logic device 202 and memory’ component 204 to process image frames that network system 200 receives over network 210 and perform various tasks. The tasks may be similar to the tasks discussed in Fig. 1. Unlike Fig. 1. however. Al system 164 may perform tasks on behalf of multiple imaging systems 100.

[0080] In some instances, network system 200 may also include other components 166. Other components 166 may be software components, applications, etc., that execute in network system 200 and may be designed to perform downstream tasks based on the output of Al system 164. For example, suppose Al system 164 identified an object in one of the image frames. An application included in the other components 166 may be designed to perform aP230033-724-W001 70052.2050W001

[0081] downstream task based on the object, such as tracking the object using the objects identified in the subsequent image frames.

[0082] As discussed above, as Al system 164 and / or other components 166 execute, Al system 164 and / or other components 166 may generate anomalies.

[0083] Fig. 3 is a block diagram for storing anomalies detected in artificial intelligence (Al) systems 164, according to some embodiments. Al systems 164A and 164C may be executing within imaging systems 100A and 100C, as discussed in Fig. 1. Al system 164B may be communicatively coupled to imaging system 100B via a network, as discussed in Fig. 2. For exemplary purposes, Al systems 164A-C have been trained on a training dataset, which may include images with labels, and placed in a real-world environment to process images collected by imaging systems 100A-C. Although three Al systems 164A-C and imaging systems 1 OOA-C are show n, the embodiments are also applicable to any number of Al systems 164.

[0084] As Al systems 164A-C receive image frames from imaging systems 1 OOA-C, Al systems 164A-C process the respective images. As discussed above, Al systems 164A-C may also self-monitor their respective outputs for anomalies. The anomalies may occur when Al systems 164A-C record an exception, generate an alert, and the like. For example, Al systems 1 4A-C may self-monitor for when they improperly analyzed an image frame by classifying an object in the image frame as unknown, failing to identify dimensions of the object, and the like.

[0085] In some instances, each of Al systems 164A-C may be configured to self-monitor their own outputs to determine when anomalies occur. For example, Al systems 164A-C may be configured with one or more rules, or with another Al system, neural netw ork model, and the like to determine when unusual or out-of-range confidence values occur or are generated by Al systems 164A-C, when Al systems 164A-C may confuse between or among different object classifications (e.g.. confusing a Tuktuk with a motorcycle or a bicycle) based on a probability distribution that is an output of the classification layer of the neural network model. In another example, Al systems 164A-C may be configured to identify tracking inconsistency with an obj ect detection algorithm. The tracking inconsistency may occur when the neural network models in Al systems 164A-C may trigger a tracking of an object, but the tracking mechanism does not engage or is waiting for the Al systems 164A-C to process more image frames to begin tracking. In another example, Al systems 164A-C may identifyP230033-724-W001 70052.2050W001

[0086] that the same object may be detected for longer than a predetermined period of time, which may indicate an anomaly in object detection. In another example. Al systems 164A-C may detect objects that have abnormal dimensions, which may indicate an error in a training dataset corresponding to the detected objects or a misclassification error by the neural network models in the Al systems 164A-C. Notably, there may be other anomalies that Al systems 164A-C may detect, which are not discussed above.

[0087] In some embodiments, when each of Al sy stems 164A-C detects an anomaly, the data and / or information corresponding to the anomaly may be saved in a data buffer 302. In some embodiments, data buffer 302 may be a memory storage that may be communicatively coupled to Al systems 164A-C, imaging systems 100A-C, and the like. For example, with reference to Fig. 2, data buffer 302 may be included in memory component 204. In some instances, data buffer 302 may also be incorporated within imaging systems 100A-C. For example, with reference to Fig. 1, data buffer 302 may be incorporated into memory component 120. There may be a single data buffer 302 for each of Al systems 164A-C (not shown) or data buffer 302 that receives data and / or information from multiple Al systems 164A-C.

[0088] In some embodiments, data and / or information saved to data buffer 302 may include one or more of image frames associated with the anomaly, imaging system 100 identifier, location of the imaging system 100, timestamp, an alert or message generated by Al systems 164A-C, a description or information corresponding to why the image and / or data were included in data buffer 302, and the like.

[0089] Data buffer 302 may also be configured with or coupled to a filtering module 304. The filtering module 304 may receive the data and / or information from Al systems 164A-C or from data buffer 302 and determine whether to store the data and / or information in data buffer 302 or to remove the data and / or information. For example, filtering module 304 may determine whether to remove duplicate data and / or information from data buffer 302, whether to combine duplicate data and / or information within data buffer 302, and the like. Filtering module 304 may also scan the data and / or information in data buffer 302 to determine whether better and / or more interesting information is already stored in data buffer 302 prior to determining whether to store additional data and / or information. Filtering module 304 may also include one or more rules that determine whether the data and / or information should be stored in data buffer 302 or should be discarded.P230033-724-W001 70052.2050W001

[0090] In some instances, smart data buffer 302 may store a maximum number of images. The maximum number of images may be predetermined or configured. Alternatively, the maximum number of images may be associated with an amount of memory associated with smart data buffer 302.

[0091] In some instances, data buffer 302 may have a configurable amount of memory. For example, data buffer 302 may be configured to store image frames from one Al system in Al systems 164A-C (which would take less memory) or from multiple Al systems 164A-C (which would take more memory). For image and video intensive applications, data buffer 302 may also be configured to store image frames or videos taken over multiple days and / or weeks. Data buffer 302 may also be configured to separate confidential and non-confidential data and / or information. Data buffer 302 may also be configured to receive different types of data from different Al systems 164A-C, from various neural network models and / or algorithms within Al systems 164A-C, and the like.

[0092] In some instances, training dataset 306 may be generated from the data and / or information stored in data buffer 302. Training dataset 306 may include image frames that generated anomalies. For example, image frames with objects that Al systems 164 may have misclassified or not detected, may be retrained using training dataset 306.

[0093] Additionally, data in data buffer 302 may be used to recreate anomalies in an environment that may test Al system 164. Additionally, data in data buffer 302 may be analyzed to determine the reasons and root causes for the anomalies.

[0094] Although the embodiments discussed here are discussed with reference to Al systems 164A-C, the embodiments are also applicable to storing anomalies in data buffer 302 generated by other components 166 discussed in Figs. 1 and 2.

[0095] Fig. 4 is a block diagram for storing anomalous data in a data buffer, according to some embodiments. Fig. 4 illustrates image capture component 130 of imaging system 100, discussed in Fig. 1, capturing image frames and passing the image frames to Al system 164, algorithm 402, and other components 166. Al system 164 may include a convolutional neural network (CNN) 404. Other components 166 may include algorithm 406, object tracking module 408, and algorithm 410 that perform downstream tasks with an output generated by CNN 404 or another component in Al system 164.

[0096] Algorithm 402 may process the image frames to identify objects, hazards, and the like. Algorithm 402 may also detect anomalies in its processing of the image frames. WhenP230033-724-W001 70052.2050W001

[0097] an anomaly is detected, algorithm 402 may pass the image frame and data and / or information corresponding to the image frame to filtering module 304.

[0098] Similarly, CNN 404 may receive the image frames captured by image capture component 130. In some embodiments, CNN 404 may identify objects in the image frames, such as cars, bicycles, motorcycles, Tuk-Tuks, and the like. Further details for identifying objects in the image frames are discussed in Fig. 7. In some instances, CNN 404 may misclassify or fail to classify one or more objects. In those instances, CNN 404 may transmit the image frame and data and / or information (e.g., the misclassified object type, timestamp, object dimensions, etc.) to filtering module 304.

[0099] In other instances, CNN 404 may properly classify the objects and pass the classified objects for downstream tasks handled by algorithm 406, object tracking module 408, and / or algorithm 410. For example, object tracking module 408 may track the location and / or movement of an object identified across multiple image frames captured at different times. Algorithms 406 and / or 410 may perform other tasks that may depend on the identified object. In some instances, each of algorithms 406, 410, and object tracking module 408 may also identify' anomalies when processing objects identified by CNN 404, and may pass the information and / or data, including image frames, timestamps, alerts associated with the anomalies, etc., to filtering module 304.

[0100] As discussed above, filtering module 304 may be configured to determine which anomalies, including data and / or information, may be saved to data buffer 302. The configuration may be on a per- Al system 164, per-component in other components 166, or per-multiple Al systems 164 or other components 166 basis. The configuration may also be based on the type of anomaly, type of data (e.g., private or public), and the like. The configuration may also be changed on demand. In some instances, the configuration may be based on one or more rules or criteria or may include a neural network model that may process various anomalies and classify' whether the anomalies may or may not be saved in data buffer 302. Some example anomalies may include a tracking inconsistency, a false positive, a classification inconsistency, and a low confidence. A tracking inconsistency may occur when a tracked object may not receive a new candidate update because Al system 164 did not detect anything even though no occlusion is detected in the context of the image frame. A false positive may occur when random false positives appear in the background of the image frame. A classification inconsistency may occur when Al system 164 detects objects but is uncertain about the classification of the object or misclassifies the object. A lowP230033-724-W001 70052.2050W001

[0101] confidence occurs when an object is detected and tracked, but the confidence levels corresponding to the object, location of the object, etc., are low, (e.g.. below a low confidence threshold).

[0102] As discussed above, data buffer 302 may include image frames and information associated with the image frames, such as timestamps, alerts, messages, and other information that may describe the anomalies associated with the image frames.

[0103] As discussed above, data buffer 302 may store a maximum number of image frames. Before the maximum number of image frames is reached, data buffer 302 may store image frames and corresponding information generated by CNN 404, algorithm 406, obj ect tracking module 408, and / or algorithm 410. Once the maximum number of image frames is reached, filtering module 304 may identify less relevant or less important image frames from the image frames stored in data buffer 302 and remove the less relevant or less important image frames from data buffer 302. The relevance and importance of each image frame may be based on how large the anomaly or inconsistency is (or the probability of the anomaly or inconsistency compared to the true value) as determined by the CNN 404, algorithm 406, object tracking module 408, and / or algorithm 410. The relevance or importance of each image frame may also be scored by filtering module 304 based on a predetermined set of rules or criteria. The set of rules or criteria may depend on the information or messages corresponding to the image frames that describe the anomaly, based on different collection reasons, based on different types of anomalies, and the like. The set of rules or criteria may be configured on demand. The relevance and importance may indicate how crucial the image frame is to training CNN 404 in Al system 164 and / or one or more algorithms 406-410 in the other components 166.

[0104] Filtering less relevant or important image frames from data buffer 302 has multiple benefits. Due to filtering, over time, image frames that are crucial to training CNN 404 in Al system 164 and / or one or more algorithms 406-410 may be stored in data buffer 302 and may be used to further train and / or optimize CNN 404 in Al system 164 and / or one or more algorithms 406-410. Further, if data buffer 302 is constrained to store maximum number of image frames, filtering module 304 may intelligently determine which image frames are crucial enough to be stored based on relevance and importance as specified by predetermined rules or criteria.P230033-724-W001 70052.2050W001

[0105] Fig. 5 is a block diagram 500 illustrating the training of Al systems, according to some embodiments. As discussed above, data buffer 302 may store data and / or information corresponding to anomalies. As also discussed above, training dataset 306 may be generated from the data and / or information in data buffer 302. The training dataset 306 may be used to retrain components of Al systems 164A-C, such as CNN 404. Additionally, the training dataset 306 may be used to identify reasons for the anomalies in downstream tasks, such as those executed by algorithms 406 and 410, and object tracking module 408.

[0106] In some instances, the training dataset 306 may be used to retrain some or all Al systems 164A-C. Further, data in data buffer 302 may be extracted to generate different types of training datasets 306. For example, one training dataset 306 may include image frames, another training dataset 306 may include public data, while yet another training dataset 306 may include private or non-public data.

[0107] Fig. 6 illustrates a process for storing data associated with anomalies in a data buffer, in accordance with an embodiment of the disclosure. One or more of the operations 602-614 of method 600 may be implemented, at least in part, in the form of executable code stored on non-transitory, tangible, machine-readable media that, when run by one or more processors, may cause the one or more processors to perform one or more of the operations 602-614.

[0108] At operation 602, an image frame is received. For example, algorithm 402 or Al system 164 may receive an image frame captured by image capture component 130 in imaging system 100.

[0109] At operation 604, an anomaly is detected in the processing of the image frame. For example, as Al system 164 processes the image frame, one or more algorithms in Al system 164 may detect an anomaly that occurs as the image frame is processed. For example, a neural network model, such as CNN 404 in Al system 164, may misclassify an object in the image frame, fail to detect the object, fail to detect the dimensions of the object, and the like. In another example, algorithm 402 may also process the image and identify an anomaly as part of its processing. In another example, an anomaly may be detected in one of the downstream tasks, such as those performed by algorithms 406, 410, and / or object tracking module 408. For example, CNN 404 may properly classify7an object in the image frame, but applications and / or algorithms that process the object may incorrectly process it.

[0110] At operation 606, a determination is made as to whether to store the data and / or information associated with the anomaly in a data buffer. For example, fdtering module 304P230033-724-W001 70052.2050W001

[0111] may determine whether to store the data and / or information corresponding to the anomaly in data buffer 302. At operation 606, if data buffer 302 stores more than a maximum number of image frames, filtering module 304 may also determine whether to remove the data and information already stored in data buffer 302 to create space for the data and information corresponding to the anomaly that is determined in operation 604. If so, method 600 proceeds to operation 608. Otherwise, method 600 ends at operation 610.

[0112] At operation 608, data and / or information corresponding to the anomaly is stored in a data buffer. For example, filtering module 304 may store the data and / or information, including the image frame, timestamp, and an alert message that is indicative of the anomaly, in data buffer 302.

[0113] Fig. 7 is a simplified diagram illustrating the neural network structure that may be implemented in one or more neural network models in Al system 164, according to some embodiments. A neural network model may include a perceptron neural network, a feed forward neural network, a multilayer perceptron network, a convolutional neural network, a radial basis functional neural network, a recurrent neural network, an LSTM (Long Short-Term Memory) network, and the like.

[0114] The neural network models may comprise a neural network architecture. The example neural network architecture may comprise an input layer 702, one or more hidden layers 704 and an output layer 706. The neural network models may be built as a collection of connected units or nodes, referred to as neurons 708. Each layer 702, 704, or 706 may comprise the same or different number of neurons or nodes 708, with neurons between layers being interconnected according to a specific topology. Each neuron 708 may be associated with an adjustable weight. The neurons 708 may be aggregated into layers 702, 704, 706 such that different layers perform different transformations on the respective input to generate a transformed output, which is an input for the subsequent layer. Further, different layers in neural network models may be combined into their own neural network models, such that an output layer of one neural network model, is an input into the next neural network model, until a final output layer 706 is reached. The number of layers 704 and neurons 708 within each layer may vary depending on the complexity and type of the neural network model.

[0115] Input layer 702 receives input data. The input data may be image data including image frames received from image capture component 130 discussed in Fig. 1. The numberP230033-724-W001 70052.2050W001

[0116] of nodes (neurons) in the input layer 702 may be determined by the dimensionality of the input data (e.g., a three-dimensional array having height, width, and color channels).

[0117] The hidden layers 704 are intermediate lay ers located between the input and output layers 702. 706 of the neural network models. Although three hidden layers 704 are shown, there may be any number of hidden layers in the neural network model. Generally, neural network models with more layers are more computationally intensive and / or accurate, while neural network models with fewer hidden layers are less computationally intensive and / or accurate. Hidden layers 704 may extract and transform the input data through series of weighted computations and activation functions associated with individual neurons.

[0118] For example, the neural network models may receive image frames at input layer 702 and generate output of output layer 706, which may be objects. To perform the transformation, each neuron 708 receives input signals (which may be input to the neural network model or an output of the preceding layer), performs a weighted sum of the inputs according to weights assigned to each connection and then applies an activation function associated with the respective neuron 708 to the result. The output of the neuron is passed to the next layer of neurons or serves as the final output of the network. The activation function may be the same or different across different layers 702, 704, 706 and may be different at neurons 708 within each layer. Example activation functions include but are not limited to Sigmoid, hyperbolic tangent, Rectified Linear Unit (ReLU), Leaky ReLU, Softmax, and / or the like. In this way, input data received at the input layer 702 is transformed by hidden layers 704 into different values indicative of data characteristics corresponding to a task that the neural network models have been trained to perform.

[0119] In some embodiments, hidden layers 704 may further be combined in lay ers and blocks. In a non-limiting embodiment, hidden layers 704 may be combined into one or more of convolutional layer(s), pooling layer(s), flattening layer(s), and / or fully connected layer(s). A convolutional layer(s) may detect features in an image frame using one or more filters. As an image frame passes through the filters in the convolution layer(s), convolutional layer(s) generate feature maps. Each filter may identify a specific feature in the image that corresponds to a feature map. A pooling layer may reduce the dimensions, e.g., height and width of the feature maps while retaining essential features in the feature maps. In some embodiments, convolutional layer(s) and pooling layer(s) may be stacked or interspersed with each other creating a deep neural network that may learn complex features. A flattening layer may follow one or more convolutional layer(s) and pooling layer(s). A flattening layer mayP230033-724-W001 70052.2050W001

[0120] flatten the feature maps that are the output of the preceding convolution layer or pooling layer to generate a one-dimensional vector. A fully connected layer includes one or more hidden layers 704 where neuron 708 of a preceding layer is connected to each neuron in the next layer (e.g., as shown in Fig. 7). A fully connected layer may receive the one-dimensional vector and combine different features identified from the convolutional layer(s), pooling layer(s) and flattening layer(s) to identify one or more objects in the image frame.

[0121] The output layer 706 is the final layer of the neural netw ork structure. It produces the network’s output or prediction based on the computations performed in the preceding layers (e.g., 702, 704). The number of nodes in the output layer depends on the nature of the task being addressed. For example, in a binary classification problem, the output layer may consist of a single node representing the probability of belonging to one class. In a multi-class classification problem, the output layer may have multiple nodes, each representing the probability of belonging to a specific class. In some embodiments, each specific class may correspond to objects in the image frame. In some instances, output layer 706 may use a softmax function that determines probabilities that objects in the image frame correspond to different classes, or if an object belongs to an unknown class.

[0122] Neural netw ork models may also be implemented by hardware, software, and / or a combination thereof. For example, neural network models may comprise a specific neural network structure implemented and run on various hardware platforms, such as but not limited to CPUs (central processing units), GPUs (graphics processing units), FPGAs (field-programmable gate arrays), Application-Specific Integrated Circuits (ASICs), dedicated Al accelerators like TPUs (tensor processing units), and specialized hardware accelerators designed specifically for the neural network computations described herein, and / or the like. Example specific hardware for neural network structures may include, but not limited to Google Edge TPU, Deep Learning Accelerator (DLA), NVIDIA Al-focused GPUs, and / or the like. The hardw are may be used to implement the neural network structure is specifically- configured based on factors such as the complexity of the neural network, the scale of the tasks (e.g., training time, input data scale, size of training dataset, etc.), and the desired performance.

[0123] Neural network models may be trained by iteratively updating the underlying weights of the neurons 708, bias parameters, and / or coefficients in the activation functions associated with neurons 708. The weights may be updated based on a loss function, such as a mean squared estimation error (MSEE), cross-entropy loss, log-loss, and the like. For example,P230033-724-W001 70052.2050W001

[0124] during training, the training data such as few-shot examples, APIs, queries, etc., are fed into neural network model over thousands of iterations. The training data flows through the network’s layers 702, 704, 706, with each layer performing computations based on its weights, biases, and activation functions until the output layer 706 produces the output.

[0125] The training data may be labeled with an expected output (e.g., a “ground truth” and a corresponding ground truth label). For example, image frames in the training dataset may be labeled with objects included in the corresponding image frames. The output generated by the output layer 706. e g., the classifications of the objects in the image frames are compared to the expected output, e.g., the labels in the image frames from the training data to compute a loss function that measures the discrepancy between the predicted output and the expected output. In another example, image frames in the training dataset may be labeled with distances of objects included in the corresponding image frames that are based on the objects’ pixel size. The output generated by the output layer 706, e.g., the classifications of the distances of objects in the image frames are compared to the expected output, e.g., the labels in the image frames from the training data to compute a loss function that measures the discrepancy between the predicted output and the expected output. In some embodiments, the negative gradient of the loss function may be computed with respect to the weights of each layer individually. This negative gradient is computed one layer at a time, iteratively backward from the last layer 706 to the input layer 702 of the neural network models. These gradients quantify the sensitivity of the network’s output to changes in the parameters. The chain rule of calculus is applied to efficiently calculate these gradients by propagating the gradients backward (in a back propagation network) from the output layer 706 to the input layer 702.

[0126] Parameters of the neural network are updated backwardly from the last layer to the input layer (backpropagating) based on the computed negative gradient using an optimization algorithm to minimize the loss. The backpropagation from the last layer 706 to the input layer 702 may be conducted for a number of training samples in a number of iterative training epochs. In this way, parameters of the neural network models may be gradually updated in a direction to result in a lesser or minimized loss, indicating the neural network has been trained to generate a predicted output value closer to the target output value with improved prediction accuracy. Training may continue until a stopping criterion is met, such as reaching a maximum number of epochs or achieving satisfactory performance on the validation data.P230033-724-W001 70052.2050W001

[0127] In a multiple neural network embodiment, the neural network models may be trained separately and then combined together and trained as a single neural network model.

[0128] Neural network parameters may be trained over multiple stages. For example, initial training (e.g., pre-training) may be performed on one set of training data, and then an additional training stage (e.g., fine-tuning) may be performed using a different set of training data, such as machine-readable code in one or more programming languages. In some embodiments, all, or a portion of parameters of one or more neural-network models being used together may be frozen, such that the "frozen" parameters are not updated during that training phase. This may allow; for example, a smaller subset of the parameters to be trained without the computing cost of updating all the parameters.

[0129] Therefore, the training process transforms the neural netw ork into an '‘updated” trained neural network with updated parameters such as weights, activation functions, and biases. The trained neural netw ork thus improves neural network technology for generating executable queries that may be executed by a database, another application interface, and the like to retrieve data.

[0130] Once training is complete, the trained neural network models may enter an inference stage where neural network models may be incorporated into Al system 164 and used to generate responses to various prompts.

[0131] Where applicable, various embodiments provided by the present disclosure can be implemented using hardw are, software, or combinations of hardw are and softw are. Also, where applicable, the various hardware components and / or software components set forth herein can be combined into composite components comprising software, hardware, and / or both without departing from the spirit of the present disclosure. Where applicable, the various hardw are components and / or softw are components set forth herein can be separated into sub-components comprising softw are, hardware, or both without departing from the spirit of the present disclosure. In addition, where applicable, it is contemplated that software components can be implemented as hardware components, and vice-versa.

[0132] Software in accordance with the present disclosure, such as program code and / or data, can be stored on one or more computer readable mediums. It is also contemplated that software identified herein can be implemented using one or more general purpose or specific purpose computers and / or computer systems, networked and / or otherwise. Where applicable,P230033-724-W001 70052.2050W001

[0133] the ordering of various steps described herein can be changed, combined into composite steps, and / or separated into sub-steps to provide features described herein.

[0134] Embodiments described above illustrate but do not limit the invention. It should also be understood that numerous modifications and variations are possible in accordance with the principles of the present invention. Accordingly, the scope of the invention is defined only by the following claims.

Claims

CLAIMSWhat is claimed is:

1. A method comprising:receiving data captured by an imaging system;determining, using one or more components in an artificial intelligence (Al) system, an anomaly in processing of the data by the one or more components in the Al system;determining to store the data corresponding to the anomaly in a data buffer; and storing the data corresponding to the anomaly in the data buffer.

2. The method of claim 1, wherein the data includes an image frame or a video.

3. The method of claim 1. wherein the Al system is included in the imaging system.

4. The method of claim 1, wherein the Al system is communicatively coupled to the imaging system.

5. The method of claim 1, further comprising:generating a training dataset from the data in the data buffer; and retraining the Al system using the training dataset.

6. The method of claim 1, wherein determining to store the data corresponding to the anomaly in the data buffer further comprises:configuring a filter module with one or more rules; anddetermining using the filtering module to store the data in the data buffer based on the one or more rules.

7. The method of claim 1, further comprising:configuring the data buffer to store different types of data.

8. A network system comprising:a logic device configured to:receive data captured by a plurality' of processing systems;determine, using one or more components in an artificial intelligence (Al) system, a plurality of anomalies in processing of the data;determine to store a subset of the data corresponding to the plurality of anomalies in a data buffer; andstore the subset of the data corresponding to the plurality of anomalies in the data buffer.

9. The system of claim 8, wherein the data includes an image frame or a video of a real-world environment.

10. The system of claim 8, wherein the Al system is included in each processing system in the plurality of processing systems.

11. The system of claim 8, wherein the Al system is communicatively coupled to the plurality of processing systems.

12. The system of claim 8, further comprising:generate a training dataset from the data in the data buffer; andretrain the Al system using the training dataset.

13. The system of claim 8, wherein to determine to store the subset of data corresponding to the plurality of anomalies in the data buffer further comprises:configure a filter module with one or more rules; anddetermine using the filtering module to store the data in the data buffer based on the one or more rules.

14. The system of claim 8, further comprising:configure the data buffer to store different types of data.

15. A non-transitory computer readable medium having instructions stored thereon, that when executed by a processor cause the processor to perform operations, the operations comprising:receiving an image frame captured by an imaging system;determining, using one or more components, an anomaly in processing of the image frame by the one or more components;determining to store the image frame and data corresponding to the anomaly in a data buffer; andstoring the image frame and the data corresponding to the anomaly in the data buffer.

16. The non-transitory computer readable medium of claim 15, wherein the one or more components are included in the imaging system.

17. The non-transitory computer readable medium of claim 15, wherein the one or more components are communicatively coupled to the imaging system.

18. The non-transitory computer readable medium of claim 15, wherein the operations further comprisegenerating a training dataset from image frames in the data buffer; and retraining the one or more components using the training dataset.

19. The non-transitory computer readable medium of claim 15, wherein to determining to store the data corresponding to the anomaly in the data buffer the operations further comprise:configuring a filter module with one or more rules; anddetermining using the filtering module to store the image frame in the data buffer based on the one or more rules.

20. The non-transitory computer readable medium of claim 15, wherein the operations further comprise configuring ty pe of image frames to be stored in the data buffer.

Images