Image processing system

Abstract

An image processing system (100), comprising an image sensor (102), a signal transport system (104), a first signal processing system (106) The image sensor (102) configured to obtain (1002) data related to an image. Further, the image sensor (102) configured to provide (1004) the data to the signal transport system. The signal transport system (104) configured to receive (1006) the data provided by the image sensor. Further, the signal transport system (104) configured to provide (1008) the data to the first signal processing system. The first signal processing system (106) configured to provide (1010), to a second signal processing system (124), a second indication (2) related to the data. The second indication is determined based on a first indication (4) based on the data exceeding a detection threshold.

Description

[0001] IMAGE PROCESSING SYSTEM

[0002] TECHNICAL FIELD

[0003] The invention relates to an image processing system. Further, the invention relates a method performed by an image processing system. The invention further relates to a computer program, a computer-readable medium, and a carrier.

[0004] BACKGROUND

[0005] Always-on applications such as image recognition, pattern detection and digital computations, such as algorithm design, adaptive computing, error detection and correction, user interface design, machine learning, are complex and energy intensive processes. There is a need to optimize the performance of such applications and computations on always-on, low-power and battery powered devices. An example is shown in US 11,388,356 Bl.

[0006] SUMMARY

[0007] It is an object of the invention to enable efficiency enhancements in an image processing system.

[0008] A first aspect of the invention relates to an image processing system. The image processing system comprises an image sensor configured to obtain data related to an image. Further, the image sensor is configured to provide the data to a signal transport system. Furthermore, the image processing system comprises a signal transport system configured to receive the data provided by the image sensor. Further, the signal transport system is configured to provide the data to the first signal processing system. Furthermore, the image processing system comprises a first signal processing system configured to provide, to a second signal processing system, a second indication related to the data. The second indication is determined based on a first indication based on the data exceeding a detection threshold.

[0009] The first signal processing system, and / or the second signal processing system, may be configured to carry out an algorithm.

[0010] The second indication may initiate a triggering of the second signal processing system, and / or the second indication may comprise a trigger initiating an activation of the second signal processing system.

[0011] The signal transport system may comprise one or more selectors configured to provide the data to one or more analog-to-digital converters, ADCs. Further, the one or more selectors are configured to provide the data to one or more analog-to-digital converters, ADCs based on a selector indication from the first signal processing system; provide the data to the second signal processing system; and / or provide the data to the second signal processing system based on a selector indication from the first signal processing system.

[0012] The first signal processing system may comprise at least one of a first analog signal processing system, a first analog-digital hybrid signal processing system, and a first digital signal processing system. The second signal processing system may comprise at least one of a second analog signal processing system, a second analog-digital hybrid signal processing system, and a second digital signal processing system.

[0013] The signal transport system may comprise one or more ADCs configured to convert the data to digital data. The one or more ADCs may be configured to provide an indication related to the digital data to the second signal processing system. The signal transport system may comprise one or more regulators configured to regulate a signal carrying the data. The signal transport system may comprise one or more memory arrays. The memory arrays configured to buffer the data; and in response to the data being buffered, and / or in response to a buffer indication, provide the buffered data to the first signal processing system.

[0014] The image sensor may comprise at least one of one or more pixels; one or more global shutter pixels; and, one or more rolling shutter pixels. The data related to an image may be obtained using at least one of a subset of one or more pixels; a subset of one or more global shutter pixels; and, a subset of one or more rolling shutter pixels.

[0015] The first signal processing system may comprise a set of crossbar arrays comprising at least one crossbar array. The subset of the set of crossbar arrays may be adapted to carry-out an algorithm. At least one crossbar array may comprise a set of memristors comprising at least one memristor. A memristor may be a titanium dioxide memristor, a silicon dioxide memristor, a polymeric memristor, and / or a ferroelectric memristor. A subset of the set of memristors may be adapted to carry-out an algorithm. The algorithm may comprise at least one of a machine learning algorithm; a pattern detection algorithm; a neural network algorithm; a face detection algorithm; and a gesture detection algorithm. The first signal processing system may be configured to obtain a feedback indication from the second signal processing system. The first signal processing system may be configured to adapt a setup of the first signal processing system based on a feedback indication from the second signal processing system. To adapt a setup of the first signal processing system may comprise to adapt one or more parameters of the first signal processing system. To adapt a setup of the first signal processing system, may comprise to adapt one or more parameters of a subset of the set of crossbar arrays. At least one of the one or more parameters may be at least one of: a memristance; a resistance; a capacitance; and, an inductance.

[0016] A second aspect of the invention relates to a method performed by an image processing system. The image processing system comprises: an image sensor, a signal transport system, a first signal processing system. The method comprises the image sensor obtaining data related to an image. Further, the method comprises the image sensor providing the data to the signal transport system. Furthermore, the method comprises the signal transport system receiving the data provided by the image sensor. Furthermore, the method comprises the signal transport system providing the data to the first signal processing system. Furthermore, the method comprises the first signal processing system providing, to a second signal processing system, a second indication related to the data. The second indication is determined based on a first indication. The first indication is based on the data exceeding a detection threshold.

[0017] The first signal processing system, and / or the second signal processing system, may carry out an algorithm.

[0018] The second indication may initiate a triggering of the second signal processing system, and / or the second indication may comprise a trigger initiating an activation of the second signal processing system.

[0019] The signal transport system may comprise one or more selectors. The one or more selectors providing the data to one or more analog-to-digital converters, ADCs; providing the data to one or more analog-to-digital converters, ADCs based on a selector indication from the first signal processing system; providing the data to the second signal processing system; and / or providing the data to the second signal processing system based on a selector indication from the first signal processing system.

[0020] The first signal processing system may comprise at least one of a first analog signal processing system, a first analog-digital hybrid signal processing system, and a first digital signal processing system. The second signal processing system may comprise at least one of a second analog signal processing system, a second analog-digital hybrid signal processing system, and a second digital signal processing system. The signal transport system may comprise one or more ADCs. The one or more ADCs converting the data to digital data. The one or more ADCs may provide an indication related to the digital data to the second signal processing system. The signal transport system may comprise one or more regulators. The one or more regulators regulating a signal carrying the data. The signal transport system may comprise one or more memory arrays. The one or more memory arrays buffering the data; and, in response to the data being buffered, and / or in response to a buffer indication, providing the buffered data to the first signal processing system.

[0021] The image sensor may comprise at least one of: one or more pixels; one or more global shutter pixels; and, one or more rolling shutter pixels. The data related to an image may be obtained using at least one of: a subset of one or more pixels; a subset of one or more global shutter pixels; and, a subset of one or more rolling shutter pixels.

[0022] The first signal processing system may comprise a set of crossbar arrays comprising at least one crossbar array. A subset of the set of crossbar arrays may be adapted to carry-out an algorithm. At least one crossbar array may comprise a set of memristors comprising at least one memristor. A memristor may be a titanium dioxide memristor, a silicon dioxide memristor, a polymeric memristor, and / or a ferroelectric memristor. A subset of the set of memristors may be adapted to carry-out an algorithm. The algorithm may comprise at least one of: a machine learning algorithm; a pattern detection algorithm; a neural network algorithm; a face detection algorithm; and a gesture detection algorithm. The first signal processing system may obtain a feedback indication from the second signal processing system. The first signal processing system may adapt a setup of the first signal processing system based on a feedback indication from the second signal processing system. Adapting a setup of the first signal processing system, may comprise adapting one or more parameters of the first signal processing system. Adapting a setup of the first signal processing system, may comprise adapting one or more parameters of a subset of the set of crossbar arrays. At least one of the one or more parameters may be at least one of: a memristance; a resistance; a capacitance; and, an inductance.

[0023] A third aspect of the invention relates to a tangible, non-transient computer-readable medium comprising instructions that, when executed by processing circuitry, cause the processing circuitry to perform operations according to the method of the second aspect of the invention. A fourth aspect of the invention relates to a computer program, comprising instructions that, when executed by processing circuitry, cause the processing circuitry to carry out the method according to the second aspect of the invention.

[0024] A fourth aspect of the invention relates to a carrier containing the computer program according to the fourth aspect of the invention, wherein the carrier is one of an electronic signal, optical signal, radio signal, computer program product, or computer-readable medium.

[0025] BRIEF DESCRIPTION OF THE DRAWINGS

[0026] Figure 1 illustrates an example of an image processing system in accordance with some embodiments.

[0027] Figure 2 illustrates an example of the image processing system in accordance with some embodiments.

[0028] Figure 3 illustrates the image processing system and the second signal processing system in accordance with some embodiments.

[0029] Figure 4 illustrates an example of the image processing system in accordance with some embodiments.

[0030] Figure 5 illustrates an example of the image processing system in accordance with some embodiments.

[0031] Figure 6 illustrates an example of the image processing system in accordance with some embodiments.

[0032] Figure 7 illustrates an example of the image processing system in accordance with some embodiments.

[0033] Figure 8 illustrates an example of the image processing system in accordance with some embodiments.

[0034] Figure 9 illustrates a flowchart of a process performed by the image processing system and the second signal processing system according to some embodiments.

[0035] Figure 10 illustrates a method performed by the image processing system according to some embodiments.

[0036] Figure 11 illustrates a computer program, a non-transitory computer readable medium, and a computer program product.

[0037] DETAILED DESCRIPTION Some of the embodiments contemplated herein will now be described more fully with reference to the accompanying drawings. Embodiments are provided by way of example to convey the scope of the subject matter to those skilled in the art.

[0038] Figure 1 illustrates an example of the image processing system 100 in accordance with some embodiments.

[0039] In the example, the image processing system 100 comprises an image sensor 102, a signal transport system 104, and a first signal processing system 106.

[0040] According to a first embodiment, the image sensor 102 is configured, or adapted, to obtain, e.g., monitor, acquire, detect, impress, receive, etc., data related to an image. The image sensor 102 is further configured to provide, e.g., supply, deliver, transmit, distribute, the data to the signal transport system 104.

[0041] The image sensor 102 is a hardware component that converts optical images, e.g., photons at specific wavelengths, into electronic signals, e.g., analog electronic signals, and / or digital electronic signals. An image sensor may be found in devices such as digital cameras, smartphones, and medical imaging equipment. Image sensors are composed of one or more photodetectors, often called pixels, that convers light intensity into electrical signals.

[0042] The image sensor 102 may comprise at least one of: one or more pixels; one or more global shutter pixels; and, one or more rolling shutter pixels. In this case, the data related to an image may be obtained using at least one of: a subset of one or more pixels; a subset of one or more global shutter pixels; and, a subset of one or more rolling shutter pixels. As an example, the image sensor 102 may comprise one or more global shutter pixels, one or more rolling shutter pixels.

[0043] A global shutter pixel is a type of pixel design found in image sensors, that allows for the simultaneous exposure and possible simultaneous readout of all pixels in an image sensor. The simultaneous exposure eliminates the temporal skew that can occur in a rolling shutter, where different parts of the image are captured at different times. The simultaneous exposure further eliminates motion artifacts such as skew, wobble, and partial exposure that are common in rolling shutter systems. However, a global shutter pixel often requires a storage element to be able to store a photodiode value in the pixel before sending the information. This means that the global shutter pixel will be larger and more complex than the rolling shutter pixel described below, and, to follow, higher power consumption when the global shutter pixel is active. A rolling shutter pixel is type of pixel design found in image sensors, where the exposure of the pixel in an image sensor and readout of the image data occur sequentially, as an example, the exposure of the pixel occurs per row and the readout of the image data is per pixel. The sequential exposure of the rolling shutter pixels may cause motion artifacts. However, a rolling shutter pixel is often smaller and simpler than a global shutter pixel, and thus have lower power consumption when active.

[0044] As an example, the image sensor 102 is comprised by one or more global shutter pixels. This enables the image sensor 102 to avoid artifacts generated by the rolling shutter pixels, with the drawback of higher complexity of the image sensor.

[0045] As another example, the image sensor 102 is comprised by one or more rolling shutter pixels. This enables image sensors of lower complexity, with the drawbacks of the presence of artifacts generated by the rolling shutter pixels.

[0046] As another example, the image sensor 102 is comprised by a combination of one or more global shutter pixel, and one or more rolling shutter pixels. In this way, as an example, a subset of the rolling shutter pixels may be activated to acquire the data related to an image for use of the first signal processing system, and, depending on the first signal processing system, the remaining pixels, i.e., global shutter pixels and rolling shutter pixels, may be activated to acquire a higher quality image. In another example, a subset of the global shutter pixels may be activated to acquire the data related to an image for use of the first signal processing system, and, depending on the first signal processing system, the remaining pixels, i.e., global shutter pixels and rolling shutter pixels, may be activated to acquire a higher quality image. According to the first embodiment, the signal transport system 104 is configured, or adapted, to receive, e.g., monitor, acquire, detect, impress, obtain, etc., the data provided by the image sensor 102, and provide, e.g., supply, deliver, transmit, distribute, the data to the first signal processing system 106. As an example, the signal transport system 104 may be an analog signal transport system, a digital signal transport system, and / or an analog-digital hybrid signal transport system.

[0047] A signal transport system in this application, is involved in the transmission and distribution of a signal from the image sensor to the first signal processing system. In its most basic form, a signal transport system, may be a collection of wires for the transmission of the signal, e.g., electrical traces, signal traces, silicone wiring, coaxial cables, twisted pairs, fiber optics. A signal transport system, as will be better described in the following embodiments, may also comprise other components, such as analog-to-digital converter (ADC), digital-to-analog converter (DAC), selector, regulator, memory array, which perform processing on the signal. According to the first embodiment, the first signal processing system 106 is configured, or adapted, to provide, e.g., supply, deliver, transmit, distribute, to a second signal processing system, a second indication related to the data. The second indication is determined based on a first indication. The first indication is based on the data exceeding a detection threshold. A first indication may be based on a number of pixels in a certain order, and / or at a certain analog value being above a certain threshold. Further, a first indication may be based on the values, as example intensity and / or magnitude, of a number of pixels, or transformed values of pixels, matching with values stored in a number of memory cells, e.g. memristor cells, in a crossbar array while the matching distance is above a certain threshold. Further, a first indication may be based on the emergence of certain numerical patterns in the output vector obtained in crossbar columns, i.e., bit-lines, such as all vector elements are zero except for a few non-zero values, Furthermore, the first indication may be an output signal of the first signal processing system indicating that the data exceed a detection threshold.

[0048] A second indication may be pixel values from the full possible area connected to the first signaling processing system or part of an area connected to the first signaling processing system giving an output after processing the analog values that is over a threshold. Further, the second indication may be a copy of first indication jointly with pixel values from the full possible areas connected to the first signal processing system. The second indication may be the first indication. The second indication may indicate that the data exceed a detection threshold. The second indication may initiate a triggering of the second signal processing system, and / or the second indication may comprise a trigger initiating an activation of the second signal processing system.

[0049] A detection threshold may be a value, e.g., a predefined value, a dynamic value, which determines whether a particular feature or object is present in the image. As an example, a detection threshold may be an output from one or more neural networks, containing one or more layers, detecting one or more objects. As another example, a detection threshold may be a pre-determined value above which the result of multiplication of input pixels with a stored pattern in the crossbar array(s) is considered to have resulted in detection of a pattern like an object or shape.

[0050] A signal processing system, i.e., a first signal processing system 106, or a second signal processing system, may be system designed to analyze, process, and / or modify signals, as example data related to an image. A signal processing system may comprise at least one of: an analog signal processing system, an analog-digital hybrid signal processing system, and a digital signal processing system. A signal processing system may be designed to carry out an algorithm on a signal.

[0051] An analog signal processing system, i.e., a first analog signal processing system, or a second analog signal processing system, may be a signal processing system designed to process analog signals. An analog-digital hybrid signal processing system, i.e., a first analog-digital hybrid signal processing system, or a second analog-digital hybrid signal processing system, may be a signal processing system designed to process analog signals, and / or digital signals. A digital signal processing system, i.e., a first digital signal processing system, or a second digital signal processing system, may be a signal processing system designed to process digital signals.

[0052] A signal processing system may comprise a set of crossbar arrays comprising at least one crossbar array.

[0053] A crossbar array is a circuit architecture comprising memory cells, e.g., dynamic random access memory cell (DRAM), static random access memory cell (SRAM), memristors, organized in a grid of intersecting horizontal wires, also called word-lines, and vertical wires, also called bit-lines, where one or more memory cells reside at each intersection. One or more matrices can be programmed into the crossbar array, where each matrix element is programmed in one or more memory cells. A memory cell may be volatile like SRAM and DRAM cells or non-volatile like memristor memory devices. As an example, a crossbar array may comprise a set of memristors comprising at least one memristor.

[0054] In general, crossbar arrays can be used to detect patterns using machine learning algorithms, notably neural networks. In this case, the weight parameters of each layer of the neural network may be programmed in one or more crossbar arrays and the network output layer will determine the presence of one or more patterns of interest. Before programming the neural network to the crossbar array, the neural network must be trained. The training of the neural network may be done in digital domain, as example in the second signal processing system, locally in the one or more crossbar arrays, or in hybrid fashion, in both the second signal processing systems and in the one or more crossbar arrays. Further, the training of the neural network may be done remotely, in a training device, a cloud server, etc.

[0055] As an example, patterns like a hand or a circle mask can be programmed in crossbar array. These types of patterns could then be combined with other pattern programmed in other crossbar arrays in numerous configurations to create composite patterns, which are incredibly complex and performant. As another example, crossbar arrays can be programmed to perform pre-preprocessing algorithm, like signal filtering. As another example, crossbar arrays could also be layered to create deep neural networks, providing greater flexibility and range of pattern matching capabilities. This enables the use of machine learning for low-power application.

[0056] A memristor, also called memory resistor, is a passive circuit element whose resistance depends on the history of current passing through it, this concept is also called memristance. The memristance resistance can be changed by applying voltage or current, with this change being gradual or abrupt depending on the design. This change is typically due to the movement of ions or defects within a thin film of material sandwiched between two electrodes. The ability to exhibit a continuous range of resistance values makes memristors suitable for analog computing. Potential applications include non-volatile storage, neuromorphic computing, analog computing, and reconfigurable electronics. There are many available memristor technologies, for example titanium dioxide memristor, silicon dioxide memristor, polymeric memristor, ferroelectric memristor. Memristor-based crossbar arrays are efficient for matrix algebra operations like vector matrix multiplications (VMM) due to massive parallelization of computations as well as in-memory processing. Parallelization contributes to latency reduction and in-memory processing leads to drastic energy saving, due to reduction in data movement as the main culprit for energy usage in digital von-Neumann based computing architecture. Matrix algebra is a key operation in many signal processing domains like pattern recognition in sensor data, but it is also the building block for machine learning algorithms like neural networks, making memristor a solution for efficient acceleration of both standard signal processing as well as machine learning algorithms.

[0057] The signal processing system, or a part of it, e.g., a first signal processing system 106, a first analog signal processing system, a subset of the set of crossbar arrays, a subset of the set of memristors, etc., may be configured to carry out an algorithm. For example, a machine learning algorithm; a pattern detection algorithm; a neural network algorithm; a face detection algorithm; and a gesture detection algorithm.

[0058] An example of face detection algorithm application comprises the use of always-on image sensors. In such a case the image sensor usually may be without infrared-cut filter to be able to add sensitivity in the near infrared spectrum (NIR). The first signal processing system, e.g., a crossbar array, would be configured with the user’s face profile, to use in a matching procedure. In this way, this invention enables single image recognition, multiple image recognition or inference of a neural network to accomplish the task all done in the analog domain. If the image is recognized, the second signal processing system may be triggered to enable a more accurate image recognition, or to enable applications or features that require performant face detection.

[0059] An example of gesture detection algorithm application comprises the use of always-one image sensors. In such case the first signal processing system continuously run a gesture detection algorithm, and when a determined gesture is found, the second signal processing system is triggered to perform additional algorithms or applications.

[0060] Figure 2 illustrates an example of the image processing system 100 in accordance with some embodiments.

[0061] According to a second embodiment, the signal transport system 104 according to the first embodiment, comprises one or more selectors 108. The one or more selectors 108 configured to provide the data to one or more ADCs 110, and / or to provide the data to the second signal processing system. The data may be provided in response to the one or more selectors receiving a selector indication from the first signal processing system 106.

[0062] A selector 108 is a component of the signal transport system that can relay a signal to different components, depending on a selector indication signal and / or a configuration. As an example, with a first selector indication a selector may transfer data related to an image from the image sensor to the ADC, with a second selector indication selector may transfer data related to an image from the image sensor to the first signal processing system, and with a third selector indication a selector may transfer data related to an image from the image sensor to the ADC and to the first signal processing system.

[0063] A selector indication may indicate whether a signal, e.g., the data related to an image, is to be transferred to a first image processing system, to the ADC, or to both. A selector indication may be a signal, as example a voltage, or may be MOSFET switches, CMOS analog multiplexers, Bipolar Junction Transistor, or physical MEMS switches, etc. A selector indication may be a state machine that controls the selector depending on what state is it in. An ADC 110 is a component that converts an analog signal, such as a sound picked up by a microphone or light entering a camera, into a digital signal.

[0064] According to a second embodiment, the signal transport system 104 according to the first embodiment, comprises one or more ADCs 110. The one or more ADCs 110 configured to convert 1012 the data to digital data. Further, the one or more ADCs 110 are configured to provide 1014 an indication related to the digital data to a digital signal processing system. According to a third embodiment, the signal transport system 104 according to the first, and / or the second, embodiment, comprises one or more regulators 112. The one or more regulators 112 configured to regulate 1016 a signal carrying the data.

[0065] A regulator 112 is a component that regulates characteristics of a signal. A regulator may be a linear voltage regulator, a switching regulator, an amplifier, etc. As an example, the regulator maintains the voltage of the signal carrying the data related to an image so that is in a range accepted by the crossbar array. One or more regulators may be part of the first signal processing system and regulate the signal carrying data across the one or more crossbar arrays. A regulator may also comprise an activation function, e.g., linear activation function, rectifier linear unit activation function, sigmoid activation function.

[0066] According to a fourth embodiment, the signal transport system 104 according to the first, the second, and / or the third, embodiment, comprises one or more memory arrays 114. The one or more memory arrays 114 configured to buffer 1018 the data. Further, the one or more memory arrays 114 are configured to provide the buffered data to the first signal processing system 106, in response to the data being buffered, and / or in response to a buffer indication. A memory array is a component that buffer information. A memory array may be configured to buffer data until the buffer reached a certain capacity of stored data, until the buffer is full, until the memory array receives a buffer indication, until a certain time from the first buffered data has passed. A buffer indication may be an indication indicating, as an example, that the buffer reached a certain capacity, that a certain time from the first buffered that has passed, that a buffer is close to the maximum capacity, that the buffered data are ready to be processed by the first signal processing system, that the data related to an image have been obtained by the image sensor, etc. As an example, a memory array may be one or more delay lines, one or more crossbar arrays, one or more memristors, SRAM, DRAM. As an example, a memory array is used to buffer data of rolling shutter pixels until the sequential exposure is completed. As another example, a memory array is used to buffer data from the first signal processing system.

[0067] According to a fifth embodiment, the first signal processing system 110 according to the first, the second, the third, and / or the fourth, embodiment, is configured to obtain 1030 a feedback indication from the second signal processing system. Further, the first signal processing system 106 is configured to adapt 1032 a setup of the first signal processing system 106 based on a feedback indication from the second signal processing system. As an example, adapting 1032 a setup may comprise adapting one or more parameters of the first signal processing system. As another example, adapting 1032 a setup may comprise adapting one or more parameters of a subset of the set of crossbar arrays 116. The one or more parameters may be, as an example, a memristance, a resistance, a capacitance, and / or and inductance. Figure 3 illustrates the image processing system 100 and the second signal processing system 124 in accordance with some embodiments. According to figure 3, the image processing system 100, comprising the image sensor 102, the signal transport system 104, and the first signal processing system 106, is illustrated. Further, the second signal processing system 124 is illustrated. As previously described, the first signal processing system 106 is configured to provide, to the second signal processing system 124, a second indication related to the data. The second indication is determined based on a first indication based on the data exceeding a detection threshold. The second indication 2 may initiate a triggering of the second signal processing system, and / or the second indication 2 may comprise a trigger initiating an activation of the second signal processing system.

[0068] Figure 4 illustrates an example of the image processing system 100 in accordance with some embodiments. According to figure 4 the image sensor 102 is a matrix of global shutter (GS) pixels (GS) 126. Further, the signal transport system 104 comprises selectors (SL) 108 and ADCs 110. Furthermore, the first signal processing system 106 comprises a crossbar array that in turn comprises memristors (MEM) 122, the first signal processing system 106 further comprises a word-line driver 128, a bit-line driver 130, a controller 132. The image processing system may further comprises an analog comparator 134. An analog comparator may be a device used in analog signal processing to compare characteristics of the signal, as example, voltages, or currents, or powers. An analog comparator may perform additional signal processing to the one performed by the first signal processing system 106.

[0069] Each of the columns of the global shutter pixels 126 provides the data related to the acquired image to a selector 108. The selectors 108 may provide the data to both the first signal processing system 106 and to the ADCs 110. In an example the selector 108, may receive one or more selector indications from the controllers 132 on whether to transmit to provide the data to the first signal processing system 106, to the ADCs 110, or to both. In another example, the selectors 108 may receive one or more selector indications from the analog comparator 134. The word-line driver 128 and the bit-line driver 130 in the first signal processing system 106 control the data and driving voltage arriving to rows and columns of the crossbar array, following indications from the controller 132. The analog comparator 134 may perform additional signal processing in addition to the processing performed by the memristors 122.

[0070] In alternative, each of the rows of the global shutter pixels 126 may provide the data related to the acquired image to a selector 108. In another alternative each of the global shutter pixels 126 may provide the data related to the acquired image to a selector 108, this requires more selectors 108 than the previous two alternatives but may not require the word-line driver 128 and the bit-line driver 130.

[0071] The selectors may provide the data related to the acquired image to the ADCs 110, and / or to the first signal processing system 106, the ADCs provide the date related to the acquired image to the second signal processing system 124. The first signal processing system may provide a signal, as example an indication, or data related to the acquired image, to the second signal processing system 124.

[0072] Figure 5 illustrates an example of the image processing system 100 in accordance with some embodiments. Compared to the image processing system 100 illustrated in figure 4, the first signal processing system 106 illustrated in figure 5 comprises multiple crossbar arrays, which in turn comprise memristors 122. The crossbar arrays are set in parallel so to allow parallel processing of input data. Each of the crossbar arrays is configured to carry out an algorithm, allowing, for example, the parallel detection of different gestures, faces, shapes, objects, etc. The structure of each of the crossbar arrays and the pattern or parameters programmed in the crossbar arrays may differ in order for the respective crossbar array to be configured to carry out a different algorithm. The results of each of the crossbar array may be further processed in the analog comparator 134. The selectors 108 may provide the data to both the first signal processing system 106 and to the ADCs 110. In an example the selector 108, may receive one or more selector indications from the controllers 132 on whether to transmit to provide the data to the first signal processing system 106, to the ADCs 110, or to both. In another example, the selectors 108 may receive one or more selector indications from the analog comparator 134. A regulator 112 regulates the signal carrying the data, as example the voltage of the signal, or the power of the signal, so that it’s characteristics, as example voltage and power, are a range accepted by the crossbar arrays of the first signal processing system 106. In one example, the image processing system 100 comprises a single regulator 112 regulating the data to all the parallel crossbar arrays. In another example, the image processing system 100 comprises multiple regulators. Each of the regulators regulating the data to each of the parallel crossbar arrays.

[0073] Figure 6 illustrates an example of the image processing system 100 in accordance with some embodiments. Compared to the image processing system 100 illustrated in figures 4 and 5, the image sensor 102 is a matrix of rolling shutter (RS) pixels 136. Further, the signal transport system 104 comprises selectors 108 and ADCs 110. Furthermore, the signal transport system 104 comprises a memory array 114 comprising a crossbar array that in turn comprises memristors 122. The memory array 114 is configured to buffer the data related to the image coming from the rolling shutter pixels 136 until the rolling acquisition of the image is complete and then transmits the data to the first signal processing system 106. The memory array 114 may be connected to the image sensor 102 in order to receive a buffer indication. A regulator 112 may be placed between the memory array 114 and the first signal processing system 106 to regulate the signal carrying the buffered data.

[0074] Figure 7 illustrates an example of the image processing system 100 in accordance with some embodiments. Compared to the image processing system 100 illustrated in figures 4, 5 and 6„ the first signal processing system 106 illustrated in figure 7 comprises multiple crossbar arrays in series. By having multiple crossbar arrays in series, the output of a crossbar array is provided the next crossbar array. This enables the first signal processing system 106 to carry out neural network algorithm. As an example, each of the crossbar arrays is configured to carry out a different layer of a neural network algorithm, and the number of crossbar arrays in series is based on the number of layers of the neural network algorithm. A regulator 112A may be placed between the selectors 108 and the first signal processing system 106 to regulate the signal carrying the data related to an image. A regulator 112B may be placed between the output of a first crossbar array and the input of a second crossbar array to regulate the signal carrying the output data of the first crossbar array. The structure of each of the crossbar arrays, and the parameters programmed in each of the crossbar arrays, may differ in order for the crossbar array to be configured to carry out a different layer of the neural network algorithm. As an example, the structure of a crossbar array is represented by the number, the disposition, the type, and / or the memristance of the memristors that compose the crossbar array. The parameters of the neural network may be trained outside the image processing system and loaded into the crossbar arrays, or trained directly in the crossbar arrays. These parameters can be updated, or changed depending on what task the first signal processing system is configured to carry out. In general, the same crossbar arrays may be re-configured and used for carrying out other algorithms.

[0075] Figure 8 illustrates an example of the image processing system 100 in accordance with some embodiments. Compared to the image processing system 100 illustrated in figure 4, the image sensor 102 is a matrix of one or more rolling shutter pixels 136 and one or more global shutter pixels 126.

[0076] Further, compared to the image processing system 100 illustrated in figure 4, the image sensor 102 may use a subset of the global shutter pixels 126 and / or the rolling shutter pixels 136 for obtaining the data related to the image that is provided to the first signal processing system 106. As an example, only a subset of the global shutter pixels 126 is used by the image sensor 102. As another example, only a subset of the rolling shutter pixels 136 is used by the image sensor 102. As another example a mix of a subset of the global shutter pixel 126 and a subset of the rolling shutter pixels 136, is used by the image sensor 102. A subset of the pixels, as example RS and / or GS pixels, is activated for the acquisition of the data related to an image of the image sensor 102 that is provided to the first signal processing system. In this way, the data related to an image has a lower resolution, as only a subset of the pixels has been used. This enables a lower consumption of the image processing system 102. The full resolution data related to the image may be obtained following an indication from the first signal processing system 106, and may be provided to the second processing system 124 for further processing.

[0077] Figure 9 illustrates a flowchart of a process performed by the image processing system 100 (illustrated in figures 3-8) the second signal processing system 124 (illustrated in figures 3-8) according to some embodiments.

[0078] As illustrated in the flowchart, the image sensor, e.g., a subset of the pixels of the sensor, obtains data related to an image. The data related to the image is provided to the first signal processing system that carries out an algorithm based on the data, e.g., a pattern detection algorithm, a face recognition algorithm, an authorization algorithm. The first signal processing system produces a first indication based on data exceeding a detection threshold, e.g., based on the result of a pattern detection algorithm, or a face recognition algorithm. If the first indication indicates that data exceeds the detection threshold, a second indication trigger the activation of the second signal processing system. If the first indication indicates that data does not exceeds the detection threshold, new data related to an image is obtained by the image sensor and the process starts again. The second signal processing system obtains data related to an image and carries out an algorithm. The data related to an image may be obtained with higher resolution, e.g., by using a bigger subset of the pixels of the sensors. The second signal processing system, based on the algorithm result, may send a feedback indication to the first signal processing system that adapts its parameters based on the feedback indication. Once the feedback indication is sent and the parameters are adapted, new data related to an image is obtained by the image sensor and the process starts again. If the second signal processing system does not send the feedback indication, new data related to an image is obtained by the image sensor and the process starts again.

[0079] Figure 10 illustrates a method 1000 performed by the image processing system according to some embodiments. The method comprises the image sensor obtaining 1002 data related to an image. Further, the method comprises the image sensor providing 1004 the data to the signal transport system. Further, the method comprises, the signal transport system receiving 1006 the data provided by the image sensor. Further, the method comprises, the signal transport system providing 1008 the data to the first signal processing system. Further, the method comprises, the first signal processing system providing 1010, to a second signal processing system, a second indication related to the data, wherein the second indication is determined based on a first indication based on the data exceeding a detection threshold.

[0080] According to an example, the first signal processing system, and / or the second signal processing system, may be configured to carry out an algorithm.

[0081] According to an example, the second indication may initiate a triggering of the second signal processing system, and / or the second indication comprises a trigger initiating an activation of the second signal processing system.

[0082] According to an example, the signal transport system may comprise one or more selectors. The one or more selectors may provide 1012 the data to one or more analog-to-digital converters, ADCs; provide 1014 the data to one or more analog-to-digital converters, ADCs based on a selector indication from the first signal processing system; provide 1016 the data to the second signal processing system; and / or provide 1018 the data to the second signal processing system based on a selector indication from the first signal processing system.

[0083] According to an example, the first signal processing system may comprise at least one of: a first analog signal processing system, a first analog-digital hybrid signal processing system, and a first digital signal processing system. According to an example, the second signal processing system may comprise at least one of: a second analog signal processing system, a second analog-digital hybrid signal processing system, and a second digital signal processing system.

[0084] According to an example, the image processing system may comprise one or more ADCs, the one or more ADCs converting 1020 the data to digital data. According to an example, the one or more ADCs may provide 1022 an indication related to the digital data to the second signal processing system.

[0085] According to an example, the image processing system may comprise one or more regulators, the one or more regulators regulating 1024 a signal carrying the data.

[0086] According to an example, the signal transport system may comprise one or more memory arrays, the one or more memory arrays: buffering 1026 the data; and, in response to the data being buffered, and / or in response to a buffer indication, providing 1028 the buffered data to the first signal processing system.

[0087] According to an example, the image sensor may comprise at least one of: one or more pixels; one or more global shutter pixels; and, one or more rolling shutter pixels. According to an example, the data related to an image may be obtained using at least one of: a subset of one or more pixels; a subset of one or more global shutter pixels; and, a subset of one or more rolling shutter pixels.

[0088] According to an example, the first signal processing system may comprise a set of crossbar arrays comprising at least one crossbar array. According to an example, a subset of the set of crossbar arrays may be adapted to carry-out an algorithm.

[0089] According to an example, at least one crossbar array may comprise a set of memristors comprising at least one memristor. According to an example, a memristor may be a titanium dioxide memristor, a silicon dioxide memristor, a polymeric memristor, and / or a ferroelectric memristor. According to an example, a subset of the set of memristors may be adapted to carry-out an algorithm.

[0090] According to an example, the algorithm may comprise at least one of: a machine learning algorithm; a pattern detection algorithm; a neural network algorithm; a face detection algorithm; and a gesture detection algorithm.

[0091] According to an example, the first signal processing system may obtain 1030 a feedback indication from the second signal processing system. According to an example, the first signal processing system may adapt 1032 a setup of the first signal processing system based on a feedback indication from the second signal processing system. According to an example, adapting 1032 a setup of the first signal processing system, may comprise adapting one or more parameters of the first signal processing system. According to an example, adapting 1032 a setup of the first signal processing system, may comprise adapting one or more parameters of a subset of the set of crossbar arrays. According to an example, at least one of the one or more parameters may be at least one of: a memristance; a resistance; a capacitance; and, an inductance.

[0092] Figure 11 illustrates a carrier 202, a computer readable medium 204, and a computer program 206.

[0093] According to an embodiment, a tangible, non-transient computer-readable medium 204 comprises instructions that, when executed by the image processing system, cause the image processing system to perform operations according to the method performed by the image processing system.

[0094] According to an embodiment, a computer program 206 comprises instructions that when executed by the image processing system, cause the image processing system to carry out the method performed by the image processing system.

[0095] According to an embodiment, a carrier 202 may contain the computer program 206, wherein the carrier is one of an electronic signal, optical signal, radio signal, computer program product, or computer-readable medium.

[0096] Examples of non-transient computer-readable media are / comprises / includes a universal serial bus (USB) memory, a plug-in card, an embedded drive, or a read-only memory.

[0097] The image processing system may comprise processing circuitry, in that case the processing circuitry may execute the instructions comprised in the computer-readable medium 204, and / or in the computer program 206. The processing circuitry may be configured to process instructions and data and may be configured to implement any sequential state machine operative to execute instructions stored as machine-readable computer programs in the memory. The processing circuitry may be implemented as one or more hardware- implemented state machines (e.g., in discrete logic, field-programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), etc.); programmable logic together with appropriate firmware; one or more stored computer programs, general-purpose processors, such as a microprocessor or digital signal processor (DSP), together with appropriate software; or any combination of the above. For example, the processing circuitry may include multiple central processing units (CPUs).

[0098] The processing circuitry may comprise a combination of one or more of a microprocessor, controller, microcontroller, central processing unit, digital signal processor, applicationspecific integrated circuit, field programmable gate array, or any other suitable computing device, resource, or combination of hardware, software and / or encoded logic operable to provide, either alone or in conjunction with other components, such as the memory, to provide image processing system, first signal processing system, and / or second signal processing system functionality. In some embodiments, the processing circuitry includes a system on a chip (SOC).

[0099] The memory may comprise any form of volatile or non-volatile computer-readable memory including, without limitation, persistent storage, solid-state memory, remotely mounted memory, magnetic media, optical media, random access memory (RAM), read-only memory (ROM), mass storage media (for example, a hard disk), removable storage media (for example, a flash drive, a Compact Disk (CD) or a Digital Video Disk (DVD)), and / or any other volatile or non-volatile, non-transitory device-readable and / or computer-executable memory devices that store information, data, and / or instructions that may be used by the processing circuitry. The memory may store any suitable instructions, data, or information, including a computer program, software, an application including one or more of logic, rules, code, tables, and / or other instructions capable of being executed by the processing circuitry and utilized by the network node. The memory may be used to store any calculations made by the processing circuitry and / or any data received via the communication interface. In some embodiments, the processing circuitry and memory is integrated.

[0100] Determining, calculating, obtaining or similar operations described herein may be performed by processing circuitry, which may process information by, for example, converting the obtained information into other information, comparing the obtained information or converted information to information stored in the network node, and / or performing one or more operations based on the obtained information or converted information, and as a result of said processing making a determination. Moreover, while components are depicted as single boxes located within a larger box, or nested within multiple boxes, in practice, computing devices may comprise multiple different physical components that make up a single illustrated component, and functionality may be partitioned between separate components. For example, a communication interface may be configured to include any of the components described herein, and / or the functionality of the components may be partitioned between the processing circuitry and the communication interface. In another example, non-computationally intensive functions of any of such components may be implemented in software or firmware and computationally intensive functions may be implemented in hardware.

[0101] In certain embodiments, some or all of the functionality described herein may be provided by processing circuitry executing instructions stored on in memory, which in certain embodiments may be a computer program product in the form of a non-transitory computer- readable storage medium. In alternative embodiments, some or all of the functionalities may be provided by the processing circuitry without executing instructions stored on a separate or discrete device-readable storage medium, such as in a hard-wired manner. In any of those particular embodiments, whether executing instructions stored on a non-transitory computer- readable storage medium or not, the processing circuitry can be configured to perform the described functionality. The benefits provided by such functionality are not limited to the processing circuitry alone or to other components of the computing device, but are enjoyed by the computing device as a whole, and / or by end users and a wireless network generally.

Claims

CLAIMS1. An image processing system (100), comprising: an image sensor (102), a signal transport system (104), a first signal processing system (106); the image sensor (102) configured to: obtain (1002) data related to an image, provide (1004) the data to the signal transport system; the signal transport system (104) configured to: receive (1006) the data provided by the image sensor, provide (1008) the data to the first signal processing system; the first signal processing system (106) configured to: provide (1010), to a second signal processing system (124), a second indication (2) related to the data, wherein the second indication is determined based on a first indication (4) based on the data exceeding a detection threshold.

2. The image processing system (100) of claim 1, wherein the first signal processing system (106), and / or the second signal processing system (124), is configured to carry out an algorithm.

3. The image processing system (100) of any one of claims 1 to 2, wherein the second indication (2) initiates a triggering of the second signal processing system (124), and / or the second indication (2) comprises a trigger initiating an activation of the second signal processing system.

4. The image processing system (100) of any one of claims 1 to 3, wherein the signal transport system (104) comprises one or more selectors (108) configured to: provide (1012) the data to one or more analog-to-digital converters, ADCs (110); provide (1014) the data to one or more analog-to-digital converters, ADCs (110) based on a selector indication from the first signal processing system (106); provide (1016) the data to the second signal processing system (124); and / or provide (1018) the data to the second signal processing system (124) based on a selector indication from the first signal processing system (106).

5. The image processing system (100) of any one of claims 1 to 4, wherein the first signal processing system (106) comprises at least one of: a first analog signal processing system, a first analog-digital hybrid signal processing system, and a first digital signal processing system; and / or wherein the second signal processing system (108) comprises at least one of: a second analog signal processing system, a second analog-digital hybrid signal processing system, and a second digital signal processing system.

6. The image processing system (100) of any one of claims 1 to 5, wherein the signal transport system (104) comprises one or more ADCs (110) configured to convert (1020) the data to digital data.

7. The image processing system (100) of claim 6, wherein the one or more ADCs (110) are configured to provide (1022) an indication related to the digital data to the second signal processing system (124).

8. The image processing system (100) of any one of claims 1 to 7, wherein the signal transport system (104) comprises one or more regulators (112) configured to regulate (1024) a signal carrying the data.

9. The image processing system (100) of any one of claims 1 to 8, wherein the signal transport system (104) comprises one or more memory arrays (114) configured to: buffer (1026) the data; and in response to the data being buffered, and / or in response to a buffer indication, provide (1028) the buffered data to the first signal processing system (106).

10. The image processing system (100) of any one of claims 1 to 9, wherein the image sensor (102) comprises at least one of: one or more pixels; one or more global shutter pixels; and, one or more rolling shutter pixels.

11. The image processing system (100) of claim 10, wherein the data related to an image is obtained using at least one of: a subset of one or more pixels; a subset of one or more global shutter pixels; and, a subset of one or more rolling shutter pixels.

12. The image processing system (100) of any one of claims 1 to 11, wherein the first signal processing system (106) comprises a set of crossbar arrays (116) comprising at least one crossbar array (118).

13. The image processing system (100) of claim 12, wherein a subset of the set of crossbar arrays (116) is adapted to carry-out an algorithm.

14. The image processing system (100) of any one of claims 12 to 13, wherein at least one crossbar array (118) comprises a set of memristors (120) comprising at least one memristor (122).

15. The image processing system (100) of claim 14, wherein a memristor is a titanium dioxide memristor, a silicon dioxide memristor, a polymeric memristor, and / or a ferroelectric memristor.

16. The image processing system (100) of any one of claims 14 to 15, wherein a subset of the set of memristors (120) is adapted to carry-out an algorithm.

17. The image processing system (100) of any one of claims 2, 13, 16, wherein the algorithm comprises at least one of: a machine learning algorithm; a pattern detection algorithm; a neural network algorithm; a face detection algorithm; and a gesture detection algorithm.

18. The image processing system (100) of any one of claims 1 to 17, wherein the first signal processing system (106) is configured to obtain (1030) a feedback indication from the second signal processing system (112).

19. The image processing system (100) of any one of claims 1 to 18, wherein the first signal processing system (106) is configured to adapt (1032) a setup of the first signal processing system (106) based on a feedback indication from the second signal processing system (112).

20. The image processing system (100) of claim 19, wherein to adapt (1032) a setup of the first signal processing system (106), comprises to adapt one or more parameters of the first signal processing system (106).

21. The image processing system (100) of any one of claims 19 to 20, comprising a set of crossbar arrays (116) according to any one of claims 12 to 16, wherein:to adapt (1032) a setup of the first signal processing system (106), comprises to adapt one or more parameters of a subset of the set of crossbar arrays.

22. The image processing system (100) of any one of claims 20 or 21, wherein at least one of the one or more parameters is at least one of: a memristance; a resistance; a capacitance; and, an inductance.

23. A method (1000) performed by an image processing system (100), the image processing system comprising: an image sensor (102), a signal transport system (104), a first signal processing system (106); the method comprising: the image sensor (102) obtaining (1002) data related to an image; the image sensor (102) providing (1004) the data to the signal transport system; the signal transport system (104) receiving (1006) the data provided by the image sensor; the signal transport system (104) providing (1008) the data to the first signal processing system; and the first signal processing system (106) providing (1010), to a second signal processing system (124), a second indication (2) related to the data, wherein the second indication is determined based on a first indication (4) based on the data exceeding a detection threshold.

24. The method (1000) of claim 23, wherein the first signal processing system (106), and / or the second signal processing system (124), carries out an algorithm.

25. The method (1000) of any one of claims 23 to 24, wherein the second indication (2) initiates a triggering of the second signal processing system (124), and / or the second indication (2) comprises a trigger initiating an activation of the second signal processing system.

26. The method (1000) of any one of claims 23 to 25, wherein the signal transport system (104) comprises one or more selectors (108), the one or more selectors (108): providing (1012) the data to one or more analog-to-digital converters, ADCs (110); providing (1014) the data to one or more analog-to-digital converters, ADCs (110) based on a selector indication from the first signal processing system (106); providing (1016) the data to the second signal processing system (124); and / orproviding (1018) the data to the second signal processing system (124) based on a selector indication from the first signal processing system (106).

27. The method (1000) of any one of claims 23 to 26, wherein the first signal processing system (106) comprises at least one of: a first analog signal processing system, a first analog-digital hybrid signal processing system, and a first digital signal processing system; and / or wherein the second signal processing system (108) comprises at least one of: a second analog signal processing system, a second analog-digital hybrid signal processing system, and a second digital signal processing system.

28. The method (1000) of any one of claims 23 to 27, wherein the image processing system (100) comprises one or more ADCs (110), the one or more ADCs converting (1020) the data to digital data.

29. The method (1000) of claim 28, comprising the one or more ADCs (110) providing (1022) an indication related to the digital data to the second signal processing system (124).

30. The method (1000) of any one of claims 23 to 29, wherein the signal transport system (104) comprises one or more regulators (112), the one or more regulators regulating (1024) a signal carrying the data.

31. The method (1000) of any one of claims 23 to 30, wherein the signal transport system (104) comprises one or more memory arrays (114), the one or more memory arrays: buffering (1026) the data; and, in response to the data being buffered, and / or in response to a buffer indication, providing (1028) the buffered data to the first signal processing system (106).

32. The method (1000) of any one of claims 23 to 31, wherein the image sensor (102) comprises at least one of: one or more pixels; one or more global shutter pixels; and, one or more rolling shutter pixels.

33. The method (1000) of claim 32, wherein the data related to an image is obtained using at least one of: a subset of one or more pixels; a subset of one or more global shutter pixels; and, a subset of one or more rolling shutter pixels.

34. The method (1000) of any one of claims 23 to 33, wherein the first signal processing system (106) comprises a set of crossbar arrays (116) comprising at least one crossbar array (H8).

35. The method (1000) of claim 34, wherein a subset of the set of crossbar arrays (116) is adapted to carry-out an algorithm.

36. The method (1000) of any one of claims 34 to 35, wherein at least one crossbar array (118) comprises a set of memristors (120) comprising at least one memristor (122).

37. The method (1000) of claim 36, wherein a memristor is a titanium dioxide memristor, a silicon dioxide memristor, a polymeric memristor, and / or a ferroelectric memristor.

38. The method (1000) of any one of claims 36 to 37, wherein a subset of the set of memristors (120) is adapted to carry-out an algorithm.

39. The method (1000) of any one of claims 24, 35, 38, wherein the algorithm comprises at least one of: a machine learning algorithm; a pattern detection algorithm; a neural network algorithm; a face detection algorithm; and a gesture detection algorithm.

40. The method (1000) of any one of claims 23 to 39, comprising the first signal processing system (106) obtaining (1030) a feedback indication from the second signal processing system (112).

41. The method (1000) of any one of claims 23 to 40, comprising the first signal processing system (106) adapting (1032) a setup of the first signal processing system (106) based on a feedback indication from the second signal processing system (112).

42. The method (1000) of claim 41, wherein adapting (1032) a setup of the first signal processing system (106), comprises adapting one or more parameters of the first signal processing system (106).

43. The method (1000) of any one of claims 41 to 42, wherein the image processing system (100) comprises a set of crossbar arrays (116) according to any one of claims 34 to 38, and wherein adapting (1032) a setup of the first signal processing system (106), comprises adapting one or more parameters of a subset of the set of crossbar arrays.

44. The method (1000) of any one of claims 42 to 43, wherein at least one of the one or more parameters is at least one of: a memristance; a resistance; a capacitance; and, an inductance.

45. A tangible, non-transient computer-readable medium comprising instructions that, when executed by an image processing system (100), cause the image processing system to perform operations according to the method of any one of claims 23 to 44.

46. A computer program, comprising instructions that, when executed by an image processing system (100), cause the image processing system to carry out the method according to any of claims 23 to 44.

47. A carrier containing the computer program of claim 46, wherein the carrier is one of an electronic signal, optical signal, radio signal, computer program product, or computer- readable medium.

Images