Systems and methods for low power image processing

Abstract

Methods, systems, and apparatus for reducing power consumption in image processing applications. A computing system can include a system-on-chip (SoC), which in turn can include a central processing unit (CPU), an embedded processor, an image signal processor (ISP), and at least one hardware accelerator. The CPU and the embedded processor can run a first and a second image processing stack. An image processing application running on the CPU or other core in the SoC can call application programming interface (APIs) provided by the embedded processor to carry out image processing operations. This alleviates the need for the CPU and the image processing stack on the CPU to carry out image processing operations.

Description

[0001] Atorney Docket No.: 56113-0818WO1

[0002] SYSTEMS AND METHODS FOR LOW POWER IMAGE PROCESSING TECHNICAL FIELD

[0003] This specification relates to image processing, and more particularly to systems and methods for low power image processing.

[0004] BACKGROUND

[0005] Image processing applications (including programs or applications executing on a device such as a camera application) can utilize image data generated by one or more image sensors on the device. For example, a camera application in a mobile device that has a display screen can display in near real time the image data captured by the one or more image sensor on the device. The camera application can provide the user various options such as image capture, video, and other image-based operations that the user can choose from while viewing the image data on the display screen.

[0006] The image processing applications can rely on the image sensor and programming instructions executing on hardware processors in the device for capturing and processing images for display to the device’s user. For example, the device can include hardware processing cores such as a central processing unit (CPU) and an image signal processor coupled with the image sensor. The central processing unit can execute an operating system and an image processing stack while the image signal processor can process image frames generated by the image sensor.

[0007] When the image processing application is launched by the user, the operating system, which runs on the CPU, can allocate memory to the image processing application and provide the image processing application access to the image processing stack on the CPU. The image processing stack on the CPU can include a set of processing steps that is carried out on the image data. The processing steps can include, for example, hot pixel correction, demosaic correction, noise reduction, shading correction, geometric correction, color correction, tone curve adjustment, edge enhancement, etc. Thus, the image data requested by the image processing application from the CPU would be processed by one or more of the abovementioned processing steps of the CPU image processing stack.

[0008] In some instances, the image data requested by the image processing application may not need to undergo one or more processing steps in the CPU image processing stack. For example, some machine learning-based image processing applications such as facerecognition the image data may not be displayed to the user but consumed by a machineAtorney Docket No.: 56113-0818WO1

[0009] learning module. In such instances, the image data may not need to be subject to the noise reduction step, as the noise reduction step is typically earned out for user presented images. However, because the image processing is carried out by the CPU image processing stack, which includes the noise reduction step, the image data, even though not needed, would undergo noise reduction. In some examples, the CPU’s image processing stack can include additional functions such as image stabilization, motion compensation, etc. . Such additional and unneeded image processing steps can result in inefficient power consumption. In particular, the CPU and the CPU image processing stack can generally consume a considerable portion of the total power consumed by the SoC. Other image processing applications may also be subject to one or more image processing steps in the CPU image processing stack that may not be needed.

[0010] SUMMARY

[0011] This specification describes techniques for reducing power consumption of image processing applications. A system-on-chip (“SoC”) can include an embedded processor that runs a dedicated image processing stack. The SoC can further include a CPU, an image signal processor (“ISP”), and at least one hardware accelerator such as a digital signal processor (“DSP”), a graphics processing unit (“GPU”), or a tensor processing unit (“TPU”). The embedded processor can receive requests from image processing applications to run image processing tasks on the dedicated image processing stack. This alleviates the need for the image processing application to request image processing tasks to be executed on the CPU image processing stack. The embedded processor dedicated image processing stack can be configured such that power consumed in executing image processing tasks on the dedicated image processing stack is less than the power consumed in executing the same image processing tasks on the CPU and the CPU image processing stack.

[0012] The embedded processor can synchronize operations of the ISP, the at least one hardware accelerator, and an on-device image sensor based on the request for performing the image processing operations. For example, the embedded processor can create a context for the image processing operation and assign image processing tasks to one or more hardware accelerators. The hardware accelerators selected can be based on the type of image processing operation. For example, the embedded processor may offload machine learning tasks to the TPU or offload image processing tasks to the GPU. In addition, the embedded processor and the dedicated image processing stack can configured the image sensor and the ISP. For example, the embedded processor can utilize the dedicated image processing stackAtorney Docket No.: 56113-0818WO1

[0013] to configure one or more parameters of the image sensor and the ISP, the parameters including, for example, exposure, aperture, white balance, etc.

[0014] The assignment of the tasks and the configuration of the image sensor and the ISP can be carried out without any involvement from the CPU. As a result, power consumption due to the CPU and the CPU image processing stack can be reduced.

[0015] The embedded processor can provide an application programming interface (API) for carrying out various operations on the SoC. The API can include image capture and processing commands, operation sequence commands, and process management commands. The API allows programs running on the CPU or on other processors to request the embedded processor to carry out image processing operations without the involvement of the CPU or the CPU image processing stack.

[0016] Particular examples of the subject mater described in this specification can be implemented so as to realize one or more of the following advantages. Providing an embedded processor with a dedicated image processing stack can allow processes to request image processing operations to be performed on the embedded processor instead of having to request the operations to be performed on the CPU or the CPU image processing stack. As the power consumed in carrying out the image processing operations on the embedded processor and the embedded processor’s dedicated image processing stack can be less than the power consumed in carrying out the same image processing operations on the CPU and the CPU image processing stack, performing image processing operations on the embedded processor can reduce power consumption. While in some examples, one or more operations in the CPU’s image processing stack can be disabled, the complexity7of the main CPU itself contributes to the high power consumption. The CPU may be composed of multiple CPU cores intended to perform general application processing and can have a complex instruction set architecture, processing pipelines, and memory architecture, all of which result in high power consumption for even the simplest processes. Embedded processors, on the other hand, can be relatively simpler, have a smaller instruction set architecture, processing pipelines and memory architecture, resulting in lower power consumption than the CPU.

[0017] By providing an API for image processing operations, the embedded processor can provide processes running on the CPU or other processors to request image processing operations with selectable image processing pipelines. The processes can provide a graph description specifying a sequence of operations. The embedded processor, based on the graph description, can synchronize execution of the image processing operations on one or more hardware accelerators on the SoC.Atorney Docket No.: 56113-0818WO1

[0018] The embedded processor and the dedicated image processing stack can facilitate timesensitive logic (e.g., where complex image sensor and ISP synchronization is needed) which can be difficult to implement on a CPU. This is because response times on the CPU can often be non-deterministic due to the use of operating system schedulers. On the other hand, the embedded processor can employ real-time scheduling and support execution of timesensitive logic.

[0019] One general aspect includes a method performed using an embedded processor implemented on a system-on-chip (SoC). The method also includes receiving, from the first processing core and by the embedded processor, a request to perform a set of image processing operations, where the request specifies processing parameters to be used in performing the set of imaging processing operations; and synchronizing operations of the ISP, the at least one hardware accelerator, and an on-device image sensor based on the request and without any involvement of the CPU and an image processing stack of a CPU implemented on the soc, may include: configuring, by the embedded processor and based on the request, the on-device image sensor to initiate the imaging operation. The method also includes configuring, by the embedded processor and using a dedicated image processing software stack resident on the embedded processor, the ISP to process streamed image data received from the image sensor using the processing parameters specified in the request, where configuring the ISP and the image sensor is performed without any involvement by the image processing stack on the CPU. and offloading, by the embedded processor, one or more image processing operations of the set of image processing operations to the at least one hardware accelerator. The method also includes providing, by the embedded processor and to the first processing core, a processed image upon competition of operations by the ISP and the at least one hardware accelerator.

[0020] Implementations may include one or more of the following features. The method where the first processing core includes one of a central processing unit, a security agent (SA) or an always-on sub-system (AOSS). The embedded processor is accessible by the first processing core using a dedicated application programming interface (API). The hardware accelerator can include at least one of a digital signal processor, a graphics processing unit, a tensor processing unit, or another processor core implemented on the soc. The set of image processing operations include an image capture operation and an image data processing operation, where the embedded processor schedules iterative execution of the image capture and the image data processing operations. The synchronization operations may include: generating, by the embedded processor, based on the request a context for execution of theAtorney Docket No.: 56113-0818WO1

[0021] set of operations. The method may include: powering on, by the embedded processor, at least one of the ISP. the image sensor and the at least one hardware accelerator prior to configuring the image senor to initiate imaging operations. The method may include: powering off, by the embedded processor, at least one of the ISP, the image sensor and the at least one hardware accelerator after competition of operations by the ISP and the at least one hardware accelerator. The request to perform the set of image processing operations includes a graph description specifying a sequence of operations, the method may include: synchronize execution of the image processing operation on the one or more cores based on the graph description. The first image processing stack includes an image noise reduction functionality' and the second image processing stack lacks the image noise reduction functionality. The second image processing stack includes an image sensor driver configured to receive image data from the image signal processor and generate configuration parameters for the image sensor and the image signal processor based on the image data.

[0022] One general aspect includes one or more non-transitoiy computer-readable storage media storing instructions that when executed by one or more processors cause the one or more processors to perform operations performed using an embedded processor implemented on a system-on-chip (SoC). The one or more non-transitory computer-readable storage media storing instructions also includes receiving, from the first processing core and by the embedded processor, a request to perform a set of image processing operations, where the request specifies processing parameters to be used in performing the set of imaging processing operations; and synchronizing operations of the ISP, the at least one hardware accelerator, and an on-device image sensor based on the request and without any involvement of the CPU and an image processing stack of a CPU implemented on the SoC, may include: configuring, by the embedded processor and based on the request, the on-device image sensor to initiate the imaging operation. The instructions also includes configuring, by the embedded processor and using a dedicated image processing software stack resident on the embedded processor, the ISP to process streamed image data received from the image sensor using the processing parameters specified in the request, where configuring the ISP and the image sensor is performed without any involvement by the image processing stack on the CPU. and offloading, by the embedded processor, one or more image processing operations of the set of image processing operations to the at least one hardware accelerator. The instructions also includes providing, by the embedded processor and to the first processing core, a processed image upon competition of operations by the ISP and the at least one hardware accelerator.Atorney Docket No.: 56113-0818WO1

[0023] Implementations may include one or more of the following features. The computer-readable storage media where the first processing core includes one of a central processing unit, a SA or an AOSS The embedded processor is accessible by the first processing core using a dedicated application programming interface (API). The hardware accelerator can include at least one of a digital signal processor, a graphics processing unit, a tensor processing unit, or another processor core implemented on the soc. The set of image processing operations include an image capture operation and an image data processing operation, where the embedded processor schedules iterative execution of the image capture and the image data processing operations. The synchronization operations may include: generating, by the embedded processor, based on the request a context for execution of the set of operations. The operations may include: powering on, by the embedded processor, at least one of the ISP, the image sensor and the at least one hardware accelerator prior to configuring the image senor to initiate imaging operations. The operations may include: powering off, by the embedded processor, at least one of the ISP, the image sensor and the at least one hardware accelerator after competition of operations by the ISP and the at least one hardware accelerator. The request to perform the set of image processing operations includes a graph description specifying a sequence of operations, the operations may include: synchronize execution of the image processing operation on the one or more cores based on the graph description. The first image processing stack includes an image noise reduction functionality and the second image processing stack lacks the image noise reduction functionality. The second image processing stack includes an image sensor driver configured to receive image data from the image signal processor and generate configuration parameters for the image sensor and the image signal processor based on the image data.

[0024] One general aspect includes a system. The system includes one or more processors; and one or more storage devices storing instructions that when executed by the one or more processors to perform operations performed using an embedded processor implemented on a system-on-chip (SoC), where the soc may include the embedded processor, a first processing core, an image signal processor (ISP), and at least one hardware accelerator, the operations may include receiving, from the first processing core and by the embedded processor, a request to perform a set of image processing operations, where the request specifies processing parameters to be used in performing the set of imaging processing operations; and synchronizing operations of the ISP. the at least one hardware accelerator, and an on-device image sensor based on the request and without any involvement of the CPU and an image processing stack of a CPU implemented on the soc, may include: configuring, by theAtorney Docket No.: 56113-0818WO1

[0025] embedded processor and based on the request, the on-device image sensor to initiate the imaging operation. The operations also include configuring, by the embedded processor and using a dedicated image processing software stack resident on the embedded processor, the ISP to process streamed image data received from the image sensor using the processing parameters specified in the request, where configuring the ISP and the image sensor is performed without any involvement by the image processing stack on the CPU, and offloading, by the embedded processor, one or more image processing operations of the set of image processing operations to the at least one hardware accelerator. The operations also include providing, by the embedded processor and to the first processing core, a processed image upon competition of operations by the ISP and the at least one hardware accelerator.

[0026] Implementations may include one or more of the following features. The system where the first processing core includes one of a central processing unit, a SA or an AOSS. The embedded processor is accessible by the first processing core using a dedicated application programming interface (API). The hardware accelerator can include at least one of a digital signal processor, a graphics processing unit, a tensor processing unit, or another processor core implemented on the soc. The set of image processing operations include an image capture operation and an image data processing operation, where the embedded processor schedules iterative execution of the image capture and the image data processing operations. The synchronization operations may include: generating, by the embedded processor, based on the request a context for execution of the set of operations. The operations may include: powering on, by the embedded processor, at least one of the ISP, the image sensor and the at least one hardware accelerator prior to configuring the image senor to initiate imaging operations. The operations may include: powering off, by the embedded processor, at least one of the ISP, the image sensor and the at least one hardware accelerator after competition of operations by the ISP and the at least one hardware accelerator. The request to perform the set of image processing operations includes a graph description specifying a sequence of operations, the operations may include: synchronize execution of the image processing operation on the one or more cores based on the graph description. The first image processing stack includes an image noise reduction functionality and the second image processing stack lacks the image noise reduction functionality. The second image processing stack includes an image sensor driver configured to receive image data from the image signal processor and generate configuration parameters for the image sensor and the image signal processor based on the image data.Atorney Docket No.: 56113-0818WO1

[0027] The details of one or more embodiments of the subject mater of this specification are set forth in the accompanying drawings and the description below. Other features, aspects, and advantages of the subject mater will become apparent from the description, the drawings, and the claims.

[0028] BRIEF DESCRIPTION OF THE DRAWINGS

[0029] Figure 1 is a block diagram of an example computing system.

[0030] Figure 2 shows a block diagram of an example arrangement of the system shown in Figure 1 to execute image processing applications.

[0031] Figure 3 shows a block diagram of an example image processing operation facilitated by the embedded processor.

[0032] Figure 4 shows a flow diagram of communication between the embedded processor and other IP devices on the SoC in relation to carrying out an image capture operation.

[0033] Figure 5 shows a block diagram of an example graph description.

[0034] Figure 6 shows a flow diagram of an example process.

[0035] Figure 7A shows an example hardware synchronization unit.

[0036] Figure 7B shows example code for generating a global sync object.

[0037] Like reference numbers and designations in the various drawings indicate like elements.

[0038] DETAILED DESCRIPTION

[0039] Figure 1 is a block diagram of an example computing system 100. The example computing system 100 can include a system-on-chip 102 (“SoC 102"). which, in turn, can include a central processing unit (CPU) 104, an embedded processor 106, a shared memory 108, and an IP / circuit block 110. The SoC 102 can be implemented in an integrated circuit of an example user / client device 130, consumer device, or mobile device, where each of these devices can include items such as a smartphone 130a, a tablet 130b, a laptop 130c, and a smartwatch or wearable device 130d. The client device 130 may also include other items such as an eNotebook, Netbook, smart speaker, or mobile computer. In some examples, the system 100 and the SoC 102 is an integrated circuit of a desktop computer, network server, or related cloud-based asset. The SoC 102 can also be coupled with an image sensor 152. While Figure 1 shows a single image sensor 152, more than one image sensors can be included. In particular, the image sensor 152 can be part of a client device 130.Atorney Docket No.: 56113-0818WO1

[0040] The CPU 104 can be a general -purpose CPU (e.g., a single or multi-core CPU). An operating system 142 can run on the CPU 140. Examples of operating system 142 can include the Android operating system, the iOS operating system, the Windows operating system, etc. The operating system 142 can run one or more image processing applications. For example, image processing applications can include photography applications, video call applications, authentication applications, face recognition applications, etc. The operating system 142 can include one or more hardware dnvers that allow applications running on the operating system 142 to control hardware. For example, the operating system 142 can run a first image processing stack 144, which can carry out image processing operations requested by applications running on the CPU The first image processing stack 144 can include image drivers, image capture and processing libraries and image processing operations. Image sensor drivers can include, for example, “Camera2” that is included in the Android operating system. The Camera2 driver has an application programming interface (API) that allows applications running on the operating system 142 to make calls to functions provided by the driver.

[0041] The CPU 140 can execute the operating system’s image processing stack that provides functionality for image processing. In some examples, the first image processing stack 144 can include operations such as hot pixel correction, demosaic correction, noise reduction, shading correction, geometric correction, color correction, tone curve adjustment, edge enhancement, etc. In some examples, the execution of the image processing stack by the CPU carries out the above-mentioned operations on all image data generated by the image sensor. For example, when an image processing application requests image data, the image processing stack may be leveraged to process the image data generated by the image sensor 152 to carry out the above-mentioned operations to generate processed image data. The CPU 140 can provide the processed image data to the image processing application. In some examples, the first image processing stack 144 can be stored in memory, which can only be accessed by the CPU 140.

[0042] The SoC 102 also can include the embedded processor 106, which can carry out specific functions related to image processing tasks. In particular, the embedded processor 106 can execute a second image processing stack 154 and include a synchronization unit 156. In some examples, the second image processing stack 154 can be similar to the first image processing stack 144 executed by the CPU 104. For example, the second image processing stack 154 also can include functionality such as hot pixel correction, demosaic correction, noise reduction, shading correction, geometric correction, color correction, tone curveAtorney Docket No.: 56113-0818WO1

[0043] adjustment, edge enhancement, etc. In some other examples, the second image processing stack 154 may include only a subset of functionality of the first image processing stack 144. For example, the second image processing stack 154 may not include functionality such as noise reduction, shading correction, phase detection auto-focus, image stabilization, motion compensation, high-dynamic-range processing, and edge enhancement . The second image processing stack 154 can also be leveraged by the embedded processor 106 to configure the image sensor 152 and the ISP 112. For example, the second image processing stack 154 can include functional blocks that can be used to generate statistics from the image data generated by the ISP 112 and include algorithms (e.g., algorithms for adjusting aperture, autoexposure, and white balance) that use the statistics to generate configuration parameters for the image sensor 152 and the ISP 112. The second image processing stack 154 can be stored in memory, which can only be accessed by the embedded processor 106.

[0044] The synchronization unit 156 can synchronize communications between various elements of the SoC 102. For example, the synchronization unit 156 can synchronize communications between the embedded processor and processing units in the circuit block 110 some of which could include hardware accelerators. The synchronization unit 156 can dynamically control and manage one or more synchronization events that are executed on the SoC 102 in support of image processing operations. Specifically, the synchronization unit 156 can generate control signaling and use one or more discrete signal values of the control signaling to arrange and synchronize operations between two or more processing units that are included in the circuit block 110, the CPU 104, and the embedded processor 106. In some examples, the synchronization unit’s operation can be implemented in the manner discussed herein in relation to Figures 7A and 7B.

[0045] The memory 108 is a system memory, shared memory, or both. In the example of Figure 1, memory 108 is depicted external to circuit block 110. However, memory’ 108 can include portions of memory that are: i) specific to circuit block 110, ii) external to circuit block 110, or iii) both. The memory 106 can be random access memory' of the SoC 102, such as static random-access memory (SRAM), dynamic random access memory (DRAM), a synchronous DRAM (SDRAM), or double data rate (DDR) SDRAM. In some implementations, aspects of memory' 108 are configured as a shared scratchpad memory that supports parallel access of its memory' resources by two or more processors of the circuit block 110. The memory 108 can also include various other types of memory, such as high bandwidth memory' (HBM), narrow memory (e.g., for storing 8-bit values), wide memory (e.g., for storing 16-bit or 32-bit values), etc.Atorney Docket No.: 56113-0818WO1

[0046] The circuit block 110 can include processing units, such as an image signal processor (ISP) 112, a tensor processing unit (TPU) 114, a digital signal processor (DSP) 116, and a graphics processing unit (GPU) 118. The circuit block 110 can also be referred to as an IP block 110, where the IP block can include one or more dedicated or proprietary hardware elements. For example, each of the ISP 112, TPU 114, DSP 116, and GPU 118 can be a respective proprietary IP block (or IP device) of a particular entity or device manufacturer.

[0047] The ISP 112 can process the image data or image frame received from the image sensor 152 to generate an image frame. The ISP 112 can carry out one or more processing tasks such as de-mosaic processing, noise reduction, filtering, color correction, etc. For example, the ISP 112 can carry' out focus, exposure, and white balance correction on the image data provided by the image sensor 152. In some other examples, the ISP 112 can be configured to merely provide data to an external driver (such as the drivers included in the first image processing stack 144 and the second image processing stack 154) that can use algorithms to determine the parameters for focus, exposure and white balance correction, and provide values of those parameters to the ISP 112. In some instances, the ISP 112 can provide data such as histograms (which is a distribution of pixel intensities across the image), noise statistics, sharpness metrics, color statistics, edge detection metrics, etc. The ISP 112 can also generate the image frame in one or more image formats. For example, the ISP 112 can generate image frames in RGB, YUV. or RAW formats.

[0048] The image sensor 152 can include a digital charged-coupled-device (CCD) type or complementary' metal-oxide-semiconductor (CMOS) type image sensor. The image sensor can be an m x n sensor matrix that generates image data associated with the m x n sensors. For example, the image sensor can include 100s of thousands to 100s of millions of such sensors. The output of the image sensor can be an image frame or image data that can be converted (by, e.g., an image signal processor) into an image frame. In some examples, the image sensor 152 can include a front-end module that includes lenses, and mechanical and / or electronic shuters, and aperture control that control the amount of light that is incident on the sensors. In some examples, the focal length of the lenses, the shuter speed, and / or the aperture size (e.g., the f-stop) can be adjustable. The image sensor 152 can also provide adjustments to the sensors, such as the gain of the sensors. Adjusting the gain of the sensors can change, for example, the intensity7of the images generated by the image sensor 152. In some instances, the adjustment of these parameters can be carried out by components external to the image sensor 152. For example, an image sensor driver can provide values of each of these parameters of the image sensor 152. In some instances, the image sensor 152 canAtorney Docket No.: 56113-0818WO1

[0049] access a register file to fetch the values of the parameters, where the register file can be writen into with values by the external components. In some instances, the register file can be local to the image sensor 152. In some other instances, the register file can be located in the memory 108 of the SoC 102.

[0050] The example computing system 100 can run one or more image processing applications. For example, the image processing application can be a photo application, a video call application, a video-conference application, etc. The image processing application can be run by the CPU 104. Image processing applications can also be run on other processors such as the secure agent (SA) 204 or the always-on sub-system (AOSS) 202. The SA 204 can be used for authentication and can support authentication applications such as face recognition applications. Once a user requests (e.g., via the front end interface provided on the device display) to launch the image processing application, the example computing system 100 can execute a series of actions to launch the image processing application. In particular, the image processing application can use APIs to execute image processing operations on the embedded processor 106, which in turn can schedule the image processing operations on one or more hardware accelerators in the circuit block 110.

[0051] Figure 2 shows a block diagram of an example arrangement of the system 100 shown in Figure 1 to execute image processing applications. In particular, Figure 2 shows the embedded processor 106 in communication with the image sensor 152, the ISP 112, the computer vision module (CVM) 206, a nanoTPU 208, the DSP 116, and the TPU 114. The embedded processor 106 also can receive requests from the CPU 102, a SA 204 and an AOSS 202. The second image processing stack 154 can include an image capture and processing library 210. an algorithm module 214, and drivers 212. In some examples, the library 210, the algorithm module 214, and the drivers 212 may be stored in a dedicated space in memory where the dedicated space may be accessed only by the embedded processor. The image capture and processing library 210 can include one or more image processing functions. The algorithm module 214 can include algorithms for adjusting focus, exposure and white balance of the sensor 152. In some instances, the algorithm module can include the “3 A” algorithms that are included in the Android operating system. In some instances, the algorithm module 214 can include a subset of the functionality of the 3A algorithm. For example, the algorithm module 214 may include one or two of the focus, exposure, and white balance functionality. The drivers 212 can include hardware drivers for the image sensor 152, the ISP 112, the CVM 206. and the TPU 114. The drivers allow software to communicate with and / or operate the respective hardware.Atorney Docket No.: 56113-0818WO1

[0052] The embedded processor 106 can also be coupled with other systems and processors such as a SA 204 and an AOSS 202. Image processing applications running on any one of the CPU 104, the SA 204 or the AOSS 202 can use an API of the embedded processor 106 to request execution of image processing operations by the embedded processor 106. The embedded processor 106, upon receiving the requests, can configure the image sensor 152 and the ISP 112, and offload any operations to one or more of the hardware accelerators such as the nanoTPU 208. the DSP 116, and the TPU 114. In particular, the embedded processor 106 can utilize the appropriate drivers 212 to offload operations to the hardware accelerators. One example of an image processing operation request is discussed below in relation to Figure 3.

[0053] Figure 3 shows a block diagram of an example image processing operation facilitated by the embedded processor 106. The embedded processor 106 can receive API requests for image processing operations from an image processing application 302 that can be running on either the CPU 104 or the SA 204. In this particular example, the image processing application 302 is a face recognition application, where the image processing operation can include capturing image data, processing the image frames, and using an inference engine to carry out face recognition. The embedded processor 106 upon receiving the request, can configure the image sensor 152 and the ISP 112. The configuration process can include using the second image processing stack 154 to initialize the image sensor 152 and the ISP 112 such that image frames are generated and processed. The embedded processor 106 can receive statistics generated by the ISP 112 and utilize the drivers 212 and the algorithm module 214 to determine values for configuration parameters of the image sensor 154 and the ISP 112. After configuring the image sensor 152 and the ISP 112, the embedded processor 106 can provide the produced image frames to the CVM 206 for additional image processing operations including cropping, warping, and downscaling. The embedded processor 106 can then provide the image frame processed by the CVM 206 to the TPU 114, which can run a machine learning inference engine for face recognition. The embedded processor 106, upon receiving the result of the inference from the TPU 114, can provide the result to the image processing application 302.

[0054] The embedded processor 106, and in particular the synchronization unit 156, in response to receiving API calls for image processing operation requests, can maintain a global state for synchronization activity spanning two or more devices (e g., the embedded processor 106, the TPU 114. the DSP 116, etc.) of the SoC 102. In one approach, the synchronization unit 156 can maintain multiple global sync objects states and determineAtorney Docket No.: 56113-0818WO1

[0055] context states with reference to the global sync objects to control and manage synchronization events at the SoC 102. Context states can correspond to a respective device on the SoC 102. For example, the synchronization unit 156 can determine a respective context state for the ISP 112, a respective context state for the TPU 114, a respective context state for the DSP 116, and a respective context state for the GPU 118. The synchronization unit 156 can use contexts and context states to allow concurrency of access between multiple IP devices and to maintain synchronization integrity with respect to those concurrent accesses. For each IP device, the context provides an abstraction where the IP device works within its own synchronization context. In general, each context can be tied to a set of registers maintained in the memory 108. where the register values correspond to the context states. One example of the communication between the embedded processor 106 and other devices on the SoC 102 is discussed below in relation to Figure 4. Additional details of an example of the configuration of the synchronization unit 156 is discussed in relation to Figures 7A and 7B.

[0056] Figure 4 shows a flow diagram 400 of communication between the embedded processor 106 and other IP devices on the SoC in relation to earn ing out an image capture operation. The flow diagram shows communications between the image processing application 302, the embedded processor 106, the image sensor 152, the ISP 112, and the CVM 206. The image processing application 302 can be executed by any one of the CPU 104, the SA 204. or the AOSS 202. In this particular example, the image processing application 302 requests an image capture operation. That is the image processing application requests the embedded processor 106 to capture image frames based on some parameters and provide the image frames to the image processing application 302. Of course, the image processing application 302 can make requests for other image processing operations, such as face recognition, in which case the flow diagram may also include communication with the TPU 114 (as described with reference to Figure 3).

[0057] The image processing application 302 sends an API call “CreateContext(params)” to the embedded processor 106. In response, the embedded processor can create a context for the device on which the image processing application 302 is running. The parameters can include, for example, “perf_power_requirement” can indicate the amount of power consumption and “priority ” can indicate the priority' in a multi-context conflict. These are only example parameters and are not exhaustive. Other parameters that address other aspects of creating context can also be included. . The embedded processor 106 can power on the devices that may be needed to carry out the image processing operation. For example, theAtorney Docket No.: 56113-0818WO1

[0058] embedded processor 106 can power on the image sensor 152, the ISP 112, and the CVM 206. In some examples, the embedded processor 106 also can configure the image sensor 152 and the ISP 112. As discussed above, embedded processor 106 can utilize the second image processing stack 154 to configure the image sensor 152 and the ISP 112, e.g., to configure various configuration parameters such as focus, aperture and white balance. In this instance, the second image processing stack 154 can specify the values for the configuration parameters of the devices. Once the devices are powered on. the embedded processor 106 can respond to the image processing application 302 with an acknowledgement “ctx” indicating that a context was successfully created.

[0059] The image processing application 302 can then send an image capture API call ‘"StreamOnfyaramsfy to the embedded processor 106. The StreamOn(params) API call can indicate to the embedded processor 106 that a request for controlling the image sensor 152 to generate a stream of image frames. To that end, the API call can include values of parameters such as “minimum_fps” indicating the minimum frames-per-second or frame rate of the capture of image frames, '’sensor outpul resolulion" indicating the output resolution of the image frames (e.g., m pixels by n pixels),£'image_output_formaf ’ indicating the color format of the image frame, etc. The embedded processor 106 can configure the control and status register (“CSRs”) associated w ith each of the IP cores. For example, the parameters received can include parameters for one or more of the image sensor 152, the ISP 112, the CVM 206. the DSP 116. the TPU 114, or other processing cores. Each of these processing cores can include their own CSRs. The embedded processor 106 can propagate the configuration parameters to the appropriate processing core to configure the respective CSRs. For example, the embedded processor 106 can configure the CSR values for the image sensor 152 and the ISP 112 based on the values of the parameters received in the API call. In another example, the size of the image sensor 152 may be different from the desired image size (included in the parameters), in such instances, the embedded processor 106 can generate values for the crop and scalar function of the ISP 112 such that the desired image size is produced. The embedded processor 106 can then instruct the image sensor 152 to begin generating image frames by sending a “StramOnO” command to the image sensor 152. The image sensor 152 can begin generating image frames, which the second image processing stack 154 can store in an image buffer.

[0060] The image processing application 302 can then send a command to capture image frames to the embedded processor 106. For example, the image processing application 302Atorney Docket No.: 56113-0818WO1

[0061] can send the Capturelmage(params) API call, which can be a request for the embedded processor 106 to capture images and provide the images to the image processing application 302. Responsive to receiving the Capturelmage(params), the embedded processor 302 can extract the values of the parameters included in the API call. In some examples, the parameters can include values of configuration parameters of one or more of the image sensor 152, the ISP 112 and the CVM 206. For example, the parameters in the API call can include values for configuring the focus, white balance, or aperture of the image sensor 152. The embedded processor 106 can generate values of CSRs from the values of parameters received in the API call. The embedded processor 106 can use these calculated values to update the values of CSRs for the image sensor 152 and the ISP 112 (in some examples, the CSRs can be updated in the manner discussed below in relation to Figures 7A and 7B). The embedded processor 106 also can configure the output of the image sensor 152 to be provided to the ISP 112. The image data generated by the image sensor 152 can then be streamed to the ISP 112. The ISP 112 can receive the image data and generate an image frame, and carry out image processing operations. Once the ISP 112 generates the image frame, the ISP 112 can indicate to the embedded processor 106 that the image frame generation is complete (by way of an “image done” message, for example).

[0062] The embedded processor 106, upon receiving an indication from the ISP 112 that the image frame has been generated, the embedded processor 106 may instruct the CVM to scale and / or crop the image frame. For example, the embedded processor 106 can send a command “ScaleAndCrop(image)” to the CVM 206. The “ScaleAndCrop” command can instruct the CVM 206 that the indicated image frame needs to be scaled and cropped. In some examples, the command can also include parameters that specify the scale and the dimensions and orientation of the cropping operation. In some instances, values of these parameters may have been received from the image processing application 302 along with the “Capturelmage(params)” API call.

[0063] Once the image frame has been scaled and cropped, the CVM 206 can indicate to the embedded processor 106 that the scaling and cropping operation has been completed. The CVM 106 can also indicate that the scaled and cropped image frame is stored in an image buffer at a particular location. Upon receiving an indication that the CVM 206 has completed the scaling and cropping operation, the embedded processor 106 can indicate to the image processing application 302 that the image capture operation has been completed. For example, the embedded processor 106 can send a message “image” to the image processing application 302 along with a memory address of where the processed image frame is stored.Atorney Docket No.: 56113-0818WO1

[0064] Upon receiving an indication that the image capture operation has been completed, the image processing application 302 may request for additional operations to be carried out. For example, the image processing application 302 can send additional API calls to the embedded processor 106 associated with the requested operations. If no additional operations are needed, the image processing application 302 can release the context created with the embedded processor 106. For example, the image processing application 302 can send a “ReleaseContext(ctx)” API call to the embedded processor 106. In response, the embedded processor 106 can obtain the value “ctx” of the context from the API call, and release the context that w as previously created. In some instances, the embedded processor 106 may also power off the image sensor 152. the ISP 112, and the CVM 206 by sending a “PowerOff” command.

[0065] It should be noted that the embedded processor 106 carries out the above image processing operations without any involvement of the CPU 104. The only involvement the CPU 104 may perhaps have is if the image processing application 302 were running on the CPU 104. In such instances, the CPU 104 may communicate with the embedded processor 106 to convey the API calls and responses. However, the CPU 104 itself does not configure the image processing devices or exchange data with the image processing devices for the requested operation. Thus, the image processing application 302, w hether being executed on the CPU 104 or some other processor, can utilize the embedded processor 106 and its second image processing stack 154 to carry out image processing operations. As the CPU 104 and the first image processing stack 144 is not involved, the high power consumption associated with the CPU 104 can be avoided.

[0066] Figure 4 shows only a few examples of the API calls that are offered by the embedded processor 106 for image processing applications. In some instances, the API of the embedded processor 106 can additionally include calls such as “SubmitCommand(context_handle, cmd buf addr, size),” and “RegisterGraph(graph_desc, graph handle).” The SubmitCommand can directly send a command buffer to the embedded processor 106 for execution without seting up a command queue at the image processing application 302. The embedded processor 106 can generally create and maintain a command queue for a context. The invocation of the SubmitCommand API call can queue the given command buffer (in the cmd buf addr) to the internal command queue set up by the embedded processor 106. In this manner, the command call can be used to decouple the embedded processor 106 command queue handling from the side of the image processing application 302. The RegisterGraph call can request the embedded processor 106 to registerAtorney Docket No.: 56113-0818WO1

[0067] a graph description. A graph description can describe a sequence of execution of operations of an image processing task or operation. In particular, the graph description can indicate the type of operations to be carried out and the sequence in which the operations are to be carried out. The embedded processor 106 can use the graph description to control and synchronize the operations of each of the processors on the SoC 102. By registering the graph, the image processing application 302 can provide the embedded processor 106 a pipeline of operations to be completed. In some instances, after the operations are complete, the image processing application can send an ‘’UnregisterGraph(graph_handle)” command to the embedded processor 106, which unregisters a previously registered graph description.

[0068] Figure 5 shows a block diagram of an example graph description 500. Specifically, the image processing application 302 can include a graph description 500 that lays out the sequence of operations that are to be performed on the SoC 102. The graph description 500 can include a set of notations that describe the operations. The graph description 500 can be a subgraph description of a larger compute graph description. The subgraph may be run multiple times without explicit interaction with the image processing application 302. Figure 5 also shows a graphical representation 502 of a first subgraph 504. The graphical representation 502 is only an example and includes operations performed by the CVM , the DSP, and the TPU. The "Dep " nodes indicate incoming data from a previous subgraph or outgoing data to the subsequent subgraph. One set of incoming data is provided to the CVM while a second set of incoming data is provided directly to a first TPU (labeled ‘C’ ). The first TPU receives the second set of incoming data and the data processed by the CVM to carry7out operations and generate an output which is provided to an output node and to a second TPU (labeled “D”). The second TPU receives inputs from the first TPU and a DSP, which processes data received from the CVM. The second TPU generates an output that is provided to a second output node. The graphical representation 502 of the subgraph 504 specifies a sequence of operations that are to be carried out by the embedded processor 106. The subgraph 504 also can specify the hardware accelerators that the embedded processor 106 can utilize to offload some operations. For example, the embedded processor 106 can offload the operations of the first TPU and the second TPU to the TPU 114.

[0069] The image processing application can provide a graph based description of the graphical representation 502 of the subgraph 504 to the embedded processor. For example, the image processing application can utilize graph description notations such as G for graph. V for a set of vertices (or processing nodes such as TPU, ISP. etc.), and E for a set of edges connecting between the vertices. The embedded processor 106 can convert the graphAtorney Docket No.: 56113-0818WO1

[0070] description based on the notations into a set of commands that execute the subgraph 504. For example, the embedded processor 106 can include commands such as CVM offload {Dep}, which causes the embedded processor 106 to offload the processing of the data received at the Dep node at the input of the CVM. Once the data at the Dep node is available, the embedded processor 106 can execute this command. Similarly, the embedded processor 106 can use the command TPU offload {CVM output, Dep}, which instructs the TPU to execute data received from the CVM and the Dep node at the input of the TPU. Once the data at the output of the CVM and the data at the Dep node are available, the embedded processor 106 can execute this command. Similarly, the embedded processor 106 can translate the graph into additional commands (e.g., DSP_offload {CVM output}) such that the computation described in the graph can be performed. The embedded processor 106 can also provide additional commands such as Config_CSRs {ID, #of CSRs, payload} (which sets the values of the CSRs for the IP cores such as the CVM, TPU, DSP, etc. to configure these cores), Timer {ID, timeout, oneshot / periodic, CMD} (which executes the command CMD (e.g., CapturelmageO) repeatedly based on the period set by the user), etc., to setup commands for the execution of the subgraph 504.

[0071] In some examples, the image processing application 302 can include graph notations or commands with API calls after the graph description has been registered with the embedded processor 106. For example, the image processing application can register the graph with a command (e.g.. CMD1). The embedded processor can store the translated description of the graph description 502 in association with the command. When the image processing application sends the command, the embedded processor 106 can execute the commands associated with that command to execute the subgraph 504. In some examples, the embedded processor 106 may not use sync objects to track internal graph dependencies.

[0072] Figure 6 shows a flow diagram of an example process 600. The process 600 can be executed, for example, on the embedded processor 106 discussed herein in relation to Figures 1-5. The following description is provided in the context of embedded processor 106 for illustrative purposes. One skilled in the art will appreciate that the operations of process 600 can be performed by any computing device.

[0073] The process 600 can include receiving, from a first processor core of an SoC and by an embedded processor, a request to perform a set of image processing operations, wherein the request specifies processing parameters to be used in performing the image processing operations (602). At least one example of this process stage has been discussed herein in relation to Figure 1 -5. For example, at least one of the CPU 104, the SA 204 or the AOSSAtorney Docket No.: 56113-0818WO1

[0074] 202 can be referred to as the first processor core. An image processing application 302 can run on any of these processing cores and send API calls to the embedded processor for carrying out image processing operations. The requests to perform image processing operations can be sent by the image processing application 302 by way of API calls and / or graph descriptions that specify the type and sequence of operations. The requests can also include parameters to be used in performing the image processing operations. For example, as shown in Figure 4, the Capturelmage(params) API call can include values of parameters that can be used by the embedded processor 106 to set configuration parameter values of the image sensor 152 and the ISP 112.

[0075] The process 600 further includes synchronizing operations of the ISP, at least one hardware accelerator, and an on-device image sensor based on the request and without any involvement of the CPU and an image processing stack of the CPU (604). As discussed herein in relation to Figure 4, the embedded processor 106 can synchronize operations of the ISP 112, the CVM 206 (the at least one hardware accelerator, which may also include the TPU 114), and the image sensor 152 (the on-device image sensor). In some examples, the CPU 104 is not involved in this synchronization operation. That is, even if the image processing application 302 is running on the CPU 104, the CPU 104 does not participate in the configuration and synchronization of operations on the SoC 102. Instead, the synchronization and configuration of operations are carried out by the embedded processor 106.

[0076] The process further includes configuring, by the embedded processor and based on the request, the on-device image sensor to initiate the image processing operation (606). As discussed above in relation to Figures 3 and 4, the embedded processor 106, and in particular the second image processing stack 154, can configure the image sensor 152 (and the ISP 112) to adjust for parameters such as focus, exposure, and white balance. The embedded processor 106 can also control the image sensor 154 to generate a stream of image data.

[0077] The process also includes configuring, by the embedded processor, and using a dedicated image processing software stack corresponding to the embedded processor, the ISP to process streamed image data received from the image sensor using the processing parameters, where configuring the ISP and the image sensor is performed without any involvement of the CPU (608). The second image processing stack 154 (also referred to as “a dedicated image processing software stack"’) resident on the embedded processor 106 can configure the ISP 112 to process streamed image data received from the image sensor 154. The configuration of the ISP 112, just as that of the image sensor 154, can be based on theAtorney Docket No.: 56113-0818WO1

[0078] values of parameters received with the request from the image processing application 302. Again, the configuring of the ISP 112 can be carried out by the embedded processor 106 without the involvement of the CPU 104.

[0079] The process also includes offloading, by the embedded processor, one or more image processing operations of the set of image processing operations to the at least one hardware accelerator (610). As discussed herein, the embedded processor 106 can offload one or more image processing operations to hardware accelerators such as the TPU 114, the DSP 116, the GPU 118, or the CVM 206. As discussed in relation to Figure 3, the embedded processor can offload image operations such as cropping, warping, and downscaling to the CVM 206, and offload face recognition inference operations to the TPU 114.

[0080] The process further includes providing, by the embedded processor and to the first processing core, a processed image upon completion of operations by the ISP and the at least one hardw are accelerator (612). Upon completion of the image processing operations by the hardware accelerators such as the CVM 206 or the TPU 114, the embedded processor 106 can store the image data in an image buffer and provide the image processing application with the memory address where the image data is stored.

[0081] In some examples, the embedded processor is accessible by the first processing core using a dedicated API. As discussed above in relation to Figures 2-5, the embedded processor 106 can provide an API that includes API calls that applications can utilize to instruct the embedded processor 106 to carry out image processing operations. The API calls can include commands, for example, to capture images, create context with the embedded processor 106, image data processing operations (e.g., inference operation by the TPU 114) etc. In some examples, the embedded processor 106 can schedule iterative execution of the image capture and image data processing operations. For example, referring to Figure 4. the embedded processor 106 may carry out the ‘'Capturelmage(params)” operation repeatedly to provide image frames to the image processing application 302. In some instances, the iterative execution can include additional image data processing operations such as filtering or inference operations.

[0082] In some examples, the embedded processor 106. based on the request from the image processing application, can generate a context for execution of the set of operations. For example, referring to Figure 4, the embedded processor 106 can generate a context in response to the request “CreateContext(params)” received from the image processing application 302.Atorney Docket No.: 56113-0818WO1

[0083] In some examples, the embedded processor 106 can power on at least one of the ISP 112, the image sensor 152 and the at least one hardware accelerator (e.g.. the CVM 206 and the TPU 114) prior to configuring the image sensor 154 to initiate imaging operation. In some examples, the embedded processor 106 can power on the image sensor 154 and the ISP 112 before configuring the image sensor 154 and the ISP 112. The embedded processor 106, after configuring the image sensor 154 and the ISP 112. can power on the at least one hardware accelerator.

[0084] In some examples, the embedded processor can power off at least one of the ISP 112, the image sensor 154 and the at least one hardware accelerator after completion of the set of image processing operations. For example, referring to Figure 4, the embedded processor 106 can power off these devices after releasing the context with the image processing application 302.

[0085] In some examples, the embedded processor 106 can receive a request that includes a graph description specifying a sequence of operations, and based on the graph description, synchronizes execution of the image processing operations on one or more cores. For example, referring to Figure 5, the image processing application 302 can register a graph description 500 with the embedded processor 106, where the graph description can specify a sequence of image processing operations. The embedded processor 106, based on the graph description, can synchronize and schedule image processing operations on the at least one hardware accelerator such as the TPU 114, the DSP 118 and the CVM 206.

[0086] In some examples, the second image processing stack 154 of the embedded processor may not include one or more functionality that is present in the first image processing stack 144 of the CPU 104. For example, the second image processing stack 154 may not include noise reduction functionality that is present in the first image processing stack 144. In some examples, one or more additional operations such as edge enhancement, tone curve adjustment, color correction, geometric correction, shading correction, demosaic correction, and hot pixel correction may not be included in the second image processing stack 154 but included in the first image processing stack 144. Reducing the functionality of the second image processing stack 154 can help in reducing the overall footprint of the second image processing application and reduce the power consumed by the embedded processor 106 when executing the second image processing stack 154.

[0087] In some examples, the second image processing stack 154 can include a set of drivers such as an image sensor driver and an ISP driver. The embedded processor 106 can utilize one or more of these drivers to configure the parameters of the image sensor 154 and / or theAtorney Docket No.: 56113-0818WO1

[0088] ISP 112. The configuration parameters can be configured to set the focus, aperture, and white balance of the image sensor 154.

[0089] As discussed above, the embedded processor 106 can present an API to the image processing application 302, where the API can include calls that the image processing application 302 can use to instruct the embedded processor 106 to execute image processing operations. The following describes the architecture that supports the provision of the API and the creation of contexts that aid in synchronizing the image processing operations on the SoC 102.

[0090] Figure 7A shows an example hardware synchronization unit 156 and Figure 7B shows example code for generating a global sync object. The synchronization unit 156 can be used to universally synchronize communications between devices of the SoC 102. such as IP devices of circuit block 110. The hardware synchronization unit 156 includes various resources for implementing universal IP synchronization, including one-to-many and many-to-many synchronization betw een IP devices of the SoC 102.

[0091] The resources of the hardware synchronization unit include control logic 702 and memory for generating, tracking, and managing sync objects or data values corresponding to synchronization events executed at the SoC 102. More specifically, the hardware synchronization unit 156 includes a first memory 704 and a second memory 706. In some implementations, first memory 704 and second memory 706 are the same memory. In some other implementations, first memory 704 and second memory 706 are different memory circuits / units.

[0092] The first memory' 704 can include a combination of SRAM circuits and CSR circuits, whereas the second memory 706 can include one or more SRAM circuits. The first memory 704 can include one or more CSR groupings 705 (“register groupings 705”), where each register grouping 705 includes a group of CSRs (or other registers), such as two or more 8-bit registers, 12-bit registers, 16-bit registers, 32-bit registers, etc. In the example of Figure 2A, the first memory 704 includes three register groupings 705; however, the first memory 704 can include more or fewer register groupings 705. In some implementations, the first memory 704 includes a register grouping 705 for each IP device of the SoC 102.

[0093] In general, for a given heterogeneous compute, the global state of one or more sync objects (or syncs) are well-defined in a structured memory' of SRAM and register resources of the hardware synchronization unit 156. The hardware synchronization unit 156 uses its control logic 702 and memory resources 704, 706 to maintain the global synchronization states of multiple devices / IP devices of the SoC 102.Atorney Docket No.: 56113-0818WO1

[0094] As mentioned above, the synchronization unit 156 can be run on the embedded processor 106 of the SoC 102, but other processing cores may also implement the synchronization unit 156. For example, the control logic 702 can be a processor or logic unit that causes the hardware synchronization unit 156 to maintain multiple global sync objects’ states (e.g., as context states). In the example of Figure 7B, the control logic 702 can generate and / or define a sync object or global sync object using the example code 755. The control logic 702 can also generate control signaling to broadcast synchronization information to consumer IP devices when a producer IP device satisfies a sync.

[0095] In some examples, a sync may be “satisfied” when the IP device associates a sync object with a particular condition. The IP device can make the association with the particular condition based on analysis or determinations made by a driver software of the IP device. The particular condition may be that IP device A (producer) has finished writing data to a buffer of the system. When, or immediately after, that condition occurs, the sync will be signaled by the corresponding IP device A, which in turn satisfies the sync (e.g., signals the sync).

[0096] The synchronization unit 156 is configured to maintain a global state for synchronization activity that spans two or more devices of the SoC 102, such as processors, processor cores, processing units, IP devices, or circuits block 110. The hardware synchronization unit 156 is configured to maintain these global synchronization states without requiring or implementing software layer translations for synchronization signals that are generated and passed between the devices.

[0097] The hardware synchronization unit 156 determines context states with reference to a global sync object that it uses to control and manage synchronization events at the SoC 102. For example, the control logic 702 can detect and / or analyze outgoing and incoming synchronization signals (or values) and determine context states based on those synchronization signals. At least one context state corresponds to a respective device on the SoC 102. In some implementations, each of the context states correspond to a respective IP device among multiple IP devices on the SoC 102.

[0098] For example, the hardware synchronization unit 156 can determine a respective context state for the ISP 112, a respective context state for the TPU 114, a respective context state for the DSP 116, and a respective context state for the GPU 118. In some implementations, the hardware synchronization unit 156 uses contexts and context states to allow concurrency of access between multiple IP devices and to maintain synchronization integrity with respect to those concurrent accesses. For each client or IP device, a contextAtorney Docket No.: 56113-0818WO1

[0099] provides an abstraction where each IP device works within its own synchronization context. In general, each context can be tied to a set of registers that is required to perform operations on a particular sync object or set of sync objects.

[0100] In some implementations, each context state is represented by a combination of values stored across the first memory 704 and the second memory 706. For a given synchronization event, an IP device can first acquire a context allocation from the hardware synchronization unit 156 to create one or more sync objects and then wait on that sync object or signal that sync object to another IP device. In some implementations, a client IP driver of the IP device is used to acquire the context allocation.

[0101] Each context can have a set of CSRs for implementing an application-programming interface (API) of the hardware synchronization unit 156. For example, a read-only CSR is used to acquire a new sync, whereas a write-only CSR is used for releasing a sync. In some implementations, the control logic 702 determines an assignment and / or allocation of registers such that a context representing synchronization events for an IP device has a corresponding set of CSRs for carrying out operations associated with those synchronization events. For clarity, a context can include one or more context states, where each context state can indicate atributes or details of a synchronization event, such as a sync signal event or a wait event.

[0102] The synchronization unit 156 generates register values that correspond to (or represent) each of the context states. For example, each context state for an IP device can be captured using a set of register values that are stored in the first memory 704 as well as one or more bookkeeping data values of the second memory 706. The hardware synchronization unit 156 includes interface connections 720-1. 720-2, which represent an interface control of the hardware synchronization unit 156. The hardware synchronization unit 156 uses interface connection 720-1 to detect, obtain, or otherwise receive incoming synchronization signals. Relatedly, the hardware synchronization unit 156 uses interface connection 720-2 to transmit, provide, or otherwise pass outgoing synchronization signals.

[0103] The control logic 702 can detect and analyze outgoing and incoming synchronization signals and generate corresponding register values based on that analysis. The hardware synchronization unit 156 uses one or more of CSRs of the first memory 704 to store the register data values that indicate (or represent) context states for each of the devices.

[0104] In the example of Figure 7A. the hardware synchronization unit 120 includes a register 740 that stores data for an “acquire” parameter, a register 741 that stores data for a “release” parameter, a register 742 that stores data for a “signal” parameter, a register 743Atorney Docket No.: 56113-0818WO1

[0105] that stores data for a “waif’ parameter, a register 744 that stores data for a “NotificationQ” parameter, and a register 750 that stores data for “IRQs.”

[0106] An example synchronization operation that uses at least registers 740, 741, 742, and 743 will now be described.

[0107] IP device A (“IP_A”) may require a sync object for a given operation. For example, IP_A may need to configure a sync object so that as soon as IP_A is done with a given task on a portion of data, IP device B (“1P_B”) can be notified to do additional work on that portion of data. The required sync object may be triggered by a software driver or other compute logic of IP device A. IP_A initiates a read of register 740 to create, produce, or otherwise obtain the identifier / ID (i.e., sync_ID) of a newly created sync object. In some implementations, the IP A obtains the ID based on this command “sync my new sync = csu_contex0->acquire.”

[0108] The system 100 initiates a call to IP_B and passes the new sync in a manner that causes IP_B to wait on a sync state as a dependency while IP_A performs its given task on the data. For example, the system 100 can use the hardware synchronization unit 156 to initiate the call to a driver or other computational logic of IP_B. The hardware synchronization unit 156 can use register 741 to specify register IDs used by IP_A and IP_B during the synchronization context that covers processing this portion data. In some implementations, the system 100 initiates the call to IP_B and passes the new sync based on this command “process_IP_B(work. my new sync).”

[0109] IP_A uses the hardware synchronization unit 156 to signal the sync by writing its corresponding sync ID to register 742. The interaction between the IP_A and the hardware synchronization unit 156 may be initiated or managed by firmware and / or hardware logic supported by IP A. In some implementations, system 100 signals the sync and writes the corresponding sync ID based on this command “IP_A_Buffer_done_IRQ_event:

[0110] csu_contexO->signal = my_new_sync.” The system 100 can initiate processes involving IP_B based on this command “process_IP_B(work, sync).” In general, IP_A and IP_B can be any of the devices of circuit block 110. Additionally, “csu context” indicates register sets representing a client context in the hardware synchronization unit 156.

[0111] The hardware synchronization unit 156 causes IP_B to wait for the dependency regarding IP_A by writing the sync_ID into register 743. In some implementations, the system 100 causes IP_B to wait for the dependency based on the command “csu_contextl->wait = sync.” IP B performs its additional “work” on the portion of data in response to the sync being signaled or after receiving notification about the sync being done. For example.Atorney Docket No.: 56113-0818WO1

[0112] can IP_B perform or execute its additional ‘"work"’ that depends on this sync based on the command "while ((notified sync = csu context I ->notificationQ) != 0).” The command may¬ be executed using the IRQ Handler of the hardware synchronization unit 156.

[0113] In some implementations, there can be multiple syncs in flight for the same context. This means, upon CSU IRQ, there could be one or more syncs notified in the notification queue. In other words, for the while loop: “while ((notified sync = csu contextl->notificationQ)!= 0),” the owner of csu_contextlwill see which one of its pending work items is dependent on the notified sync. In this example the owner is IP_B. If IP_B was waiting for only one sync, then the while loop w ill just return non-0 once.

[0114] In the above and other examples, register 740 is a read-only (RO) register, whereas registers 741, 742, and 743 are write-only registers. In some implementations, data values stored at register 740 are used to “acquire” or create a sync object, a corresponding sync_ID, or both. Data values stored at the register 744 are used to notify an IP device of activity associated with a particular sync object. For example, the TPU 114 may be waiting on 10 different sync objects and the notification register queue is used to notify the TPU 114 that sync objects 1, 3, 5 have been signaled.

[0115] The IP devices can use or read data values for IRQs stored at register 750 to obtain notifications from the hardware synchronization unit 156 for a particular synchronization object or synchronization event. For example, the hardware synchronization unit 156 can use an IRQ at register 750 to notify the TPU 114 that an event or action has occurred with regard to a sync or sync object that the TPU 114 may have been waiting for.

[0116] More specifically, the hardw are synchronization unit 156 is configured to track the signaled syncs for each context and can use IRQs at register 750 to facilitate this tracking. When an IP device or the client of that context reads register 744. each read of that register 744 returns the sync_ID of a corresponding sync that has been signaled. In some implementations, each read of that register by the IP device will also clear the returned sync_ID from the notification queue. When there are no more signaled syncs in the notification queue, any read of register 744 would return a “0”.

[0117] The hardware synchronization unit 156 can identify one or more registers of the first memory 704 that are (or have been) allocated or assigned for a given context with reference to a specific IP device. The hardware synchronization unit 156 interacts with that IP device to permit the IP device to acquire register IDs for a given context and to release register IDs for that context. Each IP device communicates information pertaining to a synchronizationAtorney Docket No.: 56113-0818WO1

[0118] event to the hardware synchronization unit 156 by writing or updating values to the CSRs of the first memory 704 that are assigned to the IP device for a given synchronization context.

[0119] The second memory 706 is used to store a bookkeeping table that includes various “Bookkeeping” values for one or more contexts, context states, or both. The second memory 706 can store the bookkeeping values using a row configuration, where a given row is used or assigned to store one or more values for a context or context state. For example, the values are stored in a “Bookkeeping” table that includes one or more entries 706-0, 706-1, 706-n. In some implementations, the bookkeeping table includes n entries (e.g., rows), 706-n, where n is an integer greater than one.

[0120] The bookkeeping table is a global state table that maintains state information for each sync object, including global sync objects that are associated with one or more synchronization events. The bookkeeping table can include an entry (e.g., a row entry) for each sync object that is acquired from, or generated by, the hardware synchronization unit 156. Each entry or row of the bookkeeping table reflects state information for each sync object and indicates how many sync objects are active or available, how many sync object entries are available in the bookkeeping table, and how many sync object entries have already been acquired.

[0121] Each of the data values for a sync object entry can correspond to a synchronization event, an IP device, or both. The control logic 702 generates the bookkeeping table based on data values obtained or read from the first memory 704. For example, the control logic 702 can extract, read, or obtain data values for a register grouping 705 and use some (or all) of those data values to generate or populate entries of the bookkeeping table stored at the second memory 706.

[0122] In the example of Figure 7A. to maintain state information for a given sync object the bookkeeping table stores data values for “sync_ID,” “state,” and “waiter queue.” In other examples, the bookkeeping table can store more or fewer data values to maintain state information for a given sync object. The “sync_ID” identifies a sync object, the “state” indicates an example state of the sync object, and the “waiter queue” indicates a queue of clients or devices that may be waiting on that sync object. An example “state” of a synchronization object can include: i) free, indicating a sync object is free to be signalled; ii) unsignalled, indicating a sync object is not being (or has not been) signalled; iii) signalled, indicating a sync object is being (or has been) signalled; and iv) error, indicating an error state associated with a sync object.Atorney Docket No.: 56113-0818WO1

[0123] The synchronization unit 156 can track and maintain a variety of parameters for each synchronization object. For example, the hardware synchronization unit 156 can use the control logic 702 to determine a timestamp (e.g., a global timestamp) that indicates the last state change of a sync object. The control logic 702 can implement a counter that counts or tracks a number of times a sync object (or synchronization event) has been signalled. In some implementations, the counter parameter is particularly relevant for recurring sync events.

[0124] As described below, the control logic 702 can configure a recurring sync using combination logic that includes AND, OR, XOR, or other related logical operators. The control logic 702 can also implement a reference count that tracks existing references to a sync. The control logic 702 can use the reference count to trigger changing a state to "free" when a reference count value is set back to ‘"0”.

[0125] The hardware synchronization unit 156 is configured to generate compositional sync objects based on a composite or combination of two or more sync objects. More specifically, the control logic 702 can use its logical operators to generate a new (or composite) sync object from two distinct sync objects. For example, if a designer is interested in events that involve signaling both sync object A and B, the control logic 702 can use an ADD operator to generate a new a sync object by ANDing sync object A and sync object B. In another example, if a designer is interested in events that involve signaling any of sync objects A, B, and C, then the control logic 702 can use an OR operator to generate a new a sync object by ORing sync object A, sync object B, and sync object C.

[0126] In yet another example, the hardware synchronization unit 156 can generate 10 different sync objects that represent 10 different images or audio samples. The 10 images can represent a single stream with different image sizes. The SoC 102 may want to perform a concurrent operation on all 10 images. So, rather than perform a one-by-one wait operation on each image, the hardware synchronization unit 120 can use the AND operator to combine all 10 sync objects to create a new sync object that represents all 10 images. The hardware synchronization unit 156 can then implement a more streamlined wait event single using the single composite sync object.

[0127] In addition to composite sync objects, the hardware synchronization unit 156 can implement certain custom or exotic sync objects. In some implementations, the hardware synchronization unit 156 can implement conditional sync objects, counter-based sync objects, QoS counter-based sync objects, and timer-based sync objects. For example, the controlAtorney Docket No.: 56113-0818WO1

[0128] logic 702 can create a conditional sync object that only triggers if when a certain condition is satisfied, where the condition can be predefined or determined on the fly.

[0129] For example, the condition can be based on a certain processing event or a register value of a CSR of the SoC 102 or the sync module 108. The control logic 702 can also create a conditional sync object using the AND, OR, XOR operators. In some examples, the control logic 702 can create timer-based sync objects that trigger after a certain time value or after expiration of a certain time-duration. In some examples, the control logic 702 can create QoS counter-based sync objects that trigger after detecting a certain value in a program counter or upon reading a certain quality of service (QoS) count value in a counter register. For example, the QoS value can be based on a number of bytes or data value for a given service parameter, such as frames per second for a graphics processing operation.

[0130] In some implementations, the hardware synchronization unit 156 uses information indicated at a data block 710 to provide data values for updating and maintaining the bookkeeping table. The information at data block 710 can represent data values that are passed from the first memory 704 to the second memory 706. The values at data block 710 can be stored or maintained at the hardware synchronization unit 156. For example, the data block 710 may be a buffer that includes a subset of register values for a context of an IP device. The control logic 702 can read or extract values from that data block 710 and use those values to update, at a corresponding row of the bookkeeping table, state information for a sync object assigned to that IP device (or context).

[0131] In some implementations, the hardware synchronization unit 156 is a floating role migrating compute element of the SoC 102. Aspects of the hardware mechanism that incorporate floating role elements will migrate between different processing devices that may¬ be included among an active set of IP devices on the SoC 102. The hardware synchronization unit 156 can include logic (e.g., software / firmware) and control elements that enable sync roles to migrate or float among the IP devices of the circuit block 110. For example, to enable floating role sync functions at a given IP device, the hardware synchronization unit 156 can implement a respective portion of its control logic and one or more interface connections at that IP device.

[0132] The components and processes discussed herein can be implemented on a computing system. In particular, a computing system including a computing device and / or a mobile computing device can be used to implement the techniques described herein. For example, one or more processes, electronic design tools, and data can be implemented on or stored in the computing device or the mobile computing device.Atorney Docket No.: 56113-0818WO1

[0133] The computing device is intended to represent various forms of digital computers, such as laptops, desktops, workstations, personal digital assistants, servers, blade servers, mainframes, and other appropriate computers. The mobile computing device is intended to represent various forms of mobile devices, such as personal digital assistants, cellular telephones, smart-phones, mobile embedded radio systems, radio diagnostic computing devices, and other similar computing devices. The components shown here, their connections and relationships, and their functions, are meant to be examples only, and are not meant to be limiting.

[0134] The computing device includes a processor, a memory, a storage device, a high-speed interface connecting to the memory and multiple high-speed expansion ports, and a low-speed interface connecting to a low-speed expansion port and the storage device. Each of the processor, the memory, the storage device, the high-speed interface, the high-speed expansion ports, and the low-speed interface, are interconnected using various busses, and may be mounted on a common motherboard or in other manners as appropriate. The processor can process instructions for execution within the computing device, including instructions stored in the memory or on the storage device to display graphical information for a GUI on an external input / output device, such as a display coupled to the high-speed interface. In other implementations, multiple processors and / or multiple buses may be used, as appropriate, along with multiple memories and types of memory. In addition, multiple computing devices may be connected, with each device providing portions of the operations (e.g., as a server bank, a group of blade servers, or a multi-processor system). In some implementations, the processor is a single threaded processor. In some implementations, the processor is a multi-threaded processor. In some implementations, the processor is a quantum computer.

[0135] The memory stores information within the computing device. In some implementations, the memory is a volatile memory unit or units. In some implementations, the memory' is a non-volatile memory unit or units. The memory7may also be another form of computer-readable medium, such as a magnetic or optical disk.

[0136] The storage device is capable of providing mass storage for the computing device. In some implementations, the storage device may be or include a computer-readable medium, such as a floppy disk device, a hard disk device, an optical disk device, or a tape device, a flash memory7or other similar solid-state memory device, or an array of devices, including devices in a storage area network or other configurations. Instructions can be stored in an information carrier. The instructions, when executed by one or more processing devices (forAtorney Docket No.: 56113-0818WO1

[0137] example, processor), perform one or more methods, such as those described above. The instructions can also be stored by one or more storage devices such as computer- or machine-readable mediums (for example, the memory, the storage device, or memory on the processor). The high-speed interface manages bandwidth-intensive operations for the computing device, while the low-speed interface manages lower bandwidth-intensive operations. Such allocation of functions is an example only. In some implementations, the high-speed interface is coupled to the memory, the display (e.g., through a graphics processor or accelerator), and to the high-speed expansion ports, which may accept various expansion cards (not shown). In the implementation, the low-speed interface is coupled to the storage device and the low-speed expansion port. The low-speed expansion port, which may include various communication ports (e.g., USB, Bluetooth, Ethernet, wireless Ethernet) may be coupled to one or more input / output devices, such as a keyboard, a pointing device, a scanner, or a networking device such as a switch or router, e.g., through a network adapter.

[0138] The computing device may be implemented in a number of different forms, as shown in the figure. For example, it may be implemented as a standard server, or multiple times in a group of such servers. In addition, it may be implemented in a personal computer such as a laptop computer. It may also be implemented as part of a rack server system. Alternatively, components from the computing device may be combined with other components in a mobile device, such as a mobile computing device. Each of such devices may include one or more of the computing device and the mobile computing device, and an entire system may be made up of multiple computing devices communicating with each other.

[0139] The mobile computing device includes a processor, a memory’, an input / output device such as a display, a communication interface, and a transceiver, among other components. The mobile computing device may also be provided with a storage device, such as a microdrive or other device, to provide additional storage. Each of the processor, the memory, the display, the communication interface, and the transceiver, are interconnected using various buses, and several of the components may be mounted on a common motherboard or in other manners as appropriate.

[0140] The processor can execute instructions within the mobile computing device, including instructions stored in the memory. The processor may7be implemented as a chipset of chips that include separate and multiple analog and digital processors. The processor may provide, for example, for coordination of the other components of the mobile computing device , such as control of user interfaces, applications run by the mobile computing device . and wireless communication by the mobile computing device .Atorney Docket No.: 56113-0818WO1

[0141] The processor may communicate with a user through a control interface and a display interface coupled to the display. The display may be, for example, a TFT (Thin-Film-Transistor Liquid Crystal Display) display or an OLED (Organic Light Emiting Diode) display, or other appropriate display technology. The display interface may include appropriate circuitry for driving the display to present graphical and other information to a user. The control interface may receive commands from a user and convert them for submission to the processor. In addition, an external interface may provide communication with the processor, so as to enable near area communication of the mobile computing device with other devices. The external interface may provide, for example, for wired communication in some implementations, or for wireless communication in other implementations, and multiple interfaces may also be used.

[0142] The memory stores information within the mobile computing device. The memory can be implemented as one or more of a computer-readable medium or media, a volatile memory' unit or units, or a non-volatile memory unit or units. An expansion memory may also be provided and connected to the mobile computing device through an expansion interface, which may include, for example, a SIMM (Single In Line Memory Module) card interface. The expansion memory may provide extra storage space for the mobile computing device, or may also store applications or other information for the mobile computing device. Specifically, the expansion memory may include instructions to carry out or supplement the processes described herein and may include secure information also. Thus, for example, the expansion memory may be provided as a security module for the mobile computing device, and may be programmed with instructions that permit secure use of the mobile computing device. In addition, secure applications may be provided via the SIMM cards, along with additional information, such as placing identifying information on the SIMM card in a non-hackable manner.

[0143] The memory may include, for example, flash memory7and / or NVRAM memory (nonvolatile random access memory), as discussed below. In some implementations, instructions are stored in an information carrier such that the instructions, when executed by one or more processing devices (for example, processor), perform one or more methods, such as those described above. The instructions can also be stored by one or more storage devices, such as one or more computer- or machine-readable mediums (for example, the memory7, the expansion memory, or memory7on the processor). In some implementations, the instructions can be received in a propagated signal, for example, over the transceiver or the external interface.Atorney Docket No.: 56113-0818WO1

[0144] The mobile computing device may communicate wirelessly through the communication interface, which may include digital signal processing circuitry in some cases. The communication interface may provide for communications under various modes or protocols, such as GSM voice calls (Global System for Mobile communications), SMS (Short Message Service), EMS (Enhanced Messaging Service), or MMS messaging (Multimedia Messaging Sendee), CDMA (code division multiple access), TDMA (time division multiple access), PDC (Personal Digital Cellular), WCDMA (Wideband Code Division Multiple Access), CDMA2000, or GPRS (General Packet Radio Service), LTE, 4G / 5G / 6G cellular, among others. Such communication may occur, for example, through the transceiver using a radio frequency. In addition, short-range communication may occur, such as using a Bluetooth. Wi-Fi, or other such transceiver (not shown). In addition, a GPS (Global Positioning System) receiver module may provide additional navigation- and location-related wireless data to the mobile computing device, which may be used as appropriate by applications running on the mobile computing device.

[0145] The mobile computing device may also communicate audibly using an audio codec, which may receive spoken information from a user and convert it to usable digital information. The audio codec may likewise generate audible sound for a user, such as through a speaker, e.g., in a handset of the mobile computing device. Such sound may include sound from voice telephone calls, may include recorded sound (e g., voice messages, music files, among others) and may also include sound generated by applications operating on the mobile computing device.

[0146] The mobile computing device may be implemented in a number of different forms, as shown in the figure. For example, it may be implemented as a cellular telephone. It may also be implemented as part of a smart-phone, personal digital assistant, or other similar mobile device.

[0147] Embodiments of the subject mater and the functional operations described in this specification can be implemented in digital electronic circuitry7, in tangibly-embodied computer software or firmware, in computer hardware, including the structures disclosed in this specification and their structural equivalents, or in combinations of one or more of them. Embodiments of the subject matter described in this specification can be implemented as one or more computer programs, i.e., one or more modules of computer program instructions encoded on a tangible non-transitory storage medium for execution by, or to control the operation of, data processing apparatus. The computer storage medium can be a machine-readable storage device, a machine-readable storage substrate, a random or serial accessAtorney Docket No.: 56113-0818WO1

[0148] memory device, or a combination of one or more of them. Alternatively or in addition, the program instructions can be encoded on an artificially-generated propagated signal, e.g., a machine-generated electrical, optical, or electromagnetic signal, that is generated to encode information for transmission to suitable receiver apparatus for execution by a data processing apparatus.

[0149] The term “data processing apparatus” refers to data processing hardware and encompasses all kinds of apparatus, devices, and machines for processing data, including by way of example a programmable processor, a computer, or multiple processors or computers. The apparatus can also be, or further include, special purpose logic circuitry, e.g., an FPGA (field programmable gate array) or an ASIC (application specific integrated circuit). The apparatus can optionally include, in addition to hardware, code that creates an execution environment for computer programs, e.g., code that constitutes processor firmware, a protocol stack, a database management system, an operating system, or a combination of one or more of them. - A computer program which may also be referred to or described as a program, software, a software application, an app, a module, a software module, a script, or code) can be written in any form of programming language, including compiled or interpreted languages, or declarative or procedural languages, and it can be deployed in any form, including as a standalone program or as a module, component, subroutine, or other unit suitable for use in a computing environment. A program may, but need not, correspond to a file in a file system. A program can be stored in a portion of a file that holds other programs or data, e.g., one or more scripts stored in a markup language document, in a single file dedicated to the program in question, or in multiple coordinated files, e.g., files that store one or more modules, sub-programs, or portions of code. A computer program can be deployed to be executed on one computer or on multiple computers that are located at one site or distributed across multiple sites and interconnected by a data communication network.

[0150] The processes and logic flows can also be performed by special purpose logic circuitry, e.g., an FPGA or an ASIC, or by a combination of special purpose logic circuitry and one or more programmed computers. Computers suitable for the execution of a computer program can be based on general or special purpose microprocessors or both, or any other kind of central processing unit. Generally, a central processing unit will receive instructions and data from a read only memory' or a random access memory' or both. The essential elements of a computer are a central processing unit for performing or executing instructions and one or more memory devices for storing instructions and data. The centralAtorney Docket No.: 56113-0818WO1

[0151] processing unit and the memory can be supplemented by, or incorporated in, special purpose logic circuitry. Generally, a computer will also include, or be operatively coupled to receive data from or transfer data to, or both, one or more mass storage devices for storing data, e.g., magnetic, magneto-optical disks, or optical disks. However, a computer need not have such devices. Moreover, a computer can be embedded in another device, e.g., a mobile telephone, a personal digital assistant (PDA), a mobile audio or video player, a game console, a Global Positioning System (GPS) receiver, or a portable storage device, e.g.. a universal serial bus (USB) flash drive, to name just a few.

[0152] Computer readable media suitable for storing computer program instructions and data include all forms of non-volatile memory, media and memory devices, including by way of example semiconductor memory devices, e.g., EPROM, EEPROM, and flash memory devices; magnetic disks, e.g., internal hard disks or removable disks; magneto-optical disks; and CD-ROM and DVD-ROM disks. -While this specification contains many specific implementation details, these should not be construed as limitations on the scope of any invention or on the scope of what may be claimed, but rather as descriptions of features that may be specific to particular embodiments of particular inventions. Certain features that are described in this specification in the context of separate embodiments can also be implemented in combination in a single embodiment. Conversely, various features that are described in the context of a single embodiment can also be implemented in multiple embodiments separately or in any suitable subcombination. Moreover, although features may be described above as acting in certain combinations and even initially be claimed as such, one or more features from a claimed combination can in some cases be excised from the combination, and the claimed combination may be directed to a subcombination or variation of a subcombination.

[0153] Similarly, while operations are depicted in the drawings and recited in the claims in a particular order, this should not be understood as requiring that such operations be performed in the particular order show n or in sequential order, or that all illustrated operations be performed, to achieve desirable results. In certain circumstances, multitasking and parallel processing may be advantageous. Moreover, the separation of various system modules and components in the embodiments described above should not be understood as requiring such separation in all embodiments, and it should be understood that the described program components and systems can generally be integrated together in a single software product or packaged into multiple software products.Attorney Docket No.: 56113-0818WO1

[0154] Particular embodiments of the subject matter have been described. Other embodiments are within the scope of the following claims. For example, the actions recited in the claims can be performed in a different order and still achieve desirable results. As one example, the processes depicted in the accompanying figures do not necessarily require the particular order shown, or sequential order, to achieve desirable results. In some cases, multitasking and parallel processing may be advantageous.

[0155] What is claimed is:

Claims

Atorney Docket No.: 56113-0818WO1CLAIMS1. A method performed using an embedded processor implemented on a system-on-chip (SoC), wherein the SoC comprises the embedded processor, a first processing core, an image signal processor (ISP), and at least one hardware accelerator, the method comprising:receiving, from the first processing core and by the embedded processor, a request to perform a set of image processing operations, wherein the request specifies processing parameters to be used in performing the set of imaging processing operations; and synchronizing operations of the ISP, the at least one hardware accelerator, and an on-device image sensor based on the request and without any involvement of the CPU and an image processing stack of a CPU implemented on the SoC, comprising:configuring, by the embedded processor and based on the request, the on-device image sensor to initiate the imaging operation,configuring, by the embedded processor and using a dedicated image processing software stack resident on the embedded processor, the ISP to process streamed image data received from the image sensor using the processing parameters specified in the request, wherein configuring the ISP and the image sensor is performed without any involvement by the image processing stack on the CPU, andoffloading, by the embedded processor, one or more image processing operations of the set of image processing operations to the at least one hardware accelerator; andproviding, by the embedded processor and to the first processing core, a processed image upon competition of operations by the ISP and the at least one hardware accelerator.

2. The method of claim 1, wherein the first processing core includes one of a central processing unit, a security agent (SA), or an always-on sub-system (AOSS).

3. The method of claim 1, wherein the embedded processor is accessible by the first processing core using a dedicated application programming interface (API).

4. The method of claim 1, wherein the hardware accelerator can include at least one of a digital signal processor, a graphics processing unit, a tensor processing unit, or another processor core implemented on the SoC.Atorney Docket No.: 56113-0818WO15. The method of claim 1, wherein the set of image processing operations include an image capture operation and an image data processing operation, wherein the embedded processor schedules iterative execution of the image capture and the image data processing operations.

6. The method of claim 1, the synchronization operations comprising:generating, by the embedded processor, based on the request a context for execution of the set of operations.

7. The method of claim 1, further comprising:powering on, by the embedded processor, at least one of the ISP, the image sensor and the at least one hardware accelerator prior to configuring the image senor to initiate imaging operations.

8. The method of claim 1, further comprising:powering off, by the embedded processor, at least one of the ISP, the image sensor and the at least one hardware accelerator after competition of operations by the ISP and the at least one hardware accelerator.

9. The method of claim 1 , wherein the request to perform the set of image processing operations includes a graph description specifying a sequence of operations, the method comprising:synchronize execution of the image processing operation on the one or more cores based on the graph description.

10. The method of claim 1, wherein the first image processing stack includes an image noise reduction functionality and the second image processing stack lacks the image noise reduction functionality.

11. The method of claim 1 , wherein the second image processing stack includes an image sensor driver configured to receive image data from the image signal processor and generate configuration parameters for the image sensor and the image signal processor based on the image data.Atorney Docket No.: 56113-0818WO112. One or more non-transitory computer-readable storage media storing instructions that when executed by one or more processors cause the one or more processors to perform operations performed using an embedded processor implemented on a system-on-chip (SoC), wherein the SoC comprises the embedded processor, a first processing core, an image signal processor (ISP), and at least one hardware accelerator, the operations comprising:receiving, from the first processing core and by the embedded processor, a request to perform a set of image processing operations, wherein the request specifies processing parameters to be used in performing the set of imaging processing operations; and synchronizing operations of the ISP, the at least one hardware accelerator, and an on-device image sensor based on the request and without any involvement of the CPU and an image processing stack of a CPU implemented on the SoC, comprising:configuring, by the embedded processor and based on the request, the on-device image sensor to initiate the imaging operation,configuring, by the embedded processor and using a dedicated image processing software stack resident on the embedded processor, the ISP to process streamed image data received from the image sensor using the processing parameters specified in the request, wherein configuring the ISP and the image sensor is performed without any involvement by the image processing stack on the CPU, andoffloading, by the embedded processor, one or more image processing operations of the set of image processing operations to the at least one hardw are accelerator; andproviding, by the embedded processor and to the first processing core, a processed image upon competition of operations by the ISP and the at least one hardware accelerator.

13. The computer-readable storage media of claim 12, wherein the first processing core includes one of a central processing unit, a security7agent (SA), or an always-on sub-system (AOSS).

14. The computer-readable storage media of claim 12, wherein the embedded processor is accessible by the first processing core using a dedicated application programming interface (API).Atorney Docket No.: 56113-0818WO115. The computer-readable storage media of claim 12, wherein the hardware accelerator can include at least one of a digital signal processor, a graphics processing unit, a tensor processing unit, or another processor core implemented on the SoC.

16. The computer-readable storage media of claim 12, wherein the set of image processing operations include an image capture operation and an image data processing operation, wherein the embedded processor schedules iterative execution of the image capture and the image data processing operations.

17. The computer-readable storage media of claim 12, the synchronization operations comprising:generating, by the embedded processor, based on the request a context for execution of the set of operations.

18. The computer-readable storage media of claim 12, the operations further comprising:powering on, by the embedded processor, at least one of the ISP, the image sensor and the at least one hardware accelerator prior to configuring the image senor to initiate imaging operations.

19. The computer-readable storage media of claim 12, the operations further comprising:powering off, by the embedded processor, at least one of the ISP, the image sensor and the at least one hardware accelerator after competition of operations by the ISP and the at least one hardware accelerator.

20. The computer-readable storage media of claim 12, wherein the request to perform the set of image processing operations includes a graph description specifying a sequence of operations, the operations comprising:synchronize execution of the image processing operation on the one or more cores based on the graph description.

21. The computer-readable storage media of claim 12, wherein the first image processing stack includes an image noise reduction functionality and the second image processing stack lacks the image noise reduction functionality.Atorney Docket No.: 56113-0818WO122. The computer-readable storage media of claim 12, wherein the second image processing stack includes an image sensor driver configured to receive image data from the image signal processor and generate configuration parameters for the image sensor and the image signal processor based on the image data.

23. A system, comprising:one or more processors; andone or more storage devices storing instructions that when executed by the one or more processors to perform operations performed using an embedded processor implemented on a system-on-chip (SoC), wherein the SoC comprises the embedded processor, a first processing core, an image signal processor (ISP), and at least one hardware accelerator, the operations comprising:receiving, from the first processing core and by the embedded processor, a request to perform a set of image processing operations, wherein the request specifies processing parameters to be used in performing the set of imaging processing operations; and synchronizing operations of the ISP, the at least one hardware accelerator, and an on-device image sensor based on the request and without any involvement of the CPU and an image processing stack of a CPU implemented on the SoC, comprising:configuring, by the embedded processor and based on the request, the on-device image sensor to initiate the imaging operation.configuring, by the embedded processor and using a dedicated image processing software stack resident on the embedded processor, the ISP to process streamed image data received from the image sensor using the processing parameters specified in the request, wherein configuring the ISP and the image sensor is performed without any involvement by the image processing stack on the CPU, andoffloading, by the embedded processor, one or more image processing operations of the set of image processing operations to the at least one hardware accelerator; andproviding, by the embedded processor and to the first processing core, a processed image upon competition of operations by the ISP and the at least one hardware accelerator.

24. The system of claim 23, wherein the first processing core includes one of a central processing unit, a security agent (SA), or an always-on sub-system (AOSS).Atorney Docket No.: 56113-0818WO125. The system of claim 23, wherein the embedded processor is accessible by the first processing core using a dedicated application programming interface (API).

26. The system of claim 23, wherein the hardware accelerator can include at least one of a digital signal processor, a graphics processing unit, a tensor processing unit, or another processor core implemented on the SoC.

27. The system of claim 23, wherein the set of image processing operations include an image capture operation and an image data processing operation, wherein the embedded processor schedules iterative execution of the image capture and the image data processing operations.

28. The system of claim 23, the synchronization operations comprising:generating, by the embedded processor, based on the request a context for execution of the set of operations.

29. The system of claim 23, the operations further comprising:powering on, by the embedded processor, at least one of the ISP, the image sensor and the at least one hardware accelerator prior to configuring the image senor to initiate imaging operations.

30. The system of claim 23, the operations further comprising:powering off, by the embedded processor, at least one of the ISP, the image sensor and the at least one hardware accelerator after competition of operations by the ISP and the at least one hardware accelerator.

31. The system of claim 23, wherein the request to perform the set of image processing operations includes a graph description specifying a sequence of operations, the operations comprising:synchronize execution of the image processing operation on the one or more cores based on the graph description.Attorney Docket No.: 56113-0818WO132. The system of claim 23, wherein the first image processing stack includes an image noise reduction functionality and the second image processing stack lacks the image noise reduction functionality.

33. The system of claim 23, wherein the second image processing stack includes an image sensor driver configured to receive image data from the image signal processor and generate configuration parameters for the image sensor and the image signal processor based on the image data.

Images