Systems and methods for low latency image sensor and image signal processor configuration

Abstract

Methods, systems, and apparatus for reducing latency of configuration of image sensor and image signal processor. A computing system can include a primary unit and a secondary unit that are coupled with the image sensor and the image signal processor. The secondary unit can receive a trigger event indicating a request from an application for image data. The secondary unit can receive image statistics of one or more image frames of a series of image frames generated by the image signal processor based on output of the image sensor. The secondary unit can adjust values of configuration parameters for the image sensor and the image signal processor. Upon receiving an indication that the primary unit has initialized, the secondary unit can provide at least one value of the configuration parameters to the primary unit.

Description

[0001] Atorney Docket No.: 56113-0819WO1

[0002] SYSTEMS AND METHODS FOR LOW LATENCY IMAGE SENSOR AND IMAGE SIGNAL PROCESSOR CONFIGURATION

[0003] TECHNICAL FIELD

[0004] This specification relates to image processing, and more particularly to systems and methods for low latency configuration of image sensors and image signal processors.

[0005] BACKGROUND

[0006] Image processing applications (including programs or applications executing on a device such as a camera application) can utilize image data generated by one or more image sensors on the device. For example, a camera application in a mobile device that has a display screen can display in near real time the image data captured by the one or more image sensor on the device. The camera application can provide the user various options such as image capture, video, and other image-based operations that the user can choose from while viewing the image data on the display screen.

[0007] The image processing applications can rely on the image sensor and programming instructions executing on hardware processors in the device for capturing and processing images for display to the device’s user. For example, the device can include hardware processing cores such as a central processing unit and an image signal processor coupled with the image sensor. The central processing unit can run an operating system and an image sensor driver while the image signal processor can process image frames generated by the image sensor.

[0008] When the image processing application is launched by the user, the operating system, which runs on the central processing unit, can allocate memory to the image processing application and provide the image processing application access to the image sensor driver. The image sensor driver can, in turn, initialize the image sensor and the image signal processor. Typically, the image frames output by the image signal processor are shown on the display of the device as they become available. In some instances, the image frames are displayed to the user even while the image sensor driver is in the process of configuring the image sensor and the image signal processor. The configuration can include, for example, seting exposure and gain parameters for the image sensor and the image signal processor. The image sensor driver can generate values for these parameters based on processing a set of initial image frames captured by the image sensor as well as ambient conditions. Thus, theAtorney Docket No.: 56113-0819WO1

[0009] device may display the set of initial image frames to the user even though the image sensor and the image signal processor are not fully configured. This can result in the image frames being displayed to the user being of low quality until the configuration of the image sensor and the image signal processor is complete. In some other instances, the image frames are not displayed until the image sensor driver has configured the image sensor and the image signal processor (fully or to predetermined threshold values). In such instances, the user can experience a delay between the time the user launches the image processing application and the time when the image frames are provided on the display.

[0010] The duration for which potentially low-quality' images are displayed to the user or the duration for which no images are displayed to the user may be longer in cases where the central processing unit is in a standby or sleep mode when the request to launch the image processing application is received. In such instances, the device may need additional time to start the central processing unit, the operating system, and the image sensor driver before beginning initialization of the image sensor and the image signal processor. This additional time can contribute to increasing the duration for which low-quality' images are displayed to the user or no images are displayed to the user.

[0011] SUMMARY

[0012] This specification describes techniques for reducing the latency of configuring image sensors and image signal processors in devices. In particular, the techniques discussed herein provide a secondary unit (e.g., an embedded processor) that includes a secondary' image sensor driver that configures the image sensor and the image signal processor while a primary unit that includes a central processing unit, an operating system, and a primary image sensor driver is initialized. Specifically, the secondary image sensor driver configures the image sensor and the image signal processor concurrently with the initialization of the primary unit. Thus, when the primary unit is initialized, the secondary unit completes at least a portion of the configuration of the image sensor and the image signal processor. As a result, when the primary’ unit provides image frames generated by the image sensor and the image signal processor for display (e.g.. on the device’s display screen), the image frames are of beter quality compared to image frames that would otherwise be displayed if the primary' unit alone were to configure the image sensor and the image signal processor upon initialization.

[0013] Alternatively, in instances where the primary unit delays the display of an initial set of image frames until the image sensor driver has configured the image sensor and the image signal processor (fully or to predetermined threshold values), the secondary unit configures theAtorney Docket No.: 56113-0819WO1

[0014] image sensor and the image signal processor while the primary unit initializes, thereby reducing the delay with which image frames are provided on the display screen of the device. Once the primary unit is initialized, the primary unit may take over, from the secondary unit, the subsequent configuring of the image sensor and the image signal processor.

[0015] The secondary image sensor driver can begin to perform configuration of the image sensor and the image signal processor in response to a trigger event such as launching of an image processing application that requests or utilizes images. The trigger event can. in parallel, trigger initialization of the application on the primary unit, where the initialization can include allocation of resources to the image processing application, checking access permissions of the image processing application, etc. While the primary unit is undergoing initialization, the secondary image sensor driver can activate the image sensor and the image signal processor, receive image statistics related to the initial image frames, and generate values for configuration parameters of the image sensor and the image signal processor such that the generated image frames have a predetermined quality. The secondary¬ unit can continue to configure the image sensor and / or the image signal processor until a threshold quality of the image frames is achieved, or the image sensor and / or the image signal processor are fully configured or configured to predetermined values, or until the primary unit has initialized. After the primary unit is initialized, the primary- unit can take over from the secondary- unit any additional configuration of the image sensor and the image signal processor.

[0016] Particular examples of the subject mater described in this specification can be implemented so as to realize one or more of the following advantages. Having a secondaryunit configure the image sensor and the image signal processor concurrently with the initialization process of the primary unit can reduce the latency in configuring the image sensor and the image signal processor. The reduction in latency can, in turn, reduce the risk of displaying low-quality- image frames to the user when an image processing application is launched by the user or can reduce the duration for which no image frames are displayed to the user. The secondary unit can be used to converge the values of the configuration parameters of the image sensor and the image signal processors as close to predetermined values as possible before the handing over the configuration to the primary- unit when the primary- unit has initialized. This reduces the burden and the time required by the primaryunit to fully configure the image sensor and the image signal processor.

[0017] In some instances, the central processing unit may be in a standby or sleep mode when the request to launch the image processing application is received. In such instances.Atorney Docket No.: 56113-0819WO1

[0018] the device may need additional time to start the central processing unit, the operating system, and the image sensor driver before beginning initialization of the image sensor and the image signal processor. This additional time can contribute to increasing the duration for which low-quality images are displayed to the user or no images are displayed to the user. By having the secondary unit fully or partially configure the image sensor and the image signal processor while the primary unit initializes, the delay associated with providing the images on the display screen can be reduced or the risk of providing low-quality image frames for display on the display screen can be reduced. Further, power consumed for configuring the image sensor and the image signal processor by the primary unit can be greater than the power consumed in configuring the image sensor and the image signal processor by the secondary unit. Therefore, configuring the image sensor and the image signal processor by the secondary unit can save power.

[0019] One general aspect includes a method for configuring an image sensor and an image signal processor of a device may include a primary unit including a primary image sensor driver and a secondary unit including a secondary image sensor driver. The method also includes receiving, at a secondary unit, a trigger event indicating a request from an application for image data, receiving, at the secondary unit from the image signal processor, image statistics of one or more image frames of a series of image frames, the image signal processor generating the series of image frames based on output of the image sensor; adjusting, by the secondary unit, values of configuration parameters for the image sensor and the image signal processor based on the image statistics; receiving, at the secondary unit, an indication that the primary' unit has initialized; and responsive to receiving the indication, providing, by the secondary unit, at least one of the values of the configuration parameters to the primary unit.

[0020] Implementations may include one or more of the following features. The method may include: iteratively adjusting, by the secondary unit, values of configuration parameters for the image sensor and the image signal processor based on the image statistics of multiple image frames of the series of image frames. The image buffer is accessible by the primary unit when the primary unit is initialized. The secondary unit receives the image statistics from the image signal processor when the primary unit is in a standby state or being initialized. The method may include: generating, by the primary image sensor driver after the primary' unit has initialized, additional values of the configuration parameters based on a subset of the series of image frames; and configuring the image sensor and the image signal processor based on the additional values of the configuration parameters. The method may include:Atorney Docket No.: 56113-0819WO1

[0021] receiving, at the secondary image sensor driver form the primary image sensor driver, the additional values of the configuration parameters; and configuring, by the secondary image sensor driver, the image sensor and the image signal processor based on the additional values of the configuration parameters. The secondary image sensor driver includes a secondary' algorithm, the method may include: generating, by the secondary algorithm, the values of the configuration parameters based on the image statistics. The primary’ image sensor driver includes more image processing functions than the secondary image sensor driver. The method may include: responsive to receiving the indication that the primary unit has initialized, at the secondary' unit, ceasing generating values of configuration parameters for the image sensor and the image signal processor. The method may include: subsequent to ceasing generating values for the configuration parameters, receiving, at the secondary unit from the primary unit, additional values for the configuration parameters; and writing, by the secondary unit, the additional values to a configuration parameter register. The primary unit is an application processor and the secondary unit is an embedded processor. The method may include: responsive to receiving the indication that the primary unit has initialized, displaying subsequently generated image frames on a display screen.

[0022] One general aspect includes one or more non-transitory computer-readable storage media storing instructions that when executed by one or more processors cause the one or more processors to perform operations for configuring an image sensor and an image signal processor of a device comprising a primary unit including a primary image sensor driver and a secondary unit including a secondary image sensor driver. The one or more non - transitory computer - readable storage media storing instructions also includes receiving, at a secondary' unit, a trigger event indicating a request from an application for image data, receiving, at the secondary unit from the image signal processor, image statistics of one or more image frames of a series of image frames, the image signal processor generating the series of image frames based on output of the image sensor; adjusting, by the secondary unit, values of configuration parameters for the image sensor and the image signal processor based on the image statistics; receiving, at the secondary unit, an indication that the primary’ unit has initialized; and responsive to receiving the indication, providing, by the secondary unit, at least one of the values of the configuration parameters to the primary unit.

[0023] Implementations may include one or more of the following features. The computer-readable storage media, where the operations may include: iteratively adjusting, by the secondary unit, values of configuration parameters for the image sensor and the image signal processor based on the image statistics of multiple image frames of the series of imageAtorney Docket No.: 56113-0819WO1

[0024] frames. The image buffer is accessible by the primary unit when the primary unit is initialized. The secondary unit receives the image statistics from the image signal processor when the primary unit is in a standby state or being initialized. The computer-readable storage media, where the operations may include: generating, by the primary image sensor driver after the primary unit has initialized, additional values of the configuration parameters based on a subset of the series of image frames; and configuring the image sensor and the image signal processor based on the additional values of the configuration parameters. The computer-readable storage media, where the operations may include: receiving, at the secondary image sensor driver form the primary' image sensor driver, the additional values of the configuration parameters; and configuring, by the secondary' image sensor driver, the image sensor and the image signal processor based on the additional values of the configuration parameters. The secondary image sensor driver includes a secondary algorithm, the operations may include: generating, by the secondary algorithm, the values of the configuration parameters based on the image statistics. The primary' image sensor driver includes more image processing functions than the secondary image sensor driver. The computer-readable storage media the operations may include: responsive to receiving the indication that the primary7unit has initialized, at the secondary7unit, ceasing generating values of configuration parameters for the image sensor and the image signal processor. The computer-readable storage media, where the operations may include: subsequent to ceasing generating values for the configuration parameters, receiving, at the secondary unit from the primary unit, additional values for the configuration parameters; and writing, by the secondary7unit, the additional values to a configuration parameter register. The primary7unit is an application processor and the secondary unit is an embedded processor. The computer-readable storage media, where the operations may include: responsive to receiving the indication that the primary unit has initialized, displaying subsequently generated image frames on a display screen.

[0025] One general aspect includes a system. The system includes one or more processors; and one or more storage devices storing instructions that when executed by the one or more processors to perform operations for configuring an image sensor and an image signal processor of a device comprising a primary unit including a primary image sensor driver and a secondary7unit including a secondary7image sensor driver, the operations may include: receiving, at a secondary unit, a trigger event indicating a request from an application for image data, receiving, at the secondary unit from the image signal processor, image statistics of one or more image frames of a series of image frames, the image signal processorAtorney Docket No.: 56113-0819WO1

[0026] generating the series of image frames based on output of the image sensor; adjusting, by the secondary unit, values of configuration parameters for the image sensor and the image signal processor based on the image statistics; receiving, at the secondary unit, an indication that the primary unit has initialized; and responsive to receiving the indication, providing, by the secondary unit, at least one of the values of the configuration parameters to the primary unit.

[0027] Implementations may include one or more of the following features. The system, where the operations may include: iteratively adjusting, by the secondary unit, values of configuration parameters for the image sensor and the image signal processor based on the image statistics of multiple image frames of the series of image frames. The image buffer is accessible by the primary' unit when the primary unit is initialized. The secondary' unit receives the image statistics from the image signal processor when the primary unit is in a standby state or being initialized. The system, where the operations may include: generating, by the primary image sensor driver after the primary unit has initialized, additional values of the configuration parameters based on a subset of the series of image frames; and configuring the image sensor and the image signal processor based on the additional values of the configuration parameters. The system, where the operations may include: receiving, at the secondary image sensor driver form the primary' image sensor driver, the additional values of the configuration parameters; and configuring, by the secondary' image sensor driver, the image sensor and the image signal processor based on the additional values of the configuration parameters. The secondary image sensor driver includes a secondary algorithm, the operations may7include: generating, by the secondary algorithm, the values of the configuration parameters based on the image statistics. The primary' image sensor driver includes more image processing functions than the secondary image sensor driver. The system, where the operations may include: responsive to receiving the indication that the primary unit has initialized, at the secondary unit, ceasing generating values of configuration parameters for the image sensor and the image signal processor. The system, where the operations may include: subsequent to ceasing generating values for the configuration parameters, receiving, at the secondary’ unit from the primary unit, additional values for the configuration parameters; and writing, by the secondary unit, the additional values to a configuration parameter register. The primary unit is an application processor and the secondary' unit is an embedded processor. The system, where the operations may include: responsive to receiving the indication that the primary unit has initialized, displaying subsequently generated image frames on a display screen.Atorney Docket No.: 56113-0819WO1

[0028] The details of one or more embodiments of the subject mater of this specification are set forth in the accompanying drawings and the description below. Other features, aspects, and advantages of the subject mater will become apparent from the description, the drawings, and the claims.

[0029] BRIEF DESCRIPTION OF THE DRAWINGS

[0030] Figure 1 is a block diagram of a portion of an example integrated circuit design. Figure 2 shows a flow diagram depicting the process steps taken by the example computing system to launch an image processing application.

[0031] Figure 3 depicts the operation of the primary’ unit and the secondary unit in configuring the image sensor and the ISP.

[0032] Figure 4 shows a timing diagram depicting the relative timings of the primary' unit and the secondary' unit in configuring the image sensor and the ISP.

[0033] Figure 5 shows a flow diagram of an example process for configuring an image sensor and an image signal processor.

[0034] Like reference numbers and designations in the various drawings indicate like elements.

[0035] DETAILED DESCRIPTION

[0036] Figure 1 is a block diagram of an example computing system 100. The example computing system 100 can include a system-on-chip 102 (“SoC 102”), which, in turn, can include a primary' unit 104, a secondary unit 106, a shared memory 108, and an IP / circuit block 110. The SoC 102 can be implemented in an integrated circuit of an example user / client device 130. consumer device, or mobile device, where each of these devices can include items such as a smartphone 130a, a tablet 130b, a laptop 130c, and a smartwatch or wearable device 130d. The client device 130 may also include other items such as an eNotebook, Netbook, smart speaker, or mobile computer. In some examples, the system 100 and the SoC 102 is an integrated circuit of a desktop computer, network server, or related cloud-based asset. The SoC 102 can also be coupled with an image sensor 152. While Figure 1 shows a single image sensor 152, more than one image sensor can be included. In particular, the image sensor 152 can be part of a client device 130.

[0037] The primary unit 104 can include a central processing unit 120 (“CPU 140”). The CPU 140 can be a general-purpose CPU (e.g., a single or multi-core CPU). An operating system 142 can run on the CPU 140. Examples of operating system 142 can include theAtorney Docket No.: 56113-0819WO1

[0038] Android operating system, the iOS operating system, the Windows operating system, etc. The operating system 142 can run one or more image processing applications. For example, image processing applications can include photography applications, video call applications, etc. The operating system 142 can include one or more hardware drivers that allow applications running on the operating system 142 to control hardware. For example, the operating system 142 can include a primary image sensor driver 144 which can control the operation of the image sensor 151 as well as communicate with an image signal processor 112 (’‘ISP 112”). The primary image sensor driver 144 can include image sensor drivers such as “Camera2” that is included in the Android operating system. The Camera2 driver has an application programming interface (API) that allows applications running on the operating system 142 to make calls to functions provided by the driver.

[0039] The SoC 102 also can include the secondary unit 106, which can carry out specific functions related to image processing tasks. In particular, the secondary unit 106 can be utilized to configure the image sensor 152 and the ISP 112. The secondary unit 106 can be implemented on an embedded processor such as the ARM M55 processor, or other scalar processors. The secondary unit 106 can include a secondary image sensor driver 154 that can communicate with the image sensor 152 and the various units of the circuit block 110. In particular, the secondary image sensor driver 154 can communicate with the ISP 112. As discussed in detail below, the secondary image sensor driver 154 can configure the image sensor 152 and the ISP 112 while the primary unit 104 initializes. The secondary image sensor driver 154 can be similar to other image sensor drivers, such as, for example the Camera2 driver included in the Android operating system. In some instances, the secondary' image sensor driver 154 can be implemented on the embedded processor including some or all the libraries needed for the operation of the driver. In this manner, an already available driver (such as the Camera2 driver) can be operated on the embedded processor without the need for the support of an entire operating system. In some instances, especially where the driver is open-source, the driver code can be readily modified to run on the embedded processor. Further, the open-source code can be modified to change the functionality’ of the driver, such as adding or removing some of the features of the driver. The secondary- image sensor driver 154 can be implemented in hardware and software. In some aspects, the secondary image sensor driver 154 or portions thereof can be implemented in firmware of the embedded processor(s) that embody / ies the secondary unit 106.

[0040] The memory 108 is a system memory, shared memory-, or both. In the example of Figure 1, memory 108 is depicted external to circuit block 11 . However, memory 108 canAtorney Docket No.: 56113-0819WO1

[0041] include portions of memory that are: i) specific to circuit block 110, ii) external to circuit block 110, or iii) both. The memory 106 can be random access memory of the SoC 102. such as static random-access memory (SRAM), dynamic random access memory (DRAM), a synchronous DRAM (SDRAM), or double data rate (DDR) SDRAM. In some implementations, aspects of memory' 108 are configured as a shared scratchpad memory that supports parallel access of its memory resources by two or more processors of the circuit block 110. The memory 108 can also include various other types of memory, such as high bandwidth memory' (HBM), narrow memory (e.g., for storing 8-bit values), wide memory (e.g., for storing 16-bit or 32-bit values), etc.

[0042] The circuit block 110 can include an image signal processor (ISP) 112, a tensor processing unit (TPU) 114, a digital signal processor (DSP) 116, and a graphics processing unit (GPU) 118. The circuit block 110 can also be referred to as an IP block 110, where the IP block can include one or more proprietary hardware elements. For example, each of the ISP 112, TPU 114, DSP 116, and GPU 118 can be a respective proprietary IP block (or IP device) of a particular entity or device manufacturer.

[0043] The ISP 112 can process the image data or image frame received from the image sensor 152 to generate an image frame. The ISP 112 can carry out one or more processing tasks such as black level adjustment, noise reduction, white balance, color filter array interpolation, RGB blending, gamma correction, RGB to Y CC conversion, edge enhancement, contrast enhancement, false chroma suppression, etc. In some other examples, the ISP 112 can be configured to merely' provide data to an external driver (such as the primary' image sensor driver 144 and the secondary image sensor driver 154) that can use algorithms to determine the parameters for focus, exposure and white balance correction, and provide values of those parameters to the ISP 112. In some instances, the ISP 112 can provide data or statistics such as histograms (which is a distribution of pixel intensities across the image), noise statistics, sharpness metrics, color statistics, etc. In some examples, the ISP 112 can generate these statistics for each area of the image and per channel (where one channel can correspond to one of the colors R, G, and B). The ISP 112 can also generate color statistics per row or column, where the color statistics can include average color value per row or column per channel. The ISP 112 can also generate the image frame in one or more image formats. For example, the ISP 112 can generate image frames in RGB, YUV, or RAW formats.

[0044] The image sensor 152 can include a digital charged-coupled-device (CCD) type or complementary' metal-oxide-semiconductor (CMOS) type image sensor. The image sensorAtorney Docket No.: 56113-0819WO1

[0045] can be an m x n sensor matrix that generates image data associated with the m x n sensors. For example, the image sensor can include 100s of thousands to 100s of millions of such sensors. The output of the image sensor can be an image frame or image data that can be converted (by, e.g., an image signal processor) into an image frame. In some examples, the image sensor 152 can include a front-end module that includes lenses, and mechanical and / or electronic shuters, and aperture control that control the amount of light that is incident on the sensors. In some examples, the focal length of the lenses, the shuter speed, and / or the aperture size (e.g., the f-stop) can be adjustable. The image sensor 152 can also provide adjustments to the sensors, such as the gain of the sensors. Adjusting the gain of the sensors can change, for example, the intensity of the images generated by the image sensor 152. In some instances, the adjustment of these parameters can be carried out by components external to the image sensor 152. For example, an image sensor driver can provide values of each of these parameters of the image sensor 152. In some instances, the image sensor 152 can access a register file to fetch the values of the parameters, where the register file can be writen into with values by the external components. In some instances, the register file can be local to the image sensor 152. In some other instances, the register file can be located in the memory 108 of the SoC 102.

[0046] The example computing system 100 can run one or more image processing applications. For example, the image processing application can be a photo application, a video call application, a video-conference application, etc. The image processing application can be run on the primary unit 104, which provides the operating system 142. Once a user requests (e.g., via the front-end interface provided on the device display) to launch the image processing application, the example computing system 100 can execute a series of actions to launch the image processing application.

[0047] Figure 2 shows a flow diagram depicting the process steps 200 taken by the example computing system 100 to launch an image processing application. The process steps can include receiving a request 202 to launch the image processing application from the user interface (UI). For example, if the example computing system 100 is a smartphone, the user can initiate the launch of the application by clicking on / selecting an application icon displayed on the display screen of the smartphone. If, for example, the example computing application is a laptop computer, the user can initiate the image processing application by double clicking an icon on the user interface of the computer. The user interface can inform the operating system 142 running on the CPU 140 that the user has requested a launch of the image processing application.Atorney Docket No.: 56113-0819WO1

[0048] Responsive to receiving the indication from the user interface that the user has requested to launch the image processing application, the operating system 142 can begin to allocate resources to the image processing application (204). For example, the operating system 142 can allocate memory' to the image processing application and can notify the applications’ main thread to initialize. The image processing application can then initialize in a user space provided by the operating system 142, load necessary libraries and prepare any user interface components. The image processing application can also call the image sensor 152 application programming interface (API) to access the image sensor hardware.

[0049] The operating system 142 can ensure that the image processing application has permission to access hardware resources such as the image sensor 152 (206). Generally, each application that is launched on the example computing system 100 can have associated access constraints that specify’ the computing resources that can be accessed by the application. The operating system 142 can utilize the access constraints associated with the image processing application and determine whether the image processing application can be provided access to the image sensor 152 hardware, the image signal processor 112, and the primary image sensor driver 144.

[0050] After ascertaining that the image processing application indeed has permission to access hardware resources, the operating system 142 can activate the primary image sensor driver 144 to initialize the image sensor 152 hardware (208). The image sensor driver typically runs in the kernel space and can wake up the image sensor module, apply appropriate power, and initiate the image sensor to capture frames. The primary image sensor driver 144 also can activate the ISP 112. At least one example of the primary’ image sensor driver 144 can include the Camera2 image sensor driver that provides a Camera2 API in the Android family of devices.

[0051] Once the primary’ image sensor driver 144 is activated, the primary image sensor driver 144 can begin to configure the image sensor and the ISP and provide frames for preview on the display screen (210). In some examples, the configuration of the image sensor 152 and the ISP 112 can include configuring the focus, exposure, gain, white balance, noise reduction, image stabilization, etc. of the image sensor 152 and / or the ISP 112. It should be noted that the configuration parameters listed here are not exhaustive and that different implementations may include different sets of configuration parameters. The primary image sensor driver 144 can utilize one or more algorithms to configure the image sensor 152 and the ISP 112. For example, the image sensor driver can utilize the “3 A” algorithm included in the Android operating system. The 3A algorithm can aid in adjustingAtorney Docket No.: 56113-0819WO1

[0052] the focus, the exposure compensation and the white balance of the image sensor. Adjusting the focus ensures sharpness of the images being captured. Adjusting exposure compensation can ensure that the brightness of the images captured are appropriate based on the ambient conditions, and adjusting the white balance can ensure that the color tones captured by the image sensor 152 are balanced for true-to-life images.

[0053] The primary image sensor driver 144 can rely on the images already captured by the image sensor 152 to adjust one or more configuration parameters. For example, the image sensor 152 can generate a first image frame Fl, which can be provided to the ISP 112. The ISP 112, in turn, can generate image statistics based on the image frame Fl and provide the image statistics to the primary image sensor driver 144. The primary image sensor driver 144, based on the image statistics, can generate updated values for one or more configuration parameters. The image sensor 152 can generate the next image frame F2 based on the updated configuration parameters. The ISP 112 can again generate statistics based on the image frame F2 and provide the statistics to the primary image sensor driver 144. The primary’ image sensor driver 144 can again adjust the configuration parameters of the image sensor 152 and the ISP 112 based on the statistics based on image frame F2. In this manner, the primary image sensor driver 144 can iteratively adjust the configuration parameters until their values converge to values that produce the threshold quality image frames.

[0054] While the primary image sensor driver 144 configures the image sensor and the ISP, the image processing application can request the image frames captured by the image sensor to be displayed on the display device of the example computing system 100. For example, the image processing application can utilize the API of the camera (e.g., Camera2 API or the Camera hardware abstraction layer (Camera HAL) in the Android OS) to configure the camera to display the captured image frames on the display screen of the device. For example, in the Camera2 API the user can use the “SurfaceView’’ class of APIs to present a live camera preview to the user on the display of the device. Thus, the image frames Fl, F2 and so on may be displayed to the user while the primary image sensor driver 144 adjusts the configuration parameters of the image sensor 152 and the ISP 112. Typically, the image sensor 152 and the ISP 112 will generate and process, respectively, the first image frame F 1 based on some preset or default values of the configuration parameters. In addition, a set of image frames after the first image frame may be generated while the primary image sensor driver 144 converges the values of the configuration parameters to the predetermined values. Therefore, it is quite likely that an initial set of image frames displayed to the user are of lowAtorney Docket No.: 56113-0819WO1

[0055] and / or changing quality until the values of the configuration parameters converge to those that result in the threshold quality of images.

[0056] In another example, while the image sensor driver 144 configures the image sensor and the ISP, the image sensor driver 144 may refrain from providing image frames for display on the display screen of the user device until the image frames reach a threshold level of image quality. In some examples, the image sensor driver 144 can measure one or more aspects of the image frames with a threshold value. In some instances, the image sensor deriver 144 can measure the average brightness of the image frame and display the image frame on the display screen only if the average brightness is above a threshold value. In some other instances, the image sensor driver 144 may compare an average sharpness of the image frame with a threshold value. In some other instances, other aspects of the image frame such as sharpness (by measuring e g., spatial frequency response) and contrast (by measuring gamma, or slope of a log tonal response curve) can be compared with their respective threshold values to determine the quality of the image frame. These and other measurable aspects of the image frame can be used separately or in combination to determine the image quality’ of the image frame. Thus, there may be a delay or latency with which the user will see the first image frames being displayed after launching the image processing application. For example, the image sensor driver 144 may not display image frames Fl to F4, and only display image frame F5, thereby resulting in a delay in displaying image frames on the display screen of the user device.

[0057] The discussion herein provides one or more approaches to reducing the latency in configuring the image sensor and the image signal processor, and thereby reducing the likelihood of displaying low-quality images (or additionally or alternatively, reducing the time to provision of relatively high quality images) to the user during the time that the primary image sensor driver 144 is determining the predetermined values of the configuration parameters. In particular, the approaches discussed herein provide a secondary' unit 106 that can begin configuring the image sensor 152 and the ISP 112 before the image frames are displayed to the user. In this manner, by the time the primary image sensor driver 144 is initialized, the secondary unit can carry out at least a portion of the process for configuring the image sensor 152 and the ISP 112. By carry ing out a portion of the process for configuring the image sensor 152 and the ISP 112 while the primary' image sensor driver 144 is being initialized, the latency of fully configuring the image sensor and the ISP 112 can be reduced. As a result, when the image frames are displayed to the user, the displayed imageAtorney Docket No.: 56113-0819WO1

[0058] frames can be of a higher quality than the image frames displayed when configuration is carried out solely by the primary unit 104.

[0059] Figure 3 depicts the operation of the primary unit 104 and the secondary unit 106 in configuring the image sensor 152 and the ISP 112. As discussed herein, the secondary' unit 106 can include a secondary image sensor driver 154, which can be similar to the primary' image sensor driver 144 in that the secondary image sensor driver 154 also can generate configuration parameters for the image sensor 152 and the ISP 112.

[0060] The secondary unit 106 can be configured to generate values for configuration parameters of the image sensor 152 and the ISP 112 and store those values in configuration parameters registers 302. The configuration parameters registers 302 can be a memory or register space on the system / shared memory 108 (Figure 1) and can be writen into by both the primary unit 104 and the secondary unit 106. That is, both the primary unit 104 and the secondary unit 106 can generate and set the values for the configuration parameters of the image sensor 152 and the ISP 112. The image sensor 152 and the ISP 112, in turn, can be configured to access the configuration parameters registers 302 to access the values for their respective configuration parameters. In some instances, the image sensor 152 and the ISP 112 can be configured to access the values from the configuration parameters registers 302 every clock cycle. In some other instances, the image sensor 152 and the ISP 112 can be configured to access values from the configuration parameters registers only if the values have changed. In such examples, control logic for the configuration parameters register 302 can detect a write operation into one or more registers of the configuration parameters register 302 and send an indication to the image sensor 152 or the ISP 112 that one or more values have changed. In some other instances, the primary unit 104 and the secondary unit 106 can be configured to communicate a signal to (or set a flag in memory) the image sensor 152 or the ISP 112 that the value of one or more configuration parameters has changed. The configuration parameters register 302 can include registers corresponding to one or more configuration parameters of the image sensor 152 and the ISP 112. For example, the registers can correspond to configuration parameters such as focus, exposure, white balance, gain etc. associated with the image sensor 152 and the ISP 112. In some examples, the configuration parameters registers 302 can be local to the image sensor 152 and the ISP 112. That is, the registers (or at least some registers) may be positioned in local memory' of the image sensor 152 or the ISP 112. In such instances, the primary unit 104 and the secondary unit 106 can write the values for the configuration parameters directly into the memory of the image sensor 152 or the ISP 112.Atorney Docket No.: 56113-0819WO1

[0061] Both the primary unit 104 and the secondary unit 106 can receive a trigger event 304 based on, e.g., the launch of an image processing application. For example, the trigger event 304 can indicate a request from an application for image-based data. Such an event can occur, for example, where an application is already running (e.g., on the primary unit 104), and the user selects or launches a feature of the application that requires image-based data. In some instances, the trigger event 304 can indicate the launch of the image processing application that can request image-based data. Here the image-based data can include data related to the images or image frames generated by the image sensor 152 or can include the image frames generated by the image sensor 152 and processed by the ISP 112. In some instances, the trigger event 304 can be generated by a user interface that provides the user the ability to launch the image processing application. For example, the user interface can be a graphical user interface that runs on the operating system 142 of the primary unit 104. In such instances, the trigger event 304 generated by the graphical user interface is provided to the secondary' unit 106 from the primary unit 104. In some other instances, the user interface can be a voice-based interface such those included in smart speakers, and the audio command received from the user to launch the image processing application can serve as a trigger event 304. Another example of a trigger event 304 can include hardware interrupt from a switch or buton (or an accelerometer) of the device. Such an interrupt can be generated by the device, for example, when the user would like to unlock the device based on face recognition. In such instances, the device is typically maintained in standby mode. The user then either presses or selects a buton or a series of butons or moves the device in a particular manner to cause the accelerometer of the device to generate a signal to indicate a request to unlock the device based on face recognition. In such instances, the hardware interrupt can be sent to both the CPU 140 and the secondary unit 106 (which can include an embedded processor or controller).

[0062] The primary unit 104, upon receiving the trigger event 304 can cause the operating system 142 to create an environment for the image processing application to launch. At least one aspect of this process has been discussed above in relation to Figure 2, where at 204. the operating system 142 can begin to allocate resources to the image processing application responsive to receiving an indication from the user interface that an image processing application launch request has been received. There can be scenarios where the CPU 140 can be in standby mode at the time of receiving the trigger event 304. One such scenario is mentioned herein where the user may request unlocking the device by using face recognition, and that the CPU 140 remains in a standby mode when locked. The CPU 140 can be inAtorney Docket No.: 56113-0819WO1

[0063] standby mode when at least a portion of the CPU 140 is powered down. Examples of portions of the CPU 140 that can be powered down during the standby state can include one or more memory modules, one or more processing cores, one or more graphical processing units, etc. In such instances, operations responsive to receiving the trigger event 304 can include additional steps of powering all the portions of the CPU 140 or at least those portions that are needed for the launching of the image processing application. In other scenarios, the CPU can be turned off when a trigger is received. For example, the user device may be configured to receive an input from the user that not only indicates that the user would like to switch the device on but also indicates that the user would like to launch an image processing application (e.g., a user pressing the power buton twice in quick succession). In such instances, the user interface can invoke the process of powering the CPU 140 and the secondary unit 106 and providing the trigger event 304.

[0064] The secondary unit 106, upon receiving the trigger event 304 can immediately begin configuring the image sensor 152 and the ISP 112. In instances where the image sensor 152 and / or the ISP 112 are powered down at the time of receiving the trigger event 304, the secondary unit, by way of the secondary image sensor driver 154, can power up the image sensor 152 and the ISP 112. In some examples, the secondary image sensor driver 154, as mentioned above, can be similar to the primary7image sensor driver 144 of the primary unit 104. In some other examples, the secondary7image sensor driver 154 can have fewer capabilities or features than the primary image sensor driver 144. For example, the secondary image sensor driver 154 may include only a subset of algorithms present in the primary image sensor driver 144 for configuring the image sensor 152 and the ISP 112. As mentioned herein, the primary image sensor driver 144 can include algorithms (e.g., the “3A” algorithms) that can be used to correct the exposure, focus, and white balance (among other features) of the image sensor 152 and the ISP 112. In such instances, the secondary image sensor driver 154 may include only a subset of the algorithms such as including only the exposure and white balance related algorithms.

[0065] Having the secondary image sensor driver 154 with relatively fewer capabilities than the primary image sensor driver 144 can reduce the memory requirement in the secondary unit 106. The reduced memory requirement may, in turn, reduce the power consumption of the secondary7unit 106. While a secondary7image sensor driver 154 with fewer capabilities may be not be able to fully configure the image sensor 152 and the ISP 112, it may at least partially configure the image sensor 152 and the ISP 112. Even partial configuration of the image sensor 152 and the ISP 112 can help reduce the latency of producing high qualityAtorney Docket No.: 56113-0819WO1

[0066] image frames. For example, the secondary' image sensor driver 154 can carry' out partial configuration of the image sensor 152 and the ISP 112 and then hand over to the primary¬ image sensor driver 144 to complete the configuration. But because the image sensor 152 and the ISP 112 are already partially configured before the primary image sensor driver 144 takes over, the additional time needed to fully configure the image sensor 152 and the ISP 112 can be reduced.

[0067] The manner in which the secondary image sensor driver 154 can configure the image sensor 152 and the ISP 112 can be similar to that discussed herein in relation to the primary image sensor driver 144. For example, the secondary' image sensor driver 154 can process image statistics related to one or more image frames generated by the image sensor 152 and the ISP 112 and accordingly adjust the values of one or more configuration parameters such that the threshold quality of image frames is generated. An image buffer 306 can store the image frames generated by the image sensor 152 and the ISP 112. The image buffer 306 can be a memory space in the shared memory 108 or can be a separate dedicated memory' for storing image frames. The image buffer 306 can be configured to store image frames as they are generated by the ISP 112. For example, as shown in Figure 3, the image frame buffer 306 stores the series of image frames Fl, F2, F3, F4, F5 . . . as they are generated by the ISP 112.

[0068] The configuration of the image sensor 152 and the ISP 112 can include configuring the focus, exposure, white balance of the image sensor, noise reduction, image stabilization, etc. It should be noted that the configuration parameters listed here are not exhaustive and that different implementations may include different sets of configuration parameters. In some instances, the set of configuration parameters for which the second image sensor driver 154 can determine values can be smaller than the set of configuration parameters for which the first image sensor driver 144 determines values. For example, while the first image sensor driver 144 may be capable of seting values for focus, exposure, and white balance, the second image sensor driver 154 may not be capable of seting the values for focus.

[0069] It should be noted that some of the dependencies that cause the primary unit 104 to take time to configure the image sensor 152 and the ISP 112 may not be applicable to the secondary unit 106. For example, the primary' image sensor driver 144 may have to deploy the whole suite of functionality provided by' the operating system to be initialized. In instances where the operating system is an Android operating system, the primary' image sensor driver 144 may have to deploy the Camera HAL (hardware abstraction layer) with an Android API. Therefore, Android itself would have to be initialized before the image sensor driver 144 can fully initialize. Further, the primary image sensor driver 144 may' be expectedAtorney Docket No.: 56113-0819WO1

[0070] to service image sensor APIs and applications when it is initialized, which means that the primary image sensor driver 144 may need to initialize all the components expected to be serviced by the API. This may also need initialization of other modules such as a graphics pipeline and memory buffers. The initialization of these components may have to take place before the image sensor driver 144 can initialize the image sensor 152 and the ISP 112— resulting in the delay. The secondary unit 106, on the other hand, can be configured for the purpose of initializing the image sensor 152 and the ISP 112. and not to service the Android API (or other APIs associated with other operating systems). Therefore, the secondary7unit 106, and in particular, the second image sensor driver 154 can be configured with less functionality than the primary image sensor driver 144 and in some examples, can be devoid of the camera API included in the primary image sensor driver 144.

[0071] The secondary image sensor driver 154 can utilize one or more algorithms to configure the image sensor 152 and the ISP 112. For example, the secondary image sensor driver 154 can utilize the “3 A"’ algorithm generally available on devices that run the Android operating system. The 3A algorithm can aid in adjusting the focus, the exposure compensation and the white balance of the image sensor. Adjusting the focus ensures sharpness of the images being captured. Adjusting exposure compensation can ensure that the brightness of the images captured are appropriate based on the ambient conditions, and adjusting the white balance can ensure that the color tones captured by the image sensor 152 are balanced for true-to-life images. In some instances, where the secondary image sensor driver 154 can determine values for fewer configuration parameters compared to those determined by the first image sensor driver 152, the second image sensor driver 154 can utilize a modified 3A algorithm. The modified 3A algorithm can be capable of adjusting only a subset of configuration parameters that can be adjusted using the 3 A algorithm. For example, the modified 3A algorithm can focus on one or two of the focus, exposure, and white balance capabilities of a full 3A algorithm. In some other instances, the modified 3A algorithm can use simplified versions of the 3A algorithms, such that the modified 3A algorithm is faster and smaller but not as accurate as the 3A algorithm. For example, the modified 3A algorithm may simply use an image brightness average to directly change exposure values.

[0072] The secondary image sensor driver 154 can rely on the images already captured by the image sensor 152 and processed by the ISP 112 to adjust one or more configuration parameters. For example, the image sensor 152 and the ISP 112 can generate the first image frame Fl. The ISP 112 can also generate image statistics based on the image frame Fl andAtorney Docket No.: 56113-0819WO1

[0073] provide the image statistics to the secondary image sensor driver 154. The secondary' image sensor driver 154, based on the image statistics received from the ISP 112, can generate updated values for the configuration parameters. The secondary image sensor driver 154 can load the generated values of the configuration parameters into the configuration parameter registers 302, which the image sensor 152 and the ISP 112 can access.

[0074] In some examples, the ISP 112 can generate statistics based on more than one image frame. For example, the ISP 112 can generate statistics based on image frame Fl and image frame F2. The ISP 112 can provide the statistics to the secondary' image sensor driver 154, which, in turn, can use the statistics to generate values for the configuration parameters.

[0075] In some examples, there may be a delay between the time the second image sensor driver 154 loads at least one new value in the configuration parameter registers 302 and the time when the changes take effect in the image sensor 152 and / or the ISP 112. For example, the secondary image sensor driver 154 may load new values in the configuration parameter registers 302 at the end of image frame F3. The new value may be, for example, for an exposure parameter of the image sensor 152. It may take the image sensor 152 some time to change its exposure value to the value loaded in the configuration parameter registers 302. This delay can be inherent to the image sensor 152. The delay can be long enough that by the time the image sensor 152 changes the exposure value, the next set of image frames has already been generated (but with the previous exposure value). For example, the delay can be more than the duration for generating the next two image frames. Which means that the new exposure value will take effect for the third image frame from the time the value was loaded into the configuration parameter registers 302.

[0076] The ISP 112 can continue to generate new statistics as new image frames are generated and provide these statistics to the secondary image frame driver 154. The secondary image sensor driver 154, in turn, can continue to generate new values of configuration parameters as new statistics are received from the ISP 112 and continue to update the configuration parameters registers 302 with the new values.

[0077] While the secondary image sensor driver 154 in the secondary’ unit 106 continues to update the values for the configuration parameters, the primary unit 104 can continue to initialize various components. But the secondary unit 106 is configured to adjust the configuration parameters of the image sensor 152 and the ISP 112 earlier (from the time the trigger event is received) that the primary unit 104 would have if operating alone. As a result, by the time the primary unit 104 initializes the primary camera driver 144 and the display, the image sensor 152 and the ISP 112 are already at least partially configured.Atorney Docket No.: 56113-0819WO1

[0078] Figure 4 shows a timing diagram 400 depicting the relative timings of the primary unit 104 and the secondary unit 106 in configuring the image sensor 152 and the ISP 112. In particular. Figure 4 shows a top portion associated with the initialization of the primary unit 104 and is similar to the timing depicted in Figure 2. The lower portion of Figure 4 shows the sequence of events in the secondary7unit 106 from the time a trigger event is generated to the time when values for the configuration parameters are generated. In the example shown in Figure 4, the trigger event corresponds to the request from the user to launch an image processing application. In the primary7unit 104, the triggering event causes the primary unit 104 to begin initializing the primary7unit 104, while in the secondary7unit 106, the trigger event can cause the secondary unit 106 to activate the image sensor 152 and the ISP 112 (402). Thus, while the primary unit 104 begins the process of waking up the CPU and / or the OS (if needed), allocating resources to the image processing application and checking permissions for access to hardware, and activate the primary image sensor driver 144 (the primary7unit may not have to activate the ISP 112 as it may already be activated by the secondary unit 106), the secondary7unit activates the image sensor 152 and the ISP 112 such that the sensor 152 and the ISP 112 can generate image frames that can be used to configure the image sensor 152 and the ISP 112.

[0079] At time tl, the image sensor 152 and the ISP 112 can begin generating image frames Fl, F2, and so on. The time tl at which the secondary7unit 106 causes the image sensor 152 and the ISP 112 to generate the first image frame is earlier than the time t2 at which the primary unit 104 would have caused the image sensor 152 and the ISP 112 to generate frames. During the duration tl to 12, the secondary unit 106 can carry' out generating values for the configuration parameters of the image sensor 152 and the ISP 112 (404) while the primary unit 104 initializes. The primary unit 104 is initialized when the primary image sensor driver 144 is activated. The primary image sensor driver 144, in turn, is activated when the primary image sensor driver 144 is in a state where it can control the image sensor 152 and / or the ISP 112.

[0080] Once the primary unit 104 is initialized, the image buffer can be accessible to the primary unit 104. The primary unit 104 can allow the image frames in the image buffer to be displayed to the user. In some instances, the latest generated image frame can be displayed to the user. For example, image frame F5 can be displayed to the user as it is the latest received image frame. However, the image frame F5 is generated after at least some configuration has been carried out by the secondary unit 106. As a result, the image quality of the image frame F5 is likely to be beter than the image quality' of the first image frame Fl.Atorney Docket No.: 56113-0819WO1

[0081] Thus, the user can see beter quality image frames when the image processing application launches and shows the live preview of the image frames captured by the image sensor 152.

[0082] In some examples, the duration between tl and t2 can be longer if the CPU 140 is in a standby state when the request for lunching the image processing application is received. As discussed above, one or more portions of the CPU 140 may be maintained in no or low power states with the trigger event is received. In such instances, the sequence of events for the primary unit 104 shown in Figure 4 can additionally include waking up or powering up the CPU 140 and loading the operating system 142 (if needed) (214). As a result, the time t2, when the primary unit 104 is ready to configure the image sensor 152 and the ISP 112 can be much later than that shown in Figure 4.

[0083] In some examples, the example computing system 100 can include an event queue where the secondary image sensor driver 154 can store the values of configuration parameters. For example, each time the secondary image sensor driver 154 calculates or determines the values of one or more configuration parameters, these values can be stored in the event queue (in addition to or instead of storing the values in the configuration parameter registers 302). In instances where the secondary image sensor driver 154 determines values of the configuration parameters every image frame, the event queue can include a series of values determined per image frame by the secondary image sensor driver 154. In some examples, the secondary image sensor driver 154 can also store statistics generated by the ISP 112 corresponding to the values determined for each image frame in the event queue. When the primary image sensor driver 144 is initialized or activated, the primary image sensor driver 144 can access the event queue for the values of the configuration parameters (and / or the statistics) already generated for the preceding image frames. In some examples, the primary image sensor driver 144 can select the values generated for the latest image frame and use these values as a basis for carrying out further configuration of the image sensor 152 and the ISP 112. In some other examples, the primary image sensor driver 144 can use values of more than one entry in the event queue as basis for carry ing out further configuration of the image sensor 152 and the ISP 112.

[0084] In some examples, when the primary unit 104 is initialized, the primary unit 104 can send a communication to the secondary unit 106 indicating that the primary' unit 104 has initialized. For example, referring to Figure 3, the primary' unit 104 can send a signal 308 to the secondary' unit 106, where receiving the signal 308 can indicate to the secondary unit 106 that the primary unit 104 has initialized. In some instances, the signal 308 can be an interrupt signal. In some other instances, the signal 308 can be in the form of a flag or bit(s) set in aAtorney Docket No.: 56113-0819WO1

[0085] register monitored by the secondary unit 106. Responsive to receiving the indication, the secondary unit 106 can stop generating values for the configuration parameters. For example, referring to Figure 4, the primary unit 104 can send a communication to the secondary unit 106 at time t2 (when the primary' unit 104 is initialized). Upon receiving the communication, the secondary unit can stop generating any values for the configuration parameters after time t2. After sending the communication to the secondary unit 106, the primary unit 104 can assume the responsibility of generating values for the configuration parameters of the image sensor 152 and the ISP 112.

[0086] In some examples, after the primary unit 104 assumes the responsibility of generating values for the configuration parameters, the primary unit 104. or more specifically the primary image sensor driver 144, can generate values of the configuration parameters and provide the values to the secondary image sensor driver 154. The secondary image frame driver, in turn, can write the values of the configuration parameters to the configuration parameter registers 302. In this manner, while the responsibility' for generating the values of the configuration parameters is shifted from the secondary unit 106 to the primary unit 104 after the activation of the primary unit 104, the task of updating the values of the configuration parameter registers 302 is retained by' the secondary unit 106. In some other examples, the primary unit 104 can also assume the responsibility of updating the values of the configuration parameter registers 302 from the secondary unit 106.

[0087] Figure 5 shows a flow diagram of an example process 500 for configuring an image sensor and an image signal processor. In particular, the process 500 can be executed by the example computing system 100 discussed herein. In some implementations, the process 500c can be specifically executed by the secondary unit 106. As such, the instructions associated with the process 500 can be stored in memory accessible to the secondary’ unit 106, where such memory' can include a memory local to the secondary unit 106 or the shared memory 108.

[0088] The process 500 includes receiving, at a secondary' unit, a trigger event indicating a request from an application for image-based data (502). At least one aspect of this processes state has been discussed herein. For example, as discussed above in relation to Figure 3, the secondary unit 106 can receive a trigger event 304 that indicates, for example, a request from a user to launch an image processing application. The image processing application can be an application that requests and uses image-based data generated by the example computing system 100.Atorney Docket No.: 56113-0819WO1

[0089] The process 500 also includes receiving, at the secondary unit from an image signal processor, image statistics of one or more image frames of a series of image frames, the image signal processor generating the series of image frames based on image data generated by the image sensor (504). As discussed herein in relation to Figures 3 and 4, the image signal processor 112 can generate image statistics based on one or more image frames of a series of image frames Fl, F2, F3, and so on. The ISP 112 can provide the image statistics to the secondary unit 106, and. in particular, to the secondary image sensor driver 154. The ISP 112 can generate the image statistics based on a single image frame or based on two or more image frames.

[0090] The process 500 also includes adjusting, by the secondary unit, values of configuration parameters for the image sensor and the image signal processor based on the image statistics (506). As discussed above in relation to Figures 2-4, the secondary unit can utilize algorithms for generating values of one or more configuration parameters of the image sensor and the image signal processor based on image statistics generated by the image signal processor. The secondary image sensor driver 154 can generate values for configuration parameters that can set the focus, exposure, white balance, etc. of the image sensor and the image signal processor. The secondary unit 106 adjusts the values of the configuration parameters based on the generated values. In some instances, the secondary unit 106 can adjust values of the configuration parameters that are stored in configuration parameter registers 302, which can be accessed by the image sensor 152 and the ISP 112. The secondary unit 106 can adjust the values of the configuration parameters of the image sensor 152 and the ISP 112 while the primary unit is still initializing.

[0091] The process 500 further includes receiving, at the secondary unit, an indication that the primary unit has initialized (508), and responsive to receiving the indication, providing, by the secondary unit to the primary unit, at least one of the values of the configuration parameters (510). As discussed herein in relation to Figures 3-4, the secondary’ unit can continue to generate and adjust the values of the configuration parameters of the image sensor 152 and the ISP 112 while the primary unit 104 is initialized. Once the primary unit 104 is initialized, the primary unit 104 can send an indication to the secondary' unit 106 that the primary unit is initialized. The secondary unit 106, upon receiving the indication, can provide the values of the configuration parameters to the primary' unit 104. The primary' unit 104 can take over responsibility from the secondary unit 106 for the generation and adjustment of the configuration parameters. The primary unit 104 can use the values of the configuration parameters provided by the secondary7unit 106 to update the configurationAtorney Docket No.: 56113-0819WO1

[0092] parameter registers 302. In some instances, the secondary unit 106 can cease generation and adjustment of the configuration parameters after receiving the indication that the primary unit 104 has initialized. As the secondary unit 106 carries out at least some configuration of the image sensor 152 and the ISP 112 prior to the primary' unit 104 initializing. Thus, by the time the primary' unit 104 initializes and the image processing application displays image frames to the user, the image frames are of relatively higher quality than when the primary unit 104 alone begins to adjust the configuration parameters after initialization. Thus, the latency in displaying higher quality7image frames to the user is reduced.

[0093] In some examples, the secondary unit 106 may be maintained in an always ON state. For instance, when the primary' unit 104 is in a standby or sleep mode, at least a portion of or the entirety of the primary unit 104 is powered down. In such instances, the secondary unit 106 can be maintained fully powered so that the secondary unit 106 can quickly respond to the trigger event and begin configuring the image sensor 152 and the image signal processor 112.

[0094] The components and processes discussed herein can be implemented on a computing system. In particular, a computing system including a computing device and / or a mobile computing device can be used to implement the techniques described herein. For example, one or more processes, electronic design tools, and data can be implemented on or stored in the computing device or the mobile computing device.

[0095] The computing device is intended to represent various forms of digital computers, such as laptops, desktops, workstations, personal digital assistants, servers, blade servers, mainframes, and other appropriate computers. The mobile computing device is intended to represent various forms of mobile devices, such as personal digital assistants, cellular telephones, smart-phones, mobile embedded radio systems, radio diagnostic computing devices, and other similar computing devices. The components shown here, their connections and relationships, and their functions, are meant to be examples only, and are not meant to be limiting.

[0096] The computing device includes a processor, a memory’, a storage device, a high-speed interface connecting to the memory and multiple high-speed expansion ports, and a low-speed interface connecting to a low-speed expansion port and the storage device. Each of the processor, the memory7, the storage device, the high-speed interface, the high-speed expansion ports, and the low-speed interface, are interconnected using various busses, and may be mounted on a common motherboard or in other manners as appropriate. The processor can process instructions for execution within the computing device, includingAtorney Docket No.: 56113-0819WO1

[0097] instructions stored in the memory or on the storage device to display graphical information for a GUI on an external input / output device, such as a display coupled to the high-speed interface. In other implementations, multiple processors and / or multiple buses may be used, as appropriate, along with multiple memories and types of memory. In addition, multiple computing devices may be connected, with each device providing portions of the operations (e.g., as a server bank, a group of blade servers, or a multi-processor system). In some implementations, the processor is a single threaded processor. In some implementations, the processor is a multi-threaded processor. In some implementations, the processor is a quantum computer.

[0098] The memory stores information within the computing device. In some implementations, the memory is a volatile memory unit or units. In some implementations, the memory is a non-volatile memory unit or units. The memory may also be another form of computer-readable medium, such as a magnetic or optical disk.

[0099] The storage device is capable of providing mass storage for the computing device. In some implementations, the storage device may be or include a computer-readable medium, such as a floppy disk device, a hard disk device, an optical disk device, or a tape device, a flash memory or other similar solid-state memory device, or an array of devices, including devices in a storage area network or other configurations. Instructions can be stored in an information carrier. The instructions, when executed by one or more processing devices (for example, processor), perform one or more methods, such as those described above. The instructions can also be stored by one or more storage devices such as computer- or machine-readable mediums (for example, the memory, the storage device, or memory on the processor). The high-speed interface manages bandwidth-intensive operations for the computing device, while the low-speed interface manages lower bandwidth-intensive operations. Such allocation of functions is an example only. In some implementations, the high-speed interface is coupled to the memory7, the display (e.g., through a graphics processor or accelerator), and to the high-speed expansion ports, which may accept various expansion cards (not shown). In the implementation, the low-speed interface is coupled to the storage device and the low-speed expansion port. The low-speed expansion port, which may include various communication ports (e.g., USB, Bluetooth, Ethernet, wireless Ethernet) may be coupled to one or more input / output devices, such as a keyboard, a pointing device, a scanner, or a networking device such as a switch or router, e.g., through a network adapter.

[0100] The computing device may be implemented in a number of different forms, as shown in the figure. For example, it may be implemented as a standard server, or multiple times in aAtorney Docket No.: 56113-0819WO1

[0101] group of such servers. In addition, it may be implemented in a personal computer such as a laptop computer. It may also be implemented as part of a rack server system. Alternatively, components from the computing device may be combined with other components in a mobile device, such as a mobile computing device. Each of such devices may include one or more of the computing device and the mobile computing device, and an entire system may be made up of multiple computing devices communicating with each other.

[0102] The mobile computing device includes a processor, a memory. an input / output device such as a display, a communication interface, and a transceiver, among other components. The mobile computing device may also be provided with a storage device, such as a microdrive or other device, to provide additional storage. Each of the processor, the memory, the display, the communication interface, and the transceiver, are interconnected using various buses, and several of the components may be mounted on a common motherboard or in other manners as appropriate.

[0103] The processor can execute instructions within the mobile computing device, including instructions stored in the memory. The processor may be implemented as a chipset of chips that include separate and multiple analog and digital processors. The processor may provide, for example, for coordination of the other components of the mobile computing device , such as control of user interfaces, applications run by the mobile computing device , and wireless communication by the mobile computing device .

[0104] The processor may communicate with a user through a control interface and a display interface coupled to the display. The display may be, for example, a TFT (Thin-Film-Transistor Liquid Cr stal Display) display or an OLED (Organic Light Emiting Diode) display, or other appropriate display technology. The display interface may include appropriate circuitry for driving the display to present graphical and other information to a user. The control interface may receive commands from a user and convert them for submission to the processor. In addition, an external interface may provide communication with the processor, so as to enable near area communication of the mobile computing device with other devices. The external interface may provide, for example, for wired communication in some implementations, or for wireless communication in other implementations, and multiple interfaces may also be used.

[0105] The memory stores information within the mobile computing device. The memory can be implemented as one or more of a computer-readable medium or media, a volatile memory unit or units, or a non-volatile memory unit or units. An expansion memory may also be provided and connected to the mobile computing device through an expansionAtorney Docket No.: 56113-0819WO1

[0106] interface, which may include, for example, a SIMM (Single In Line Memory Module) card interface. The expansion memory may provide extra storage space for the mobile computing device, or may also store applications or other information for the mobile computing device. Specifically, the expansion memory may include instructions to cany' out or supplement the processes described herein and may include secure information also. Thus, for example, the expansion memory may be provided as a security module for the mobile computing device, and may be programmed with instructions that permit secure use of the mobile computing device. In addition, secure applications may be provided via the SIMM cards, along with additional information, such as placing identifying information on the SIMM card in a non-hackable manner.

[0107] The memory may include, for example, flash memory and / or NVRAM memory (nonvolatile random access memory), as discussed below. In some implementations, instructions are stored in an information carrier such that the instructions, when executed by one or more processing devices (for example, processor), perform one or more methods, such as those described above. The instructions can also be stored by one or more storage devices, such as one or more computer- or machine-readable mediums (for example, the memory, the expansion memory, or memory on the processor). In some implementations, the instructions can be received in a propagated signal, for example, over the transceiver or the external interface.

[0108] The mobile computing device may communicate wirelessly through the communication interface, which may include digital signal processing circuitry' in some cases. The communication interface may provide for communications under various modes or protocols, such as GSM voice calls (Global System for Mobile communications), SMS (Short Message Service), EMS (Enhanced Messaging Service), or MMS messaging (Multimedia Messaging Service), CDMA (code division multiple access), TDMA (time division multiple access), PDC (Personal Digital Cellular), WCDMA (Wideband Code Division Multiple Access), CDMA2000, or GPRS (General Packet Radio Service), LTE, 4G / 5G / 6G cellular, among others. Such communication may occur, for example, through the transceiver using a radio frequency. In addition, short-range communication may occur, such as using a Bluetooth, Wi-Fi, or other such transceiver (not shown). In addition, a GPS (Global Positioning System) receiver module may provide additional navigation- and location-related wireless data to the mobile computing device, which may be used as appropriate by applications running on the mobile computing device.Atorney Docket No.: 56113-0819WO1

[0109] The mobile computing device may also communicate audibly using an audio codec, which may receive spoken information from a user and convert it to usable digital information. The audio codec may likewise generate audible sound for a user, such as through a speaker, e.g., in a handset of the mobile computing device. Such sound may include sound from voice telephone calls, may include recorded sound (e g., voice messages, music files, among others) and may also include sound generated by applications operating on the mobile computing device.

[0110] The mobile computing device may be implemented in a number of different forms, as shown in the figure. For example, it may be implemented as a cellular telephone. It may also be implemented as part of a smart-phone, personal digital assistant, or other similar mobile device.

[0111] Embodiments of the subject mater and the functional operations described in this specification can be implemented in digital electronic circuitry, in tangibly-embodied computer softw are or firmware, in computer hardware, including the structures disclosed in this specification and their structural equivalents, or in combinations of one or more of them. Embodiments of the subject matter described in this specification can be implemented as one or more computer programs, i.e., one or more modules of computer program instructions encoded on a tangible non-transitory storage medium for execution by, or to control the operation of, data processing apparatus. The computer storage medium can be a machine-readable storage device, a machine-readable storage substrate, a random or serial access memory device, or a combination of one or more of them. Alternatively or in addition, the program instructions can be encoded on an artificially-generated propagated signal, e.g., a machine-generated electrical, optical, or electromagnetic signal, that is generated to encode information for transmission to suitable receiver apparatus for execution by a data processing apparatus.

[0112] The term “data processing apparatus” refers to data processing hardw are and encompasses all kinds of apparatus, devices, and machines for processing data, including by¬ way of example a programmable processor, a computer, or multiple processors or computers. The apparatus can also be. or further include, special purpose logic circuitry, e.g., an FPGA (field programmable gate array) or an ASIC (application specific integrated circuit). The apparatus can optionally include, in addition to hardware, code that creates an execution environment for computer programs, e.g., code that constitutes processor firmware, a protocol stack, a database management system, an operating system, or a combination of one or more of them. -Atorney Docket No.: 56113-0819WO1

[0113] A computer program which may also be referred to or described as a program, software, a software application, an app. a module, a software module, a script, or code) can be written in any form of programming language, including compiled or interpreted languages, or declarative or procedural languages, and it can be deployed in any form, including as a standalone program or as a module, component, subroutine, or other unit suitable for use in a computing environment. A program may, but need not, correspond to a file in a file system. A program can be stored in a portion of a file that holds other programs or data, e.g., one or more scripts stored in a markup language document, in a single file dedicated to the program in question, or in multiple coordinated files, e.g., files that store one or more modules, sub-programs, or portions of code. A computer program can be deployed to be executed on one computer or on multiple computers that are located at one site or distributed across multiple sites and interconnected by a data communication network.

[0114] The processes and logic flows can also be performed by special purpose logic circuitry7, e.g., an FPGA or an ASIC, or by a combination of special purpose logic circuitry and one or more programmed computers. Computers suitable for the execution of a computer program can be based on general or special purpose microprocessors or both, or any other kind of central processing unit. Generally, a central processing unit will receive instructions and data from a read only memory7or a random access memory7or both. The essential elements of a computer are a central processing unit for performing or executing instructions and one or more memory devices for storing instructions and data. The central processing unit and the memory can be supplemented by, or incorporated in, special purpose logic circuitry. Generally, a computer will also include, or be operatively coupled to receive data from or transfer data to, or both, one or more mass storage devices for storing data, e.g.. magnetic, magneto-optical disks, or optical disks. However, a computer need not have such devices. Moreover, a computer can be embedded in another device, e.g., a mobile telephone, a personal digital assistant (PDA), a mobile audio or video player, a game console, a Global Positioning System (GPS) receiver, or a portable storage device, e.g., a universal serial bus (USB) flash drive, to name just a few.

[0115] Computer readable media suitable for storing computer program instructions and data include all forms of non-volatile memory, media and memory7devices, including by way' of example semiconductor memory7devices, e.g., EPROM, EEPROM, and flash memory7devices; magnetic disks, e.g., internal hard disks or removable disks; magneto-optical disks; and CD-ROM and DVD-ROM disks. -While this specification contains many specific implementation details, these should not be construed as limitations on the scope of anyAtorney Docket No.: 56113-0819WO1

[0116] invention or on the scope of what may be claimed, but rather as descriptions of features that may be specific to particular embodiments of particular inventions. Certain features that are described in this specification in the context of separate embodiments can also be implemented in combination in a single embodiment. Conversely, various features that are described in the context of a single embodiment can also be implemented in multiple embodiments separately or in any suitable subcombination. Moreover, although features may be described above as acting in certain combinations and even initially be claimed as such, one or more features from a claimed combination can in some cases be excised from the combination, and the claimed combination may be directed to a subcombination or variation of a subcombination.

[0117] Similarly, while operations are depicted in the drawings and recited in the claims in a particular order, this should not be understood as requiring that such operations be performed in the particular order shown or in sequential order, or that all illustrated operations be performed, to achieve desirable results. In certain circumstances, multitasking and parallel processing may be advantageous. Moreover, the separation of various system modules and components in the embodiments described above should not be understood as requiring such separation in all embodiments, and it should be understood that the described program components and systems can generally be integrated together in a single software product or packaged into multiple software products.

[0118] Particular embodiments of the subject mater have been described. Other embodiments are within the scope of the following claims. For example, the actions recited in the claims can be performed in a different order and still achieve desirable results. As one example, the processes depicted in the accompanying figures do not necessarily require the particular order shown, or sequential order, to achieve desirable results. In some cases, multitasking and parallel processing may be advantageous.

[0119] What is claimed is:

Claims

Atorney Docket No.: 56113-0819WO1CLAIMS1. A method for configuring an image sensor and an image signal processor of a device comprising a primary unit including a primary’ image sensor driver and a secondary unit including a secondary image sensor driver, comprising:receiving, at a secondary unit, a trigger event indicating a request from an application for image data.receiving, at the secondary unit from the image signal processor, image statistics of one or more image frames of a series of image frames, the image signal processor generating the series of image frames based on output of the image sensor;adjusting, by the secondary' unit, values of configuration parameters for the image sensor and the image signal processor based on the image statistics;receiving, at the secondary unit, an indication that the primary’ unit has initialized; and responsive to receiving the indication, providing, by the secondary unit, at least one of the values of the configuration parameters to the primary unit.

2. The method of claim 1, further comprising: iteratively adjusting, by the secondary unit, values of configuration parameters for the image sensor and the image signal processor based on the image statistics of multiple image frames of the series of image frames.

3. The method of claim 1, further comprising:storing, the series of image frames in an image buffer, wherein the image buffer is accessible by the primary' unit when the primary’ unit is initialized.

4. The method of claim 1, wherein the secondary unit receives the image statistics from the image signal processor when the primary unit is in a standby state or being initialized.

5. The method of claim 1, further comprising:generating, by the primary image sensor driver after the primary’ unit has initialized, additional values of the configuration parameters based on a subset of the series of image frames; andconfiguring the image sensor and the image signal processor based on the additional values of the configuration parameters.

6. The method of claim 5, further comprising:Atorney Docket No.: 56113-0819WO1receiving, at the secondary image sensor driver form the primary image sensor driver, the additional values of the configuration parameters; andconfiguring, by the secondary image sensor driver, the image sensor and the image signal processor based on the additional values of the configuration parameters.

7. The method of claim 1, wherein the secondary image sensor driver includes a secondary algorithm, the method comprising:generating, by the secondary algorithm, the values of the configuration parameters based on the image statistics.

8. The method of claim 1, wherein the primary image sensor driver includes more image processing functions than the secondary image sensor driver.

9. The method of claim 1, further comprising:responsive to receiving the indication that the primary unit has initialized, at the secondary unit, ceasing generating values of configuration parameters for the image sensor and the image signal processor.

10. The method of claim 9, further comprising:subsequent to ceasing generating values for the configuration parameters, receiving, at the secondary unit from the primary unit, additional values for the configuration parameters; andwriting, by the secondary unit, the additional values to a configuration parameter register.

11. The method of claim 1, wherein the primary unit is an application processor and the secondary unit is an embedded processor.

12. The method of claim 1, further comprising:responsive to receiving the indication that the primary unit has initialized, displaying subsequently generated image frames on a display screen.

13. One or more non -transitory computer-readable storage media storing instructions that when executed by one or more processors cause the one or more processors to performAtorney Docket No.: 56113-0819WO1operations for configuring an image sensor and an image signal processor of a device comprising a primary unit including a primary image sensor driver and a secondary unit including a secondary image sensor driver, the operations comprising:receiving, at a secondary unit, a trigger event indicating a request from an application for image data,receiving, at the secondary unit from the image signal processor, image statistics of one or more image frames of a series of image frames, the image signal processor generating the series of image frames based on output of the image sensor;adjusting, by the secondary' unit, values of configuration parameters for the image sensor and the image signal processor based on the image statistics;receiving, at the secondary unit, an indication that the primary unit has initialized; and responsive to receiving the indication, providing, by the secondary unit, at least one of the values of the configuration parameters to the primary unit.

14. The computer-readable storage media of claim 13, the operations further comprising:iteratively adjusting, by the secondary unit, values of configuration parameters for the image sensor and the image signal processor based on the image statistics of multiple image frames of the series of image frames.

15. The computer-readable storage media of claim 13, the operations further comprising:storing, the series of image frames in an image buffer, wherein the image buffer is accessible by the primary' unit when the primary' unit is initialized.

16. The computer-readable storage media of claim 13, wherein the secondary unit receives the image statistics from the image signal processor when the primary unit is in a standby state or being initialized.

17. The computer-readable storage media of claim 13, the operations further comprising:generating, by the primary image sensor driver after the primary' unit has initialized, additional values of the configuration parameters based on a subset of the series of image frames; andconfiguring the image sensor and the image signal processor based on the additional values of the configuration parameters.Atorney Docket No.: 56113-0819WO118. The computer-readable storage media of claim 17, the operations further comprising:receiving, at the secondary image sensor driver form the primary image sensor driver, the additional values of the configuration parameters; andconfiguring, by the secondary image sensor driver, the image sensor and the image signal processor based on the additional values of the configuration parameters.

19. The computer-readable storage media of claim 13, wherein the secondary image sensor driver includes a secondary algorithm, the operations comprising:generating, by the secondary' algorithm, the values of the configuration parameters based on the image statistics.

20. The computer-readable storage media of claim 13, wherein the primary image sensor driver includes more image processing functions than the secondary image sensor driver.

21. The computer-readable storage media of claim 13, the operations further comprising:responsive to receiving the indication that the primary unit has initialized, at the secondary unit, ceasing generating values of configuration parameters for the image sensor and the image signal processor.

22. The computer-readable storage media of claim 21, the operations further comprising:subsequent to ceasing generating values for the configuration parameters, receiving, at the secondary' unit from the primary' unit, additional values for the configuration parameters; andwriting, by the secondary unit, the additional values to a configuration parameter register.

23. The computer-readable storage media of claim 13, wherein the primary unit is an application processor and the secondary unit is an embedded processor.

24. The computer-readable storage media of claim 13, the operations further comprising:responsive to receiving the indication that the primary' unit has initialized, displaying subsequently generated image frames on a display screen.

25. A system, comprising:Atorney Docket No.: 56113-0819WO1one or more processors; andone or more storage devices storing instructions that when executed by the one or more processors to perform operations for configuring an image sensor and an image signal processor of a device comprising a primary unit including a primary image sensor driver and a secondary unit including a secondary image sensor driver, the operations comprising:receiving, at a secondary unit, a trigger event indicating a request from an application for image data,receiving, at the secondary7unit from the image signal processor, image statistics of one or more image frames of a series of image frames, the image signal processor generating the series of image frames based on output of the image sensor;adjusting, by the secondary unit, values of configuration parameters for the image sensor and the image signal processor based on the image statistics;receiving, at the secondary unit, an indication that the primary unit has initialized; and responsive to receiving the indication, providing, by the secondary unit, at least one of the values of the configuration parameters to the primary unit.

26. The system of claim 25, the operations further comprising:iteratively adj usting, by the secondary unit, values of configuration parameters for the image sensor and the image signal processor based on the image statistics of multiple image frames of the series of image frames.

27. The system of claim 25, the operations further comprising:storing, the series of image frames in an image buffer, wherein the image buffer is accessible by the primary unit when the primary unit is initialized.

28. The system of claim 25, wherein the secondary unit receives the image statistics from the image signal processor when the primary7unit is in a standby state or being initialized.

29. The system of claim 25, the operations further comprising:generating, by the primary7image sensor driver after the primary7unit has initialized, additional values of the configuration parameters based on a subset of the series of image frames; andAtorney Docket No.: 56113-0819WO1configuring the image sensor and the image signal processor based on the additional values of the configuration parameters.

30. The system of claim 29, the operations further comprising:receiving, at the secondary image sensor driver form the primary7image sensor driver, the additional values of the configuration parameters; andconfiguring, by the secondary image sensor driver, the image sensor and the image signal processor based on the additional values of the configuration parameters.

31. The system of claim 25, wherein the secondary7image sensor driver includes a secondary algorithm, the operations comprising:generating, by the secondary algorithm, the values of the configuration parameters based on the image statistics.

32. The system of claim 25, wherein the primary7image sensor driver includes more image processing functions than the secondary image sensor driver.

33. The system of claim 25, the operations further comprising:responsive to receiving the indication that the primary unit has initialized, at the secondary unit, ceasing generating values of configuration parameters for the image sensor and the image signal processor.

34. The system of claim 33, the operations further comprising:subsequent to ceasing generating values for the configuration parameters, receiving, at the secondary unit from the primary unit, additional values for the configuration parameters; andwriting, by the secondary unit, the additional values to a configuration parameter register.

35. The system of claim 25, wherein the primary unit is an application processor and the secondary7unit is an embedded processor.

36. The system of claim 25, the operations further comprising:Attorney Docket No.: 56113-0819WO1responsive to receiving the indication that the primary unit has initialized, displaying subsequently generated image frames on a display screen.

Images