Photographic cache management method, and chip system, device and storage medium

Abstract

Provided in the present application are a photographic cache management method, and a chip system, a device and a storage medium. The method comprises: receiving a first operation on a camera application; then, on the basis of an algorithm for a first image frame, determining a cache management mode for the first image frame; and on the basis of an algorithm for a second image frame, determining a cache management mode for the second image frame, wherein when the algorithm for the first image frame is different from the algorithm for the second image frame, the first cache management mode is different from the second cache management mode. Meeting the requirement of each frame for a sufficient cache avoids a temporary memory shortage caused by an algorithm change during photographing to affect the entire photographic process, and also reduces the application of unnecessary caches.

Description

A method for managing camera buffers, a chip system, a device, and a storage medium.

[0001] This application claims priority to Chinese Patent Application No. 202411538932.1, filed with the State Intellectual Property Office of China on October 30, 2024, entitled "A Method for Managing Camera Cache, Chip System, Device and Storage Medium", the entire contents of which are incorporated herein by reference. Technical Field

[0002] This application relates to the field of terminal technology, and in particular to a shooting cache management method, chip system, device and storage medium. Background Technology

[0003] Smartphones, tablets, and other electronic devices typically offer camera functions. These cameras allow users to capture images and videos. Furthermore, camera functions usually include various shooting modes, such as portrait, professional, and night scene modes. Smartphones also employ various image processing algorithms to process the raw images, such as beautification, blurring, and filters. Different algorithms have different caching requirements and processing logic. For example, when entering portrait mode, the device can process each frame of the image using beautification algorithms based on the user's selected beautification settings. However, beautification algorithms need to retain the original input data for other operations (such as merging), thus requiring a new cache to be allocated.

[0004] In the existing technology, during camera use, the buffer management mode is determined when the electronic device distributes the data, and the same buffer management mode is used for the processing of each frame, which results in the inability to adapt to the actual algorithm's buffer requirements for each frame. Summary of the Invention

[0005] Firstly, this application provides a shooting cache management method. This method includes receiving a first operation on a camera application, wherein the first operation includes a user's operation on the camera application icon 1001 on the desktop 100, which is used to launch the camera application and configure a camera stream. Alternatively, the first operation includes a user's operation to switch between different shooting modes. As shown in Figures 1A(b) to 1B(a), when the user switches the camera application from photo mode to portrait mode, after receiving the first operation on the camera application, the camera application calls the configure_stream interface function to configure the stream and sends the configuration information to the hardware abstraction layer. The camera hardware abstraction layer receives a load request through the process_capture_request (PCR) interface function, calls the camera device driver in the kernel layer, and the camera device driver's image sensor acquires images, continuously transmitting the first image stream back to the hardware abstraction layer through the shooting path. The first image stream includes a first image frame and a second image frame.

[0006] The buffer management mode for the first image frame is determined based on its algorithm, and similarly, the buffer management mode for the second image frame is determined based on its algorithm. The buffer management modes for the first and second image frames can be either inplace or outplace. When the algorithms for the first and second image frames are different, the first buffer management mode differs from the second buffer management mode. For example, if the algorithm category of the first image frame requires retaining the input image data for other operations, such as a beautification algorithm, the first buffer management mode is outplace, allocating both an input buffer and an output buffer. If the algorithm category of the second image frame only retains the output image data after processing, such as a watermarking algorithm, then the algorithm for the second image frame is inplace, in which only one output buffer needs to be allocated.

[0007] Therefore, this solution can output the cache management mode based on the algorithm of the current frame. Different cache management modes correspond to different algorithm categories for the first and second image frames. This satisfies the sufficient cache requirements of each frame, avoids temporary memory shortages caused by algorithm changes during the shooting process, thus avoiding affecting the entire shooting process, and reduces unnecessary cache allocation.

[0008] In conjunction with the first aspect, in one possible implementation, an additional TBM manager is added to the existing TBM manager, and different TBM managers are distinguished by flag bits. For example, three TBM managers are set: a TBM-bypass manager, a TBM-inplace manager, and a TBM-outplace manager. Each of the three TBM managers corresponds to a different buffer management mode. The TBM-bypass manager performs buffer management in bypass mode, the TBM-inplace manager performs buffer management in inplace mode, and the TBM-outplace manager adopts outplace buffer management mode for the first image frame, allocating two buffers (input buffer and output buffer) and performing a copy operation to copy the data from the input buffer to the output buffer. The TBM-outplace manager performs buffer management in outplace mode. The TBM-inplace manager adopts inplace buffer management mode for the second image frame, requiring only one output buffer.

[0009] Therefore, this solution can achieve frame-by-frame cache management by scheduling three TBM managers using the TBM Management Pool, even though the current TBM object can only support one mode.

[0010] In conjunction with the first aspect, in one possible implementation, after receiving the user's first operation, during the streaming phase, the electronic device configures a tiny stream, a preview stream, and a capture stream. The tiny stream is an image stream generated during capture or preview scenarios, where the image data captured by the image sensor is downsampled (regardless of whether it's a capture or preview). For each frame captured by the image sensor, a decision module determines whether to downsample to form a tiny stream; therefore, the decision module can access the configuration file when deciding on the tiny stream. The configuration file can be an XML file named "Feature XML". The configuration file stores the algorithms to be used for different shooting information such as HDR, lighting conditions, low light, and shooting modes.

[0011] Based on the image information of the first image frame in the first image stream, and in conjunction with the configuration file, determine the corresponding feature A for the image information of the first image frame. Algorithm A, which needs to be used under feature A, is then applied to process the first image frame.

[0012] Therefore, since the tiny stream is processed earlier in the image data capture process, the image capture information is obtained by accessing the configuration file in a timely manner when deciding on the tiny stream. Combined with the configuration file, the algorithm for each frame can be quickly determined, thus improving processing efficiency.

[0013] In conjunction with the first aspect, in one possible implementation, when the third image frame of the first image stream does not require processing by the algorithm, the third image frame is in bypass mode and no cache is configured for the third image frame, which reduces unnecessary cache allocation and copying, thereby further optimizing performance and power consumption.

[0014] In conjunction with the first aspect, in one possible implementation, the chip manufacturer's platform can also set a node flag during the traffic allocation phase to identify whether the node's cache management mode is bypass mode, inplace mode, or outplace mode. Optionally, the node's cache management mode can be identified as outplace mode. Setting the node flag to an inactive state prevents the electronic device from managing the cache according to the node flag.

[0015] Therefore, this application enables the chip manufacturer's platform to pre-configure a buffer management mode during the distribution phase, thereby preventing the original buffer management mode strategy of the chip manufacturer's platform from being executed during the distribution phase, thus avoiding the impact on the electronic device when executing the buffer configuration for each frame.

[0016] In conjunction with the first aspect, in one possible implementation, more than one TBM can be set up for a cache management mode to address the situation of insufficient TBM manager scheduling.

[0017] In a second aspect, this application provides a chip system applied to an electronic device including a display screen and a memory; the chip system includes one or more interface circuits and one or more processors; the interface circuits and processors are interconnected via lines; the interface circuits are used to receive signals from the memory of the electronic device and send signals to the processor, the signals including computer instructions stored in the memory; when the processor executes the computer instructions, the electronic device performs a method as described in the first aspect and any of its possible design embodiments.

[0018] Thirdly, this application also provides an electronic device including a display screen, a memory, and one or more processors. The display screen, memory, and processors are coupled. The memory stores computer program code, which includes computer instructions that, when executed by the processor, cause the electronic device to perform the methods described in the first aspect and any of its possible designs.

[0019] Fourthly, this application provides a computer storage medium including computer instructions that, when executed on an electronic device, cause the electronic device to perform a method as described in the first aspect and any possible design thereof.

[0020] Understandably, the beneficial effects of the chip system of the second aspect, the electronic device of the third aspect, and the computer storage medium of the fourth aspect provided above can be referred to the beneficial effects of the first aspect and any of its possible design methods, which will not be repeated here. Attached Figure Description

[0021] Figure 1A is a schematic diagram of a camera interface provided in an embodiment of this application;

[0022] Figure 1B is a schematic diagram of a camera interface provided in an embodiment of this application;

[0023] Figure 2 is a schematic diagram of the software structure of an electronic device provided in an embodiment of this application;

[0024] Figure 3 is a schematic diagram of the hardware structure of an electronic device provided in an embodiment of this application;

[0025] Figure 4 is a schematic diagram of a target cache manager for an electronic device provided in an embodiment of this application;

[0026] Figure 5 is a flowchart illustrating a shooting cache management method provided in an embodiment of this application. Detailed Implementation

[0027] The technical solutions of the embodiments of this application are described below with reference to the accompanying drawings. In the description of the embodiments of this application, the terminology used in the following embodiments is for the purpose of describing specific embodiments only and is not intended to limit the application. As used in the specification and appended claims of this application, the singular expressions "a," "the," "the," "the," and "this" are intended to also include expressions such as "one or more," unless the context clearly indicates otherwise. It should also be understood that in the following embodiments of this application, "at least one" and "one or more" refer to one or more (including two). The term "and / or" is used to describe the relationship between related objects, indicating that three relationships can exist; for example, A and / or B can represent: A alone, A and B simultaneously, or B alone, where A and B can be singular or plural. The character " / " generally indicates that the preceding and following related objects are in an "or" relationship.

[0028] References to "one embodiment" or "some embodiments" in this specification mean that one or more embodiments of this application include a specific feature, structure, or characteristic described in connection with that embodiment. Therefore, the phrases "in one embodiment," "in some embodiments," "in other embodiments," "in still other embodiments," etc., appearing in different parts of this specification do not necessarily refer to the same embodiment, but rather mean "one or more, but not all, embodiments," unless otherwise specifically emphasized. The terms "comprising," "including," "having," and variations thereof mean "including but not limited to," unless otherwise specifically emphasized. The term "connection" includes direct connections and indirect connections, unless otherwise stated. "First" and "second" are used for descriptive purposes only and should not be construed as indicating or implying relative importance or implicitly specifying the number of technical features indicated.

[0029] In the embodiments of this application, the words "exemplarily" or "for example" are used to indicate examples, illustrations, or explanations. Any embodiment or design described as "exemplarily" or "for example" in the embodiments of this application should not be construed as being more preferred or advantageous than other embodiments or design solutions. Specifically, the use of the words "exemplarily" or "for example" is intended to present the relevant concepts in a specific manner.

[0030] The shooting cache management method provided in this application embodiment can be used in scenarios where electronic devices use various shooting modes in the camera function to shoot. Some terms will be explained below.

[0031] 1. Buffer: In the image processing pipeline, a buffer serves as a data transfer and temporary storage mechanism, including an input buffer and an output buffer. The input buffer temporarily stores the raw pixel data so that subsequent processing steps can access it. The output buffer is used to temporarily store the image after a series of algorithmic processing steps, for saving and display.

[0032] 2. Cache management mode: For different image processing algorithms, there are three different data processing modes.

[0033] The first mode is bypass mode, which means bypassing the processing logic of a certain node and directly passing the input data to the next node. For example, when a node is not running an algorithm, it is in bypass mode, and no new buffer is allocated.

[0034] The second mode is the inplace mode. In this mode, it processes the raw data directly; the input image is the output image. For example, adding watermarks or filters to an image requires the algorithm to modify the input image data directly. The output buffer is the same as the input buffer, so only one output buffer needs to be allocated in the inplace mode.

[0035] The third mode, outplace mode, requires allocating a new buffer for computation and then copying the results to the output buffer. This necessitates allocating two buffers: an input buffer and an output buffer, and performing a copy operation to move data from the input buffer to the output buffer. For example, when running a blurring algorithm, the original image data in the input buffer is first processed using a depth algorithm to obtain image data A. Then, the processed image data A is copied to the output buffer. A segmentation algorithm is then used to process image data A to obtain image data B. Finally, image data A and image data B in the output buffer are fused to obtain the final processed image.

[0036] In the following text, the above three modes will be referred to as "cache management modes".

[0037] For example, some shooting modes include complex image processing algorithms. A mode may include multiple algorithms, such as a shooting mode that includes beautification, bokeh, and filters. The following will use a mobile phone as an example to illustrate this.

[0038] The electronic device can display a desktop 100 as shown in Figure 1A(a). The desktop 100 includes application icons, such as a camera application icon 1001. The user can open the camera application by operating the camera application icon 1001. In response to the user's operation on the camera application icon 1001, as shown in Figure 1A(b), the electronic device displays a camera function interface 101. The interface 101 includes a photo preview interface 1010, a shooting mode selection area 1011, a photo control 1012, and a zoom area 1015. The photo control 1012 is used to trigger a photo capture. The zoom area 1015 can be a slider used to adjust the zoom level of the electronic device's camera. As the zoom level is adjusted, the image displayed by the electronic device can switch from the image captured by camera 1 to the image captured by camera 2. The preview area 1010 is used to display the processed image data in real time. The shooting mode selection area 1011 also includes selection controls for various shooting modes, such as photo mode 1013 and portrait mode 1014.

[0039] As one possible implementation, in response to a user's operation on the camera application icon 1001, as shown in Figure 1A(b), when the electronic device displays the camera function interface 101, the preview area 1010 displays a preview image in the shooting mode by default, and the shooting mode control 1013 is in a selected state. As another possible implementation, in response to a user's operation on the camera application icon 1001, the preview area 1010 displays a preview image of the previously selected shooting mode by default. For example, if the shooting mode selected before the camera application exits is portrait mode, in response to a user's operation on the camera application icon 1001, as shown in Figure 1B(a), the preview area 1010 displays a preview image in portrait mode by default, and the portrait mode control 1014 is in a selected state.

[0040] As another possible implementation, when the electronic device displays the camera function interface 101, the user can switch between different shooting modes. For example, as shown in Figure 1A(b), when the camera application is in shooting mode, in response to the user's operation in the mode selection area 1011 (such as clicking the portrait mode control 1014, or sliding to the right in the mode selection area), as shown in Figure 1B(a), the camera application switches to portrait mode and displays a preview image in portrait mode in the preview area 1010.

[0041] Optionally, in portrait mode, as shown in Figure 1B(a), the camera function interface 101 may further include a blur control 1016 and a beautification control 1017, used to provide the user with image blurring and image beautification functions, respectively. Specifically, in response to the user's operation on the blur control 1016, as shown in Figure 1B(b), the camera application displays the image processed by the blur algorithm in the preview area 1010, and the blur control 1016 is displayed as selected. In response to the user's operation on the beautification control 1017, as shown in Figure 1B(c), the camera application displays the image processed by the beautification algorithm in the preview area 1010, and the beautification control 1017 is displayed as selected. Alternatively, as shown in Figure 1B(d), after enabling the blur control 1016, in response to the user's operation, the algorithm corresponding to the beautification control 1017 can be further enabled, and the image processed by the blur algorithm and beautification algorithm can be displayed in the preview area 1010. That is, the image is overlaid with the image processed by the blur algorithm and beautification algorithm and displayed in the preview area 1010. Optionally, in portrait mode, the camera function interface 101 may also include controls for providing other image processing functions, such as filter controls (not shown in the figure), for providing different styles of filter algorithms to process the image.

[0042] The above describes how to display the preview interface for different shooting modes, using portrait mode as an example to illustrate that various algorithms can be used to process images and display the processed images in the preview interface. It should be understood that in other shooting modes, different algorithms can also be switched to process the original image based on the default settings of the shooting mode and / or user operations, and the processed preview image will be displayed in the preview area 1010.

[0043] The display preview interface involves algorithmic processing and data transmission of the original image, which will be further described below with reference to the hardware and software architecture diagram in Figure 2. It should be understood that the following description only covers some steps in the preview process and does not constitute a specific limitation on the electronic device's role in the camera preview process. In other embodiments, more or fewer steps may be included than those described below.

[0044] As shown in Figure 2, the layered architecture divides the software into several layers, each with a clear role and division of labor. Layers communicate with each other through software interfaces. In some embodiments, the operating system is divided into four layers, from top to bottom: the application layer, the application framework layer, the hardware abstraction layer (HAL), and the driver layer. For clarity, Figure 2 also shows the hardware layer to illustrate the relationship between the relevant hardware and software.

[0045] The application layer can include a series of application packages. As shown in Figure 2, application packages can include applications such as a camera and a gallery. The camera provides shooting functionality. The gallery application provides image browsing and editing functionality. It is understandable that the application layer can also include applications such as image editors.

[0046] The application framework layer provides application programming interfaces (APIs) and programming frameworks for applications within the application layer. The application framework layer includes some predefined functions. In this embodiment, the application framework layer may include a camera access interface, which may include camera management and camera devices. The camera access interface provides APIs and programming frameworks for camera applications. Understandably, the application framework layer may also include a window manager, content provider, view system, phone manager, resource manager, notification manager, etc.

[0047] The Hardware Abstraction Layer (HAL) is an interface layer located between the application framework layer and the driver layer. The HAL layer abstracts the hardware of the hardware layer, providing a unified hardware interface for the upper layers. The HAL layer can also provide a virtual hardware platform for the operating system. In this embodiment, the HAL may include a camera hardware abstraction layer, a decision module, an image processing module, etc. The camera hardware abstraction layer can provide virtual hardware for camera device 1 (camera 1), camera device 2 (camera 2), or more camera devices. The image processing module may include an algorithm library for performing algorithmic processing operations on images, such as watermarking, beautification, scaling, stitching, and enhancing sharpness. The image processing module also includes a thin front end (TFE) module for performing color correction, de-mosaicing, and other processing on each frame of the image stream acquired by camera 1 and / or camera 2. The image processing module may also include an image processing engine (IPE) module for further processing of the images processed by the thin front end module. The image processing module may also include a target cache management module, and the hardware abstraction layer may also include a decision module; the specific functions of these two modules will be further described later.

[0048] The hardware abstraction layer also includes modules such as realtimeMcx, which can be used to store images acquired by the image sensor. For details, please refer to relevant existing technologies, which will not be elaborated here.

[0049] The driver layer is the layer between hardware and software. It includes drivers for various hardware components, such as camera drivers, digital signal processor drivers, and image processor drivers.

[0050] In this embodiment, after receiving a user's operation on the camera application icon 1001, the electronic device calls the `open` interface function of the camera hardware abstraction layer to start the camera application. For example, after receiving a user's operation on the camera application icon 1001, the camera application calls the `open` interface function of the camera hardware abstraction layer to start the camera and calls the `configure_stream` interface function to allocate the stream. After the camera hardware abstraction layer loads the request through the `process_capture_request(PCR)` interface function, it calls the camera device driver in the driver layer. The camera device driver then drives the image sensor and other hardware devices to respond to the recording command and acquire image data, and drives the image signal processor to preprocess the image. The image sensor stably acquires raw image data at a set frame rate (e.g., one frame every 33 milliseconds) and sends it back to the image processing module.

[0051] During the streaming phase, electronic devices are configured with a tiny stream, a preview stream, and a photo stream. The tiny stream is an image stream generated during photo capture or preview scenarios, which is a downsampled image stream of the image data captured by the image sensor (regardless of whether it's a photo capture or a preview). For example, its resolution can be 640x480, which can reduce the computational burden during image recognition.

[0052] Optionally, during the streaming phase, a node flag can be set for each node to identify its buffer management mode: bypass, inplace, or outplace. As mentioned earlier, in bypass mode, no new buffer is allocated; in inplace mode, only the output buffer needs to be allocated; and in outplace mode, both input and output buffers need to be allocated and copied. When the image processing module calls algorithms from the algorithm library to process each frame of image data acquired by the image sensor, the target buffer manager (TBM) within the image processing module allocates buffers for the algorithm processing based on the node flag. If the node flag is set to outplace mode, two different buffers (input and output) need to be allocated, and data copying occurs during image data processing. After the image is processed, the camera hardware abstraction layer can display the processed image.

[0053] As described above, in existing technologies, since the cache management mode is determined during the streaming phase, the image processing algorithms required for different frames may vary even within the same shooting mode. For example, in portrait mode, image frames acquired when the beautification function is not enabled do not require beautification processing before being displayed, while image frames acquired after the beautification function is enabled do. To avoid temporary memory shortages caused by algorithm changes during the shooting process, which could affect the entire shooting process, existing technologies typically select the outplace mode directly during the streaming phase. This results in the electronic device still requesting input and output buffers for each frame and performing copy operations. For example, as shown in Figure 4(a), for image frames A, B, and C, only image frame A requires two buffers when using the beautification algorithm, while image frame B only requires the output buffer for watermarking processing. Image frame C, which does not require algorithm processing, does not need to request a buffer. However, because the outplace mode is selected during the streaming phase, the electronic device requests additional buffers for both image frames B and C. This leads to a waste of performance and power consumption during the shooting process, and may even cause stuttering, resulting in a poor user experience.

[0054] To address the aforementioned issues, this application provides a method for managing shooting buffers. The method determines the buffer management mode for each frame. If the current frame is determined to be in place mode, only the output buffer needs to be allocated. If the current frame is determined to be in bypass mode, no new buffer is allocated, thereby reducing unnecessary buffer allocations.

[0055] For example, the aforementioned electronic devices may include smartphones, personal digital assistant (PDA) computers, tablet computers, laptop computers, camcorders, video recorders, cameras, smartwatches, smart wristbands, cellular phones, in-vehicle computers, and other devices with camera applications. This application does not impose any special limitations on the specific form of the aforementioned electronic devices.

[0056] Referring to Figure 3, which is a hardware structure diagram of an electronic device provided in an embodiment of this application. As shown in Figure 3, taking a mobile phone as an example, the electronic device may include: a processor 310, an external memory interface 320, an internal memory 321, a universal serial bus (USB) interface 330, a charging management module 340, a power management module 341, a battery 342, an antenna 1, an antenna 2, a mobile communication module 350, a wireless communication module 360, an audio module 370, a speaker 370A, a receiver 370B, a microphone 370C, a headphone jack 370D, a sensor module 380, a gyroscope sensor 380B, buttons 390, a motor 391, an indicator 392, a camera 393, a display screen 394, and a subscriber identification module (SIM) card interface 395, etc.

[0057] It is understood that the structure illustrated in this embodiment does not constitute a specific limitation on the electronic device. In other embodiments, the electronic device may include more or fewer components than illustrated, or combine some components, or split some components, or have different component arrangements. The illustrated components may be implemented in hardware, software, or a combination of software and hardware.

[0058] Processor 310 may include one or more processing units, such as: application processor (AP), modem processor, graphics processing unit (GPU), image signal processor (ISP), controller, memory, video codec, digital signal processor (DSP), baseband processor, and / or neural network processing unit (NPU), etc. The different processing units may be independent devices or integrated into one or more processors.

[0059] The charging management module 340 receives charging input from a charger, which can be either a wireless or wired charger. The power management module 341 connects to the battery 342, the charging management module 340, and the processor 310. The power management module 341 receives power from the battery 342 and / or the charging management module 340.

[0060] The wireless communication function of electronic devices can be realized through antenna 1, antenna 2, mobile communication module 350, wireless communication module 360, modem processor and baseband processor, etc.

[0061] The electronic device implements the display function of this application through a GPU, a display screen 394, and an application processor. The GPU is a microprocessor for image processing, connected to the display screen 394 and the application processor. The GPU is used to perform mathematical and geometric calculations and for graphics rendering. The processor 310 may include one or more GPUs, which execute program instructions to generate or modify display information.

[0062] Electronic devices can perform shooting functions through an ISP, one or more cameras 393, a video codec, a GPU, a display 394, and an application processor. The camera 393, controlled by exposure, can continuously output Bayer images. Each frame of the Bayer image is processed by the phone's ISP and then encoded to obtain the final image.

[0063] The sensors other than camera 393 constitute sensor module 380, which may include pressure sensor 380A, gyroscope sensor 380B, barometric pressure sensor 380C, magnetic sensor 380D, accelerometer sensor 380E, proximity sensor 380F, proximity light sensor 380G, fingerprint sensor 380H, temperature sensor 380J, touch sensor 380K, ambient light sensor 380L, bone conduction sensor 380M, etc. Detailed descriptions of the functions of each sensor can be found in relevant sections of conventional technology and will not be elaborated upon here.

[0064] Internal memory 321 can be used to store computer executable program code, which includes instructions. Processor 310 executes various functional applications and data processing of the electronic device by running the instructions stored in internal memory 321.

[0065] Electronic devices can implement audio functions through audio modules 370, speakers 370A, receivers 370B, microphones 370C, headphone jacks 370D, and application processors. Examples include audio recording during video recording.

[0066] The software system of the aforementioned electronic device can adopt a layered architecture, event-driven architecture, microkernel architecture, microservice architecture, or cloud architecture. This application embodiment uses the layered architecture Android system as an example to exemplify the software structure of the electronic device.

[0067] The shooting method provided in this application can be executed in an electronic device with the above-described hardware and software structure. The buffer processing method of this solution will be described below with reference to Figures 2 and 3, and Figures 4 and 5.

[0068] As an optional implementation, if the target cache manager (hereinafter referred to as "TBM manager") in the image processing module of the electronic device can simultaneously support the three modes of bypass, inplacing and outplace, then after determining the cache management mode of each frame, the target cache of image and video data can be managed frame by frame through a TBM object.

[0069] Since the existing TBM manager can only support one mode, as an alternative implementation method, as shown in Figure 4(b), two additional TBM managers can be added to the existing image processing module. A TBM pool can then be used to schedule the three TBM managers to achieve frame-by-frame buffer management. The following sections, in conjunction with Figures 1A, 1B, 4, and 5, illustrate the specific implementation and effects of the above method by launching the camera application and switching between different shooting modes during the use of the camera application.

[0070] S401 receives the first operation for the camera application and configures the camera stream.

[0071] For example, as shown in Figure 1A(a), the first operation includes a user's action on the camera application icon 1001 on the desktop 100, which is used to launch the camera application and configure the camera stream (hereinafter referred to as "streaming"). Alternatively, the first operation includes a user's action of switching between different shooting modes. As shown in Figures 1A(b) to 1B(a), when a user switches the camera application from photo mode to portrait mode, the camera application generally re-streams. Optionally, the user's first operation on the camera application may also include other operations to trigger the camera to perform streaming, such as switching between different camera devices, such as switching from a rear camera to a front camera. It should be understood that in the prior art, all operations that trigger the camera application to configure camera stream parameters belong to the aforementioned first operation of this solution.

[0072] After receiving the first operation from the camera application, the camera application calls the configure_stream interface function to configure the stream and sends the configuration information to the hardware abstraction layer.

[0073] For example, in this embodiment, as described above, the streaming configuration includes streaming information for configuring the tiny stream, preview stream, and capture stream. A request carrying the streaming information is sent to the camera hardware abstraction layer, causing the camera driver in the camera hardware abstraction layer to drive the image sensor to capture images. In this embodiment, the streaming information may also include one or more of the following shooting request identifiers: shooting mode, image format (e.g., JPEG, RAW), resolution, frame rate, autofocus (AF), exposure control (AE), white balance (WB), ISO sensitivity, pixel encoder mode, timestamp, etc.

[0074] It should be understood that the flow allocation information may include more or less configuration information than the aforementioned configuration information. For details, please refer to the existing technology for the description of information in the flow allocation stage, which will not be repeated here.

[0075] As an optional implementation, after the stream is distributed, the camera hardware abstraction layer...

[0076] After receiving the request (load request), the process_capture_request(PCR) interface function calls the camera device driver in the kernel layer. The camera device driver then drives the image sensor to acquire the image and drives the image signal processor to preprocess the image.

[0077] S402, Obtain the first image stream based on the distribution information.

[0078] Specifically, after the image sensor acquires the original image according to the distribution information, it continuously transmits the first image stream back to the hardware abstraction layer through the shooting path.

[0079] S403, determines the algorithm for each frame in the first image stream based on the configuration file.

[0080] Specifically, electronic devices pre-store configuration files, which can be XML files named "Feature XML". These configuration files store the algorithms to be used for different shooting information, such as HDR, lighting conditions, low light, and shooting modes. Specifically, the configuration file stores multiple features under different shooting information, each corresponding to a different algorithm for image processing. For example, for shooting information of "rear mode and escape mode", there is a feature named "forenormalmodebeauty", which pre-sets for overlay and skin-smoothing algorithms.

[0081] As an optional implementation, as mentioned above, since the decision module decides whether to downsample each frame of image captured by the image sensor to form a tiny stream, the decision module can access the configuration file when deciding on the tiny stream, thereby obtaining the image capture information in a timely manner. Based on the capture information of each frame of the acquired first image stream, the decision module, in conjunction with the configuration file, determines the feature corresponding to each frame, thus determining the algorithm for each frame in the first image stream. Specifically, for example, the decision module, based on the image information of the first image frame in the first image stream, in conjunction with the configuration file, determines that the image information of the first image frame corresponds to feature A. Under feature A, a blurring algorithm is required to process the first image frame. As another example, the decision module, based on the image information of the second image frame in the first image stream, in conjunction with the configuration file, determines that the image information of the second image frame corresponds to feature B. Under feature B, a watermarking algorithm is required to process the second image frame. Yet another example, the decision module, based on the image information of the third image frame in the first image stream, in conjunction with the configuration file, determines that the image information of the third image frame corresponds to feature C. Under feature C, no algorithm is required to process the third image frame.

[0082] During the execution of step S402, step S403 will be performed simultaneously.

[0083] S404, determine the buffer management mode for each frame based on the algorithm for each frame in the first image stream.

[0084] As an optional implementation, after determining the algorithm for each frame in the aforementioned first image stream, the decision module can determine the cache management mode for each frame based on the algorithm category. Specifically, for example, referring to the aforementioned description of cache management modes, the cache management mode is determined according to the algorithm category. When the algorithm category of the first image frame is an algorithm that needs to retain the input image data for other operations, such as a beautification algorithm or a blurring algorithm, then the cache management mode for this frame is determined to be the outplace mode. As another example, when the algorithm category of the second image frame is an algorithm that only retains the output image data after algorithm processing, such as a watermarking algorithm or a filter algorithm, then the cache shooting mode for this frame is determined to be the inplace mode. Furthermore, when the third image frame does not have any algorithm processing, the cache management mode for this frame is determined to be the bypass mode.

[0085] Optionally, a frame buffer management mode flag can be used to identify the buffer management mode of each frame, with the bypass field corresponding to the bypass mode, the inplace field corresponding to the inplace mode, and the outplace field corresponding to the outplace mode.

[0086] S405 configures the cache corresponding to each frame according to the cache management mode of each frame.

[0087] Since a single TBM object can currently only support one mode, this embodiment adds two additional TBM managers to the existing single TBM manager, distinguishing them by flags. For example, as shown in Figure 4(b), three TBM managers can be configured: a TBM-bypass manager, a TBM-inplace manager, and a TBM-outplace manager. These three TBM managers correspond to different cache management modes: the TBM-bypass manager performs cache management in bypass mode, the TBM-inplace manager performs cache management in inplace mode, and the TBM-outplace manager performs cache management in outplace mode. To manage these three TBM managers, this embodiment provides a TBM manager pool, which schedules TBM managers for cache management according to the cache management mode.

[0088] Specifically, the decision module sends the "outplace" flag for the buffer management mode of the first image frame to the image processing module. For example, as shown in Figure 4(b), the TBM management pool in the image processing module determines the corresponding TBM-outplace manager based on the "outplace" field. The TBM-outplace manager then adopts the outplace buffer management mode for the first image frame, allocates two buffers (input buffer and output buffer), and performs a copy operation to copy the data from the input buffer to the output buffer. If the decision module sends the "inplace" flag for the buffer management mode of the second image frame to the image processing module, the TBM-inplace manager adopts the inplace buffer management mode for the second image frame, requiring only one output buffer to be allocated. If the decision module sends the "bypass" flag for the buffer management mode of the third image frame C1 to the image processing module, the TBM-bypass manager adopts the bypass management mode for the third image frame C1, without allocating a new buffer. Therefore, this solution adds bypass and inplace attributes to the TBM Pool, and outputs the cache management mode of the decision based on the algorithm of the current frame. The TBM manager to be used is selected based on the cache management mode. This can determine the cache management mode according to the actual algorithm requirements of each frame, which can not only meet the sufficient cache requirements of each frame, but also reduce unnecessary buffer allocation.

[0089] S406, process each frame of image using the buffer corresponding to each frame to obtain the second image stream.

[0090] After configuring the corresponding cache in the target cache manager, the image processing module uses the cache to process each frame of the first image stream. For example, when a blurring algorithm is needed to process the aforementioned first image frame, an input buffer and an output buffer can be requested according to the outplace cache management mode to serve as caches during the image blurring algorithm processing. After processing, the first image frame yields the first processed image frame. After each frame in the first image stream is processed by the algorithm using the corresponding cache for each frame, the image processing module continuously sends the processed images frame by frame to the camera application to obtain the second image stream.

[0091] S407, display the second image stream.

[0092] After receiving the second image stream, the camera application displays it. For example, as shown in Figure 1B(b), the electronic device continuously displays the second image stream processed by the blurring algorithm in the preview area 1010.

[0093] As an optional implementation, in step S405 above, more than one TBM can be set up for a cache management mode to address the situation of insufficient TBM manager scheduling. For example, when using a TBM-outplace manager to provide cache for the algorithm to process frame N in outplace mode, due to the complexity of the algorithm, the algorithm may process images slowly, resulting in the request for frame N+1 arriving before the processing of frame N is completed. To avoid insufficient cache resources, multiple TBM managers can be set up. When frame N+1 is determined to also need to use outplace cache management mode after steps S403-404, the cache of frame N+1 can be configured using other idle TBM managers in step S405.

[0094] It is worth noting that, as an optional implementation, the node tags configured by the developer as described above will be retained in the distribution information in step S401. To prevent the node tags in the distribution stage from affecting the cache configuration of each frame, the node tags need to be set to an inactive state in step S401 so that the electronic device does not perform cache management according to the node tags. Specifically, when the node tag in the distribution information is "outplace" and the cache management mode determined for the current frame is inplace mode, the log of the current frame will display 1, the cache mode of the current frame is "inplace", and the outplace information does not apply to the current frame "disable outplace feature for current frame".

[0095] In the scheme of this application embodiment, the determination of the cache management mode is optimized from the original determination in the distribution stage to determination with each frame, thereby avoiding the waste of performance and power consumption caused by cache management according to the mode.

[0096] This application also provides an electronic device, which may include a display screen, a memory, and one or more processors (such as a CPU, GPU, NPU, etc.). The display screen, memory, and processor are coupled. The memory is used to store computer program code, which includes computer instructions. When the processor executes the computer instructions, the electronic device can perform various functions or steps performed by the device in the above method embodiments.

[0097] This application also provides a chip system including at least one processor and at least one interface circuit. The processor and the interface circuit are interconnected via lines. For example, the interface circuit can be used to receive signals from other devices (e.g., the memory of an electronic device). As another example, the interface circuit can be used to send signals to other devices (e.g., the processor). Exemplarily, the interface circuit can read instructions stored in the memory and send the instructions to the processor. When the instructions are executed by the processor, the electronic device can perform the steps in the above embodiments. Of course, the chip system may also include other discrete devices, and this application does not specifically limit this.

[0098] This embodiment also provides a computer storage medium storing computer instructions. When the computer instructions are executed on an electronic device, the electronic device performs the aforementioned method steps to implement the image processing method described above.

[0099] This embodiment also provides a computer program product that, when run on a computer, causes the computer to perform the aforementioned steps to implement the image processing method described above.

[0100] In addition, embodiments of this application also provide an apparatus, which may specifically be a chip, component, or module. The apparatus may include a connected processor and a memory; wherein the memory is used to store computer execution instructions, and when the apparatus is running, the processor may execute the computer execution instructions stored in the memory to cause the chip to execute the image processing methods in the above-described method embodiments.

[0101] In this embodiment, the electronic device, computer storage medium, computer program product or chip are all used to execute the corresponding method provided above. Therefore, the beneficial effects that can be achieved can be referred to the beneficial effects of the corresponding method provided above, and will not be repeated here.

[0102] Through the above description of the embodiments, those skilled in the art can clearly understand that, for the sake of convenience and brevity, only the division of the above functional modules is used as an example. In actual applications, the above functions can be assigned to different functional modules as needed, that is, the internal structure of the device can be divided into different functional modules to complete all or part of the functions described above.

[0103] In the several embodiments provided in this application, it should be understood that the disclosed apparatus and methods can be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative. For instance, the division of modules or units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another device, or some features may be ignored or not executed. Furthermore, the mutual coupling or direct coupling or communication connection shown or discussed may be through some interfaces; the indirect coupling or communication connection between devices or units may be electrical, mechanical, or other forms.

[0104] The unit described as a separate component may or may not be physically separate. The component shown as a unit can be one physical unit or multiple physical units, that is, it can be located in one place or distributed in multiple different places. Some or all of the units can be selected to achieve the purpose of the solution in this embodiment according to actual needs.

[0105] Furthermore, the functional units in the various embodiments of this application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit.

[0106] If the integrated unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a readable storage medium. Based on this understanding, the technical solutions of the embodiments of this application, essentially or in other words, the parts that contribute to the prior art, or all or part of the technical solutions, can be embodied in the form of a software product. This software product is stored in a storage medium and includes several instructions to cause a device (which may be a microcontroller, chip, etc.) or processor to execute all or part of the steps of the methods of the various embodiments of this application. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0107] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of this application and are not intended to limit it. Although this application has been described in detail with reference to preferred embodiments, those skilled in the art should understand that modifications or equivalent substitutions can be made to the technical solutions of this application without departing from the spirit and scope of the technical solutions of this application.

Claims

1. A method for managing shooting buffers, characterized in that, include: After receiving the first operation applied to the camera, a first image stream is acquired, the first image stream including a first image frame and a second image frame; The algorithm for the first image frame determines that the first image frame is in the first cache management mode; The first cache of the first image frame is configured according to the first cache management mode, and the first cache is used to process the first image frame using the first algorithm. The second image frame is determined to be in the second cache management mode based on the algorithm used for the second image frame; the algorithm used for the first image frame is different from the algorithm used for the second image frame, and the first cache management mode is different from the second cache management mode. The second cache of the second image frame is configured according to the second cache management mode, and the second cache is used by the second algorithm to process the second image frame.

2. The method according to claim 1, characterized in that, The electronic device includes a first target cache manager and a second target cache manager; Configuring the first cache of the first image frame according to the first cache management mode includes: determining a first target cache manager according to the first cache management mode, and configuring the first cache using the first target cache manager; Configuring the second cache of the second image frame according to the second cache management mode includes: determining a second target cache manager according to the second cache management mode, and configuring the second cache using the second target cache manager.

3. The method according to claim 1, characterized in that, The first operation includes switching between different shooting modes, launching a camera application, or switching between different camera devices.

4. The method according to claim 1, characterized in that, The first cache management mode and the second cache management mode are either inplace mode or outplace mode. In the inplace mode, an output cache needs to be configured, and in the outplace mode, an input cache and an output cache need to be configured, and the data in the cache needs to be copied.

5. The method according to claim 1, characterized in that, The algorithms for the first image frame and the second image frame are either algorithms that need to retain the image data processed by the input algorithm, or algorithms that only retain the image data processed by the algorithm.

6. The method according to claim 1, characterized in that, Also includes: Before determining that the first image frame is in the first cache management mode according to the algorithm of the first image frame, the shooting information of the first image frame is obtained; The algorithm for determining each frame in the first image stream is based on the shooting information of the first image frame and the configuration file. The configuration file is an XML file that stores the algorithm corresponding to the shooting information.

7. The method according to claim 6, characterized in that, Also includes: After receiving the first operation applied to the camera, the configuration file is accessed when deciding whether to downsample the first image stream to form a tiny stream.

8. The method according to claim 1, characterized in that, The first image stream also includes a third image frame, which does not require processing by an algorithm; The third image frame is determined to be in bypass mode, and no caching is configured for the third image frame.

9. The method according to claim 1, characterized in that, After receiving a first operation on the camera application, obtaining the first image stream includes: receiving the first operation on the camera application, configuring streaming information, and obtaining the first image stream according to the streaming information; wherein, the streaming information includes node tags; It also includes setting the node label to an inactive state.

10. A chip system, characterized in that, The chip system is applied to an electronic device including a display screen and a memory. The chip system includes an interface circuit and a processor. The interface circuit and the processor are interconnected via a line. The interface circuit is used to receive signals from the memory of the electronic device and send signals to the processor, the signals including computer instructions stored in the memory. When the processor executes the computer instructions, it causes the electronic device to perform the method as described in any one of claims 1-9.

11. An electronic device, characterized in that, include: A display screen, one or more processors, and one or more memories; the one or more processors are coupled to the display screen and the one or more memories; the one or more memories are used to store computer program code, the computer program code including computer instructions that, when the one or more processors execute the computer instructions, cause the electronic device to perform the method as described in any one of claims 1-9.

12. A computer-readable storage medium, characterized in that, Includes computer instructions that, when executed on an electronic device, cause the electronic device to perform the method as described in any one of claims 1-9.

Images