Graphics processing apparatus, method, and related device

Abstract

Disclosed in embodiments of the present application are a graphics processing apparatus, a method, and a related device. The apparatus comprises: a processor used for generating a preset first graphics task command; a sensing processing unit used for: acquiring sensor data, processing the sensor data to obtain a user behavior event, and modifying the first graphics task command on the basis of the user behavior event to obtain a second graphics task command; and a graphics processing unit used for performing graphics rendering on the basis of the second graphics task command to obtain a first image. The graphics processing apparatus in the present application can ensure a rendering effect for real-time interaction while satisfying low power consumption requirements.

Description

A graphics processing apparatus, method and related equipment

[0001] This application claims priority to Chinese Patent Application No. 202411759018.X, filed with the China National Intellectual Property Administration on November 30, 2024, entitled "A Graphic Processing Apparatus, Method and Related Equipment", the entire contents of which are incorporated herein by reference. Technical Field

[0002] This application relates to the field of image processing technology, and in particular to a graphics processing apparatus, method and related equipment. Background Technology

[0003] As user experience (UX) design becomes increasingly sophisticated, more refined dynamic interface designs make the display effects of terminal devices more interactive and experiential.

[0004] Taking the Always-on Display (AoD) scenario on a mobile phone as an example, users can choose different animation effects according to their preferences. When the phone enters the screen-off state (such as standby), they can trigger relevant animation effects (such as a 3D digital clock, butterfly flapping wings, changing text signatures, animated emoticons, physics engine animations, etc.) by tapping the screen or looking at the screen. In other words, real-time user interaction enhances the visual playability and sophistication of the experience. All of these interactive feedback animations require waking up the processor (such as the Central Processing Unit (CPU)), and the CPU drives the Graphics Processing Unit (GPU) to perform real-time calculations and rendering based on user behavior events.

[0005] Assuming a user triggers AoD animations for 1 hour per day, the CPU power consumption for waking up would be approximately 50mAh. However, with the continuous evolution of the operating system (OS) and the increasing sophistication and complexity of AoD animations, even more power consumption will be required.

[0006] Furthermore, there is an increasing number of other low-power scenarios where GPUs are needed for graphics processing. For example, artificial intelligence (AI) glasses require GPUs to perform distortion correction when playing videos. Similarly, mobile phones, in low-power video playback scenarios, need to use microcontroller units (MCUs) to drive the GPU to handle tasks such as displaying subtitles or animated GIFs.

[0007] Therefore, how to meet the low power consumption requirements and ensure the rendering effect of real-time interactive animations in certain specific animation scenarios is an urgent problem to be solved. Summary of the Invention

[0008] This application provides a graphics processing apparatus, method, and related devices that can ensure real-time interactive rendering effects while meeting low power consumption requirements.

[0009] In a first aspect, embodiments of this application provide a graphics processing apparatus, which may include: a processor for generating a preset first graphics task command; a perception processing unit for: acquiring sensor data, processing the sensor data to obtain a user behavior event, and modifying the first graphics task command according to the user behavior event to obtain a second graphics task command; and a graphics processor for performing graphics rendering based on the second graphics task command to obtain a first image.

[0010] This application provides a graphics processing device applicable to graphics rendering scenarios such as always-on display animations on mobile phones. Specifically, it divides the image processing task into a pre-defined template (e.g., a first graphics task command) and a part that needs to be modified based on user behavior events. The modified task command (e.g., a second graphics task command) is then used as a GPU rendering task to obtain the final display image (i.e., the first image). In other words, no online processor involvement is required during the graphics rendering task. Only the offline template generated by the processor and slight modifications made to the template by the perception processing unit based on user behavior events are needed to obtain the task to be rendered by the GPU. Since this process does not require online processor involvement (e.g., no need to wake up the processor in a low-power state), energy consumption can be greatly saved. Furthermore, since this rendering task is a real-time response based on user needs, it can ensure rendering effects that guarantee real-time interaction with the GPU. In summary, this application proposes a high-energy-efficiency rendering scheme that combines offline and online collaboration. This effectively reduces processor load, avoids waking up the processor, thereby reducing device power consumption, and enables the GPU to achieve a high-end, sophisticated rendering effect that allows real-time interaction with the user, greatly improving the user experience.

[0011] In one possible implementation, the device further includes a storage unit; the processor is further configured to store the first graphics task command in the storage unit; the perception processing unit is specifically configured to modify the first image task command in the storage unit according to the user behavior event to obtain a second graphics task command. In this embodiment, the processor sends pre-generated image task commands to the storage unit for pre-storage. Thus, when the perception processing unit needs to modify the pre-generated image task command after a user behavior event is triggered, it can read it from the storage unit for modification, or it can directly modify the task command stored in the storage unit online. This achieves offline template generation by the processor and online rendering of refined effects with real-time interaction with the user by the graphics processor.

[0012] In one possible implementation, the preset first graphics task command includes a preset image task command template, the user behavior event is used to trigger the target animation effect, and the second graphics task command is an image task command capable of realizing the target animation effect. In this embodiment, the processor pre-generates the preset first graphics task command, which may specifically include an image task command template for realizing the target animation effect, i.e., a fixed or basic command portion for rendering the target animation effect. The sensor data acquired by the perception processing unit, and the user behavior event obtained based on the sensor data analysis, are the relevant operations used to trigger the target animation effect (e.g., user manual operation or eye gaze). Correspondingly, the second graphics task command obtained by the perception processing unit after modifying the preset first graphics task command is the task command capable of realizing the target animation effect. That is, the processor pre-generates the template portion of the rendering task, and the perception processing unit modifies the template online to obtain a rendering task command with user interaction feedback, ensuring real-time interactive rendering effects while simultaneously meeting low power consumption requirements.

[0013] In one possible implementation, the processor is further configured to enter a sleep state after generating the first graphics task command. In this embodiment, the processor can enter a sleep state after generating the preset template task command, and therefore does not need to participate or interact during the subsequent rendering of the first image by the graphics processor. That is, the processor can remain in a sleep state (such as hibernation, standby, or other low-power states without being woken up), which can greatly save power consumption. Furthermore, since there is no need to go through the process of waking up the processor or interact with the CPU, the rendering efficiency of the GPU can be improved.

[0014] In one possible implementation, the sensing processing unit is always online. In this embodiment, the sensing processing unit can always be online to acquire sensor data in real time and process it to obtain user behavior events. That is, the always-on sensing processing unit can monitor user operations and other behaviors in real time to respond to user needs and improve user experience.

[0015] In one possible implementation, the processor, the graphics processor, and the sensing processing unit are integrated on the same chip. In this embodiment, integrating the processor, graphics processor, and sensing processing unit onto the same chip, i.e., belonging to a single System-on-a-Chip (SoC), further reduces signal transmission between components, improves data transmission efficiency, reduces overall power consumption, and lowers system complexity and cost, thereby comprehensively improving the SoC's performance.

[0016] In one possible implementation, the apparatus further includes a display subsystem for performing display processing on the first image to obtain a second image, the second image being displayed on a display screen. In this embodiment, the graphics processing apparatus further includes a display subsystem for performing further display processing (e.g., rotation, scaling, or layer overlay operations) on the first image rendered by the graphics processor, to better and more appropriately display it on the display screen.

[0017] In one possible implementation, the first graphics task command includes multiple first graphics task commands, and the second graphics task command includes multiple second graphics task commands. The perception processing unit is specifically used to schedule the current image task command among the multiple second graphics task commands to the graphics processor. The graphics processor is specifically used to perform graphics rendering based on the current image task command to obtain the current image. In this embodiment, if the first graphics task command includes multiple first graphics task commands, and correspondingly the second graphics task commands include multiple second graphics task commands, then during the process of sending the second graphics task commands to the graphics processor for rendering, the perception processing unit can schedule multiple second graphics task commands to the graphics processor respectively. The graphics processor can then render the current graphics (such as the current first image) based on the image task commands scheduled by the perception processing unit. Afterward, the perception processing unit can schedule the next second graphics task command to the graphics processor, repeating the above process until the graphics processor executes all the second graphics task commands to render all the first images corresponding to the target animation. Thus, the scheduling and execution of multiple second graphics task commands are achieved through the interaction between the perception processing unit and the graphics processor.

[0018] In one possible implementation, the first graphics task command includes multiple first graphics task commands, and the second graphics task command includes multiple second graphics task commands. The device further includes: a scheduler, used to schedule the current image task command among the multiple second graphics task commands to the graphics processor; the graphics processor is specifically used to perform graphics rendering based on the current image task command to obtain the current image. In this embodiment, the first graphics task command includes multiple first graphics task commands, and correspondingly, the second graphics task command includes multiple second graphics task commands. During the process of the perception processing unit sending the second graphics task commands to the graphics processor for rendering, the scheduler can schedule multiple second graphics task commands to the graphics processor respectively. The graphics processor can then render the current graphics (such as the current first image) based on the image task commands scheduled by the scheduler. Afterwards, the scheduler can schedule the next second graphics task command to the graphics processor, repeating the above process until the graphics processor executes all the second graphics task commands to render all the first images corresponding to the target animation. Thus, the scheduling and execution of multiple second graphics task commands are achieved through the interaction between the scheduler and the graphics processor. This avoids excessive involvement of the perceptual processing unit (PSU) in the rendering process, thus conserving its power. In summary, by adding a scheduler to trigger GPU rendering, the PSU can further streamline the graphics software stack running on it.

[0019] Secondly, embodiments of this application provide a graphics processing method applied to a graphics processing device, the device including a processor, a graphics processor, and a perception processing unit; the method including: generating a preset first graphics task command through the processor; acquiring sensor data through the perception processing unit, processing the sensor data to obtain user behavior events, and modifying the first graphics task command according to the user behavior events to obtain a second graphics task command; and performing graphics rendering based on the second graphics task command through the graphics processor to obtain a first image.

[0020] In one possible implementation, the apparatus further includes a storage unit; the method further includes: storing the first graphics task command in the storage unit via the processor; and modifying the first graphics task command according to the user behavior event via the perception processing unit to obtain a second graphics task command, which includes: modifying the first image task instruction in the storage unit according to the user behavior event via the perception processing unit to obtain a second graphics task command.

[0021] In one possible implementation, the preset first graphic task command includes a preset image task command template, the user behavior event is used to trigger the target animation effect, and the second graphic task command is an image task command capable of realizing the target animation effect.

[0022] In one possible implementation, the method further includes: the processor entering a sleep state after generating the first graphics task command.

[0023] In one possible implementation, the apparatus further includes a display subsystem; the method further includes: performing display processing on the first image through the display subsystem to obtain a second image, the second image being displayed on a display screen.

[0024] In one possible implementation, the first graphics task command includes a plurality of first graphics task commands, and the second graphics task command includes a plurality of second graphics task commands; the method further includes: scheduling the current image task command among the plurality of second graphics task commands to the graphics processor through the perception processing unit; the step of performing graphics rendering based on the second graphics task command through the graphics processor to obtain a first image includes: performing graphics rendering based on the current image task command through the graphics processor to obtain a current image.

[0025] In one possible implementation, the apparatus further includes a scheduler; the first graphics task command includes a plurality of first graphics task commands, and the second graphics task command includes a plurality of second graphics task commands; the method further includes: scheduling the current image task command among the plurality of second graphics task commands to the graphics processor through the scheduler; the step of performing graphics rendering based on the second graphics task command through the graphics processor to obtain a first image includes: the graphics processor is specifically configured to perform graphics rendering based on the current image task command to obtain a current image.

[0026] Thirdly, this application provides a terminal device that may include a graphics processing device and a display screen provided in any of the implementations of the first aspect described above.

[0027] Fourthly, this application provides a system-on-a-chip (SoC) that may include a graphics processing device provided by any of the implementations of the first aspect described above.

[0028] Fifthly, this application provides a semiconductor chip that may include a graphics processing device provided by any of the implementations of the first aspect described above.

[0029] Sixthly, this application provides a chip system including the graphics processing apparatus provided in any of the implementations of the first aspect described above. In one possible design, the chip system further includes a memory for storing program instructions and data necessary or related to the operation of the chip system. The chip system may be composed of chips or may include chips and other discrete devices.

[0030] In a seventh aspect, this application provides a terminal device, which includes a graphics processing apparatus, wherein the graphics processing apparatus is provided by any implementation of the first aspect described above. The terminal device may further include a memory coupled to the graphics processing apparatus, which stores necessary program instructions and data of the terminal device. The terminal device may also include a communication interface for communicating with other devices or communication networks.

[0031] Eighthly, this application provides a computer-readable storage medium storing a computer program that, when executed by a graphics processing device, implements the graphics processing method flow described in any one of the second aspects above.

[0032] In a ninth aspect, embodiments of this application provide a computer program product including instructions that, when executed by a graphics processing device, cause the computing device to perform the graphics processing method flow described in any of the second aspects above. Attached Figure Description

[0033] To more clearly illustrate the technical solutions in the embodiments of this application or the background art, the accompanying drawings used in the embodiments of this application or the background art will be described below.

[0034] Figure 1A is a schematic diagram of a lock screen animation scene provided in an embodiment of this application.

[0035] Figure 1B is a schematic diagram of another lock screen animation scene provided in an embodiment of this application.

[0036] Figure 1C is a schematic diagram of an always-on display animation scene provided in an embodiment of this application.

[0037] Figure 2A is a schematic diagram of the structure of a graphics processing device provided in an embodiment of this application.

[0038] Figure 2B is a schematic diagram of another graphics processing device provided in an embodiment of this application.

[0039] Figure 2C is a schematic diagram of the structure of another graphics processing device provided in the embodiments of this application.

[0040] Figure 3A is a schematic diagram of the process of constructing the main task description in the offline configuration stage according to an embodiment of this application.

[0041] Figure 3B is a schematic diagram of the rendering and display process in an online rendering stage provided by an embodiment of this application.

[0042] Figure 4A is a schematic diagram of the control path and data path architecture design of a graphics processing device provided in an embodiment of this application.

[0043] Figure 4B is a schematic diagram of another control path and data path architecture design in a graphics processing device provided in an embodiment of this application.

[0044] Figure 5A is a schematic diagram of a graphics software stack running on a processor according to an embodiment of this application.

[0045] Figure 5B is a schematic diagram of a graphics software stack running on a perception processing unit according to an embodiment of this application.

[0046] Figure 6 is a flowchart illustrating a graphics processing method provided in an embodiment of this application. Detailed Implementation

[0047] The embodiments of this application will now be described with reference to the accompanying drawings. The terms "first," "second," "third," and "fourth," etc., in the specification, claims, and accompanying drawings of this application are used to distinguish different objects, not to describe a specific order. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover non-exclusive inclusion. For example, a process, method, system, product, or device that includes a series of steps or units is not limited to the listed steps or units, but may optionally include steps or units not listed, or may optionally include other steps or units inherent to these processes, methods, products, or devices. The reference to "embodiment" herein means that a specific feature, structure, or characteristic described in connection with an embodiment may be included in at least one embodiment of this application. The appearance of this phrase in various places in the specification does not necessarily refer to the same embodiment, nor is it a separate or alternative embodiment mutually exclusive with other embodiments. It will be explicitly and implicitly understood by those skilled in the art that the embodiments described herein can be combined with other embodiments.

[0048] As used in this specification, the terms "component," "module," "system," etc., are used to refer to computer-related entities, hardware, firmware, combinations of hardware and software, software, or software in execution. For example, a component can be, but is not limited to, a process running on a processor, a processor, an object, an executable file, an execution thread, a program, and / or a computer. As illustrated, applications running on computing devices and computing devices can both be components. One or more components may reside in a process and / or an execution thread, and components may be located on a single computer and / or distributed among two or more computers. Furthermore, these components can be executed from various computer-readable media on which various data structures are stored. Components can communicate, for example, via local and / or remote processes based on signals having one or more data packets (e.g., data from two components interacting with another component between a local system, a distributed system, and / or a network, such as the Internet interacting with other systems via signals).

[0049] First, some of the terms used in this application will be explained to facilitate understanding by those skilled in the art.

[0050] (1) An integrated circuit (IC) is a miniature electronic device or component. Using certain processes, the transistors, resistors, capacitors, inductors, and other components required for a circuit, along with interconnected wiring, are fabricated on a small piece or several small pieces of semiconductor wafers or dielectric substrates, and then packaged in a casing to form a miniature structure with the required circuit function; that is, an IC chip is an integrated circuit formed by placing a large number of microelectronic components (transistors, resistors, capacitors, etc.) on a plastic substrate to make a chip.

[0051] (2) Double Data Rate (DDR) memory, also known as DDR SDRAM (Double Data Rate Synchronous Dynamic Random Access Memory), is commonly referred to as DDR. SDRAM stands for Synchronous Dynamic Random Access Memory. The "synchronous" aspect refers to the fact that the memory requires a synchronous clock for operation; internal command transmission and data transfer are based on this clock. DDR is a storage device that loses data when power is off and requires periodic refresh to maintain data integrity. The DDR SDRAM subsystem consists of three parts: the DDR controller, the DDR PHY, and the DRAM memory chips.

[0052] (3) Static Random-Access Memory (SRAM) is a type of random access memory. The term "static" means that the data stored in this type of memory is permanently retained as long as power is supplied. In contrast, the data stored in Dynamic Random-Access Memory (DRAM) needs to be updated periodically. However, when the power supply is interrupted, the data stored in SRAM will still be lost (this is called volatile memory), unlike ROM or flash memory which can retain data even after power is off.

[0053] (4) NAND-flash memory is a non-volatile memory based on NAND technology. Compared with traditional Flash memory, NAND Flash has higher storage density, lower power consumption and longer life.

[0054] (5) User-mode drivers run in user mode and can only access limited system resources. User-mode programs interact with the operating system through system calls to obtain necessary system services. User-mode drivers are typically used to perform operations that do not require direct access to hardware, such as file system management or network communication.

[0055] (6) Kernel Mode Driver (KMD) is a driver program that runs within the operating system and is responsible for managing and maintaining the normal operation of hardware devices. KMD runs in the system kernel (similar to Ring 0 level), has direct access to system resources, and can provide efficient and stable device control.

[0056] (7) Driver Development Kit (DDK) is a device driver development kit. DDK provides a rich set of Application Programming Interface (API) functions and driver development templates.

[0057] (8) A Domain Specific Accelerator (DSA) is a programmable processor customized for a specific domain, capable of accelerating certain applications to achieve better performance and cost-effectiveness. Compared to general-purpose processors (such as CPUs and GPUs), DSAs exhibit higher efficiency and performance when handling specific tasks.

[0058] (9) Memory Management Unit (MMU) is a piece of computer hardware responsible for handling the processor's memory access requests. Its functions include virtual address to physical address translation (i.e., virtual memory management), memory protection, control of the central processing unit cache, and, in simpler computer architectures, bus arbitration and memory switching.

[0059] First, to facilitate understanding of the embodiments of this application, the specific technical problem to be solved by this application is further analyzed and proposed below. Currently, the implementation of always-on display animation effects mainly includes the following two technical solutions:

[0060] Option 1: SensorHub & ACPU Intelligent AOD Switching Solution

[0061] The main idea of ​​this solution is as follows: 1) For scenarios involving complex animations or high computing power requirements, the main processor (ARM CPU, ACPU) is woken up on demand for AoD display and rendering. Once the animation is complete or the user interaction is finished, the system switches to SensorHub. 2) For static AoD scenarios or dynamic AoD capabilities already supported by SensorHub, the AoD runs under the SensorHub solution.

[0062] The disadvantages of Option 1:

[0063] The problems with this solution are: 1) A more refined and realistic AoD effect requires frequent wake-ups of the ACPU to drive the GPU to perform rendering and computation tasks. 2) The CPU is currently running full graphics driver software (such as Vulkan and Gles), which places a heavy load on the CPU.

[0064] Option 2: Image Sequence Frame Scheme

[0065] The main idea of ​​this solution is as follows: 1) The CPU pre-determines the corresponding image sequence frames based on the user's configuration and selection, and then puts the image sequence frame content into a contiguous block of DDR memory. 2) When the AoD scene wake-up starts, the Micro Controller Unit (MCU) directly configures the Display Subsystem (DSS) to send the data to the display.

[0066] The disadvantages of Option 2:

[0067] The problems with this solution are: 1) The animation frame rate has increased from 60fps to 120fps, and the screen resolution has evolved from 1260x2720 to 1340x3080 and 1440x3360, which greatly increases the storage requirements for sequence frames. 2) The sequence frames cannot form real-time interaction with the user, and cannot achieve exquisite animation effects.

[0068] In summary, existing technologies cannot simultaneously achieve low power consumption and high-precision display functionality for real-time interaction in always-on display scenarios. Therefore, the technical problems to be solved by the embodiments of this application include: taking the reduction of power consumption in AoD scenarios as a starting point, how to drive the GPU to perform rendering without starting the CPU; furthermore, how to drive the GPU to achieve real-time interactive and personalized rendering effects based on user behavior events.

[0069] To facilitate understanding of the embodiments of this application, the following lists the scenarios in which the graphics processing device in this application is applied. For example, it may include the following multiple scenarios.

[0070] Please refer to Figure 1A, which is a schematic diagram of a lock screen animation scene provided in an embodiment of this application. In this scene, the mobile phone can display dynamic effects in the lock screen state. Users can select or customize different lock screen animation effects (such as animated emojis) according to their personal preferences, which not only increases the personalized customization options of the mobile phone, but also makes the lock screen interface more vivid and interesting. For example, when the user's terminal device (taking a mobile phone as an example) is currently in the lock screen state, as shown in the user interface 21 in Figure 1A, the lock screen interface includes customized emojis (such as winking emojis, sleepy emojis, surprised emojis, mischievous emojis, smiling emojis, or cute emojis, etc.). When the user raises the phone, shakes the phone, or performs other operations that change the phone's posture, the user interface 21 can present some feedback animation effects, as shown in the user interface 22 in Figure 1A. Compared with the user interface 21, some emojis in the user interface 22 have expanded, contracted, or changed in expression, presenting vivid, interesting, and delicate real-time interactive effects. For example, please refer to Figure 1B, which is a schematic diagram of another lock screen animation scenario provided by the embodiments of this application. In this scenario, the user's click operation is set to trigger the feedback animation effect. As shown in the user interface 31 in Figure 1B, when the phone is in the lock screen state, the user clicks on one of the Emoji expressions, and the Emoji expression changes from "yawning" to "playfully sticking out tongue" as shown in the user interface 32 in Figure 1B, realizing a personalized real-time interactive effect in the lock screen state.

[0071] In the feedback animation effects shown in Figure 1A or Figure 1B under the lock screen state, the mobile phone needs to implement effects such as the inflation and expansion of the Emoji sphere, collision and compression, or expression changes. The collision of the Emoji sphere requires real-time flexible body simulation and physical simulation calculations, as well as real-time ambient light and shadow. These all need to be driven by user behavior events (such as lifting the phone or tapping the phone screen) to drive the GPU to calculate and render in real time to achieve the corresponding animation effects (also referred to as motion effects). For example, the cartoon expressions in user interface 21 in Figure 1A or user interface 31 in Figure 1B can be generated by the processor in the graphics processing device of this application using a preset first graphics task command as a preset image task command template. Then, under user operation (such as shaking the phone or tapping the Emoji on the phone screen), the perception processing unit can modify the first graphics task command according to the user's operation behavior to obtain a second graphics task command, which is finally sent to the GPU for rendering to obtain the target motion effect (that is, the changed Emoji in user interface 22 or user interface 32).

[0072] Please refer to Figure 1C, which is a schematic diagram of an always-on display animation scene provided in an embodiment of this application. In this scene, assuming the user's mobile phone has the always-on display (AoD) function enabled, the user can view information such as date, time, or remaining battery power without turning on the screen after the screen is off. Furthermore, the user can also set interesting animated emoticons to achieve a personalized interactive always-on display. For example, as shown in user interface 50 in Figure 1C, the user has entered the always-on display interface, which displays a cartoon avatar wearing a hat and glasses, and the time, date, and remaining battery power below the cartoon avatar. When the user's gaze is directed from the left side of the phone screen, the scene shown in user interface 51 in Figure 1C is displayed, showing that the cartoon avatar's gaze moves from the center to the left as the user looks; when the user's gaze is directed from the right side of the phone, the scene shown in user interface 52 in Figure 1C is displayed, showing that the cartoon avatar's gaze moves from the left side to the right as the user looks. In other words, when the user's eyes move left and right, the cartoon avatar's eyes will also move left and right, as if interacting with the user's eyes, thus achieving a vivid and interesting always-on display animation.

[0073] In the aforementioned always-on display animation scenario, the mobile phone needs to achieve the effect of the cartoon avatar's eye movement. This eye movement requires real-time calculation of motion vectors or the provision of relevant parameters. These calculations and renderings are driven by user behavior events (such as the user's gaze direction) to drive the GPU in real-time calculation and rendering to achieve the corresponding animation effect. For example, the cartoon emoticon in user interface 50 in Figure 1C can generate a preset first graphics task command (i.e., a preset image task command template) through the processor in the graphics processing unit. The animation effect of the cartoon emoticon changing its gaze under the user's gaze can be obtained by modifying the first graphics task command based on the user's gaze behavior using a second graphics task command, which is then sent to the GPU for rendering to obtain the target animation effect (i.e., the cartoon emoticon with changed gaze in user interface 51 or user interface 52).

[0074] It should be noted that the graphics processing device in this embodiment can also be applied to other motion effect scenarios, such as other scenarios where mobile phones or other smart devices need to call the GPU for graphics processing in a low-power state. For example, when AI glasses play videos, they need to call the GPU to perform some distortion correction processing. Another example is that in low-power video playback scenarios, mobile phones need to use the MCU to drive the GPU to handle bullet comments, etc., which will not be listed here.

[0075] Based on the above, please refer to Figure 2A, which is a schematic diagram of a graphics processing device provided in an embodiment of this application. The graphics processing device 10 specifically includes a processor 101, a perception processing unit 102, and a graphics processor 103 (which may be simply referred to as GPU 103). Optionally, the graphics processing device 10 shown in Figure 2A can be a system-on-a-chip (SOC), and further, it can be located in any terminal device (the terminal devices referred to in this application are all devices containing the graphics processing device 10, which will not be repeated hereafter), such as computers, mobile phones, tablets, smart wearable devices, or mobile devices. The graphics processing device 10 can also specifically be a chip or chipset, or a circuit board carrying a chip or chipset, which can operate under necessary software drivers.

[0076] The processor 101 can be used to handle various computing tasks, such as running an operating system, applications, or processing system instructions to control multiple hardware or software components connected to the processor 101. Further, the processor 101 can also be used to coordinate the rendering process, execute algorithms and logic, prepare rendering instructions, or send instructions to the perception processing unit 102. Specifically, the processor 101 in this application can be pre-configured with the main rendering task description, i.e., generating offline rendering task commands (such as the first graphics task command). During subsequent online rendering, the online coordinated rendering process can be executed solely through the perception processing unit 102, without needing to start the processor 101. Optionally, the processor 101 can be a central processing unit (CPU), one of the CPU's processor cores or processing units, or other types of processors (such as a general-purpose processor or an accelerator with a specific function). Further, the processor 101 also includes a cache memory, which can be used to store instructions or data that the processor 101 has just used or is reusing. If the processor 101 needs to reuse the instruction or data, it can directly retrieve it from the cache memory. In some embodiments, if the graphics processing device 10 is a SoC, it can employ various different processor architectures, such as ARM, MIPS, x86, or RISC-V. It is understood that the processor 101 can also have other general-purpose functions, such as directly scheduling the GPU 103 to perform graphics rendering tasks in a wake-up state or normal operating state.

[0077] The perception processing unit 102 can acquire sensor data and process and analyze it to obtain user behavior events. Furthermore, the perception processing unit 102 can modify the main rendering task description generated by the processor 101 based on the user behavior events to achieve real-time interactive and personalized rendering effects. Optionally, the perception processing unit 102 can also be called a sensor control center / hub, or a processing unit that includes a sensor hub and other control logic. The main function of the sensor hub is to connect and process data from various sensors, including fusing data from different types of physical sensors, and decomposing or fusing data from multiple physical sensors into a virtual sensor as needed to meet various functional requirements. Furthermore, the perception processing unit 102 may specifically include a microcontroller unit (MCU), a communication interface, a signal processing unit, or a coprocessing unit. For example, the MCU can be used to generate auxiliary task descriptors, the communication interface can be used to receive sensor data (signals) reported by various sensors, the signal processing unit can be used to analyze and process the sensor signals reported by the sensors, and the coprocessing unit can be used to perform other types of calculations or processing tasks. In general, the perception processing unit 102 can take over some of the work of the processor 101 when the processor 101 is in hibernation, sleep, power-saving mode, or low-power state, assisting the GPU 103 in graphics rendering. For example, without starting the processor 101, the perception processing unit 102 can drive the GPU 103 to perform graphics rendering based on user behavior events, which can effectively reduce power consumption in some motion effect scenes (such as AoD motion effects).

[0078] The graphics processing unit 103 (GPU 103 for short) is responsible for graphics and image processing operations, including graphics rendering and graphics processing tasks. Specifically, the GPU 103 primarily handles graphics-related computational tasks, such as input, vertex processing, geometry processing, rasterization, pixel processing, output merging, and display, to ensure a smooth and clear visual experience for users when playing games, watching videos, browsing images, or performing other graphics-related operations. In this embodiment, the GPU 103 can be used, under the scheduling of the perception processing unit 102 or the scheduler 106 (described in detail in Figure 2C), to execute graphics rendering tasks based on a second graphics task command to obtain the image corresponding to the target animation. Optionally, if the graphics processing device 10 is a SoC, the GPU 103 is typically tightly integrated with the processor to achieve high-performance graphics processing. It is understood that the GPU 103 can also have other general graphics processing functions such as video encoding / decoding and rendering, which will not be described in detail here.

[0079] Further, in one possible implementation, please refer to Figure 2B, which is a schematic diagram of another graphics processing device provided in an embodiment of this application. This graphics processing device 10 may further include a display subsystem 104, and optionally, it may also include a storage unit 105. In some embodiments, the display subsystem 104 is also coupled to the display screen 20, and the perception processing unit 102 is also coupled to various sensors 30.

[0080] The Display Subsystem (DSS) 104 is a hardware block responsible for retrieving pixel data (such as a first image) from the storage unit 105, processing it to obtain an image for final display (such as a second image), and finally sending it to the display screen 20 (such as an LCD panel or a DisplayPort monitor). Exemplarily, the DSS hardware can be divided into two main parts: a Display Controller (DISPC) for processing the acquired pixel data, performing color conversion, compositing (such as rotation, scaling, or layer overlay), and other pixel domain operations; and peripherals for encoding the raw pixel data into standard display signals, such as Mobile Industry Processor Interface (MIPI), Dots Per Inch (DPI), or Device Independent Pixel (DP).

[0081] Storage unit 105 can be a power-loss volatile memory, whose stored contents are lost when power is off. In this application, storage unit 105 can be read-write running memory, also known as internal memory, or simply main memory. Its function is to temporarily store computational data in processor 101, and to temporarily store data exchanged between processor 101 and perception processing unit 102, graphics processor 103, other functional modules, or external memory. It can serve as a storage medium for temporary data of the operating system or other running programs. For example, the operating system on processor 101 sends a pre-generated preset image task command template (i.e., the first graphics task command) to storage unit 105 for storage. The second graphics task command obtained by perception processing unit 102 after modifying the image task command template can also be stored in storage unit 105. Furthermore, the first image rendered by GPU 103 can also be stored in storage unit 105. Optionally, the storage unit 105 may include a GPU rendering task description area and a frame buffer area. The former stores image task commands to be executed by the GPU, and the latter stores image sequence frames to be displayed by the display subsystem. For example, the storage unit 105 may include one or more of dynamic random access memory (DRAM), static random access memory (SRAM), and synchronous dynamic random access memory (SDRAM). DRAM further includes Double Data Rate Synchronous Dynamic Random Access Memory (DDR SDRAM), second-generation Double Data Rate Synchronous Dynamic Random Access Memory (DDR2), third-generation Double Data Rate Synchronous Dynamic Random Access Memory (DDR3), and fourth-generation Low Power Double Data Rate Synchronous Dynamic Random Access Memory (LPDDR4x).

[0082] The display screen 20 is used to display images or videos, etc. The display screen 20 may include a display panel, which may be a liquid crystal display (LCD), an organic light-emitting diode (OLED), an active-matrix organic light-emitting diode (AMOLED), a flexible light-emitting diode (FLED), a MiniLED, a MicroLED, a Micro-OLED, a quantum dot light-emitting diode (QLED), etc. In some embodiments, the terminal device containing the graphics processing device 10 may include one or more display screens 20.

[0083] Various sensors 30, exemplarily, may include gyroscope sensors, accelerometers, touch sensors, infrared sensors, etc. For example, a gyroscope sensor can be used to determine the motion posture of the terminal device, and an accelerometer can be used to detect the magnitude of the terminal device's acceleration in various directions (generally three axes). When the terminal device is stationary, the magnitude and direction of gravity can be jointly detected to identify the terminal device's posture. For example, in the scenario corresponding to Figure 1A above, the posture data of the terminal device can be detected by the gyroscope sensor and the accelerometer, and the detected posture data can be transmitted to the perception processing unit 102 to determine the posture change of the terminal device (i.e., user behavior events). Finally, the display screen 20 displays the collision of Emoji expressions or expression changes (i.e., target animations) that match the posture change of the terminal device. For example, a touch sensor can be mounted on the display screen 20. The touch sensor and display screen 20 together form a touchscreen to detect touch operations applied to or near the screen. The detected touch operations are then transmitted to the perception processing unit 102 to determine the type of touch event (i.e., user behavior event). Finally, the display screen 20 provides visual output (i.e., target animation) related to the touch operation. For instance, in the scenario corresponding to Figure 1B, the system can detect when a user clicks on an Emoji on the screen, triggering a change in the corresponding Emoji. Another example is an infrared camera that can capture the reflection of infrared light from the user's eyes and analyze the reflected signals using image processing algorithms to determine information such as eye position, movement, and pupil dilation. For instance, in the scenario corresponding to Figure 1C, the system can trigger a corresponding change in the cartoon emoji's gaze based on the user's orientation while looking at the phone screen.

[0084] Furthermore, when the aforementioned graphics processing device 10 is located on a system-on-a-chip (SoC), the SoC may also include a memory controller, input / output (I / O) interfaces, communication interfaces, external memory, or signal processors. The memory controller is responsible for managing system memory access, including read and write operations; the I / O interfaces (such as USB, HDMI, PCIe) can be used to communicate with external devices (such as monitors, keyboards, or mice); the communication interfaces are responsible for handling wireless and wired communication functions (such as Wi-Fi, Bluetooth, or Ethernet); and the memory (which may include non-volatile memory (such as flash memory) and volatile memory (such as DRAM)) is used to store program code and data.

[0085] In Figure 2A above, the processor 101, the perception processing unit 102, and the graphics processor 103 can be located on the same chip. Optionally, in Figure 2B above, the processor 101, the perception processing unit 102, the graphics processor 103, and the display subsystem 104 can be located on the same chip. Optionally, in Figure 2C above, the processor 101, the perception processing unit 102, the graphics processor 103, the display subsystem 104, and the scheduler 106 can be located on the same chip.

[0086] In various embodiments of this application, a semiconductor chip, also simply referred to as a chip, can be a collection of integrated circuits fabricated using integrated circuit technology on an integrated circuit substrate (typically a semiconductor material such as silicon), and its outer layer is usually encapsulated by a semiconductor packaging material. The integrated circuit may include metal-oxide-semiconductor (MOS) transistors, bipolar transistors, or diodes, etc. A semiconductor chip can operate independently or under the action of necessary driving software to realize various functions such as communication, computing, or storage.

[0087] In another possible implementation, the graphics processing device 10 of this application may also be configured with a scheduler. Please refer to FIG2C, which is a schematic diagram of another graphics processing device provided in an embodiment of this application. The graphics processing device 10 shown in FIG2C is based on the graphics processing device 10 shown in FIG2B with the addition of a scheduler 106, wherein...

[0088] Scheduler 106 can be used to distribute the rendering task scheduling between the perception processing unit 102 and the graphics processor 103. For example, the perception processing unit 102 first submits a set of rendering task pointers to scheduler 106, and scheduler 106 resolves the dependencies and submits pointers to specific rendering tasks to GPU 103. That is, after receiving the pointer sets corresponding to multiple second graphics task commands sent by the perception processing unit 102, scheduler 106 can be responsible for coordinating the entire rendering process, including allocating image task commands to the graphics processor 103 or managing data flow, thus replacing the perception processing unit 102 in controlling the rendering process and ensuring that each rendering step is executed in sequence. By adding scheduler 106, a lightweight graphics driver software stack can be implemented on the perception processing unit 102 to reduce the load on the perception processing unit 102.

[0089] It is understood that the interface connection relationships between the modules illustrated in the embodiments of this application are merely illustrative and do not constitute a structural limitation on the graphics processing device 10 or the terminal device containing the graphics processing device 10. In other embodiments of this application, the graphics processing device 10 may also employ different interface connection methods or combinations of multiple interface connection methods as described in the above embodiments. The structures illustrated in Figures 2A, 2B, and 2C do not constitute a specific limitation on the graphics processing device 10. In other embodiments of this application, the graphics processing device 10 may include more or fewer components than illustrated, or combine certain components, or split certain components, or arrange different components. The illustrated components may be implemented in hardware, software, or a combination of software and hardware.

[0090] Based on the application scenarios shown in Figures 1A-1C and the structure of any one of the graphics processing devices 10 in Figures 2A-2C, the functions implemented by the graphics processing device 10 in the embodiments of this application are specifically described, which may include the following:

[0091] The processor 101 is used to generate a preset first graphics task command; the perception processing unit 102 is used to acquire sensor data, process the sensor data to obtain user behavior events, and modify the first graphics task command according to the user behavior events to obtain a second graphics task command; the graphics processor 103 is used to perform graphics rendering based on the second graphics task command to obtain a first image.

[0092] Specifically, the processor can pre-generate a preset first graphics task command under normal operating conditions. This first graphics task command is used by the GPU 103 to render a first image. The first graphics task command may specifically include a task descriptor (also called a task description) composed of a series of rendering commands and related rendering parameters (such as global variables or state parameters). The perception processing unit 102 acquires various sensor data collected by various sensors 30 and processes and analyzes the sensor data (such as accelerometer data, touch sensor data, or infrared sensor data) to obtain user behavior events (such as raising the phone, tapping the phone screen, or looking at the phone screen) used to trigger target animations. Further, the perception processing unit 102 modifies the first graphics task command based on the analyzed user behavior events. For example, it first generates an auxiliary task descriptor based on the user behavior events, and then modifies the main task descriptor based on the main task descriptor, thereby generating a GPU rendering task descriptor that matches the user behavior events. Modifying the task command can include altering relevant parameters in the first graphics task command, such as adding, modifying, or deleting parameters. For example, filling in data such as a Uniform buffer. The perception processing unit 102 can modify the task command directly from the first graphics task command stored in the storage unit 105. Alternatively, the perception processing unit 102 can read the command from the storage unit 105, modify it, and then send the modified second graphics task command to the storage unit for storage. Subsequently, the perception processing unit can send the address information or pointer set (or pointer) of the second graphics task command stored in the storage unit 105 to the graphics processor 103. The graphics processor 103 can then read the second graphics task command from the storage unit 105 to execute a rendering task. Specifically, this may include parsing the second graphics task command, generating GPU rendering instructions, and performing calculations or rendering based on the GPU rendering instructions and relevant rendering parameters. For example, a series of multiple first graphics task commands can also be called rendering task descriptors. A series of first images can form the image sequence frames required for the target animation. That is, the GPU can render multiple first images, thereby rendering the image sequence frames corresponding to the target animation based on the user's pre-configured and selected settings.

[0093] For example, please refer to Figure 3A. Figure 3A is a schematic flowchart of constructing a main task description in the offline configuration stage according to an embodiment of this application. Specifically, the simplified steps of the processor constructing the main task description offline may include:

[0094] S30 allows users to configure rendering tasks through the application. For example, users can select target animations for corresponding scenarios based on their interests, such as setting the phone's animation on the lock screen to the Emoji expressions in Figure 1A or Figure 1B above, or setting the phone's animation in the always-on display state to the cartoon expressions in Figure 1C above.

[0095] S31, the processor 101 constructs a main rendering task description (also known as a main task descriptor) for the GPU 103. For example, when the processor 101 is in a wake-up state (or normal working state), it pre-constructs a main task descriptor corresponding to the target animation effect selected by the user's configuration. The main task descriptor may include the descriptor of the preset template of the target animation effect.

[0096] S32, the processor 101 transmits the main rendering task description pointer set to the perception processing unit 102. For example, the processor 101 transmits the first address information (such as the main rendering task description pointer set) corresponding to the main rendering task description to the perception processing unit 102. Optionally, the processor 101 and the perception processing unit 102 may agree on the storage address in advance, so that the processor 101 does not need to send the first address information to the perception processing unit 102.

[0097] S33, processor 101 stores the main rendering task description of GPU 103. For example, processor 101 stores a pre-constructed main task descriptor in the storage unit 105 at an address or location corresponding to the main rendering task description pointer set.

[0098] This application provides a graphics processing device applicable to graphics rendering scenarios such as mobile phone always-on display animations. Specifically, it divides the image processing task into a preset template part (such as a first graphics task command) and a part that needs to be modified based on user behavior events. The modified task command (such as a second graphics task command) is then used as a GPU rendering task to obtain the final display image (i.e., the first image). In other words, the processor 101 does not need to participate online during the graphics rendering task. Only the template part generated offline by the processor 101 and the slight modification of the template part by the perception processing unit 102 based on user behavior events are needed to obtain the task that the GPU 103 needs to render. Since this process does not require the processor 101 to participate online (e.g., it does not need to wake up the processor 101 in a low-power state), it can greatly save energy. Furthermore, since this rendering task is a real-time response based on user needs, it can meet the rendering effect of ensuring real-time interaction of the GPU. In summary, this application proposes a high-efficiency rendering scheme that combines offline and online processing. This scheme can effectively reduce processor load, avoid waking up the processor, thereby reducing device power consumption. Furthermore, it can drive the GPU to achieve a high-end and refined rendering effect that allows real-time interaction with the user, greatly enhancing the user experience.

[0099] In one possible implementation, the graphics processing device 10 further includes a storage unit 105; the processor 101 is further configured to store the first graphics task command in the storage unit; the perception processing unit 102 is specifically configured to modify the first image task command in the storage unit according to the user behavior event to obtain a second graphics task command. Optionally, the storage unit may be DRAM storage, and outside of the SOC, the storage unit 105 is not limited to DRAM, but may also be on-chip storage or other storage on the SOC.

[0100] In this embodiment, the processor 101 sends pre-generated image task commands to the storage unit 105 for pre-storage. In this way, when the perception processing unit 102 needs to make corrections based on the pre-generated image task commands after a user behavior event is triggered, it can read them from the storage unit 105 for modification, or it can directly modify the task commands stored in the storage unit 105 online. This enables the processor 101 to generate templates offline and the graphics processor 103 to achieve a refined rendering effect that interacts with the user in real time online.

[0101] In one possible implementation, the preset first graphical task command includes a preset image task command template, the user behavior event is used to trigger the target animation effect, and the second graphical task command is an image task command capable of implementing the target animation effect. The target animation effect can be a screen-off animation effect or a lock screen emoji animation effect, etc. For example, a screen-off animation effect refers to displaying visual effects through dynamic effects when the phone screen is off, making the screen-off process or the interface after closing more vivid and interesting. Such effects typically include animations or transition effects, allowing the phone to exhibit a personalized visual experience even when the screen is off. For example, a lock screen emoji animation effect refers to setting up a follow-up emoji function on the phone screen so that the emojis or patterns on the lock screen interface can change with the user's movements or gaze, increasing the fun of using the phone and improving the user experience. In this embodiment, the processor 101 pre-generates the preset first graphical task command, which specifically includes an image task command template for implementing the target animation effect, that is, a fixed or basic command part for rendering the target animation effect. The sensor data acquired by the perception processing unit 102, and the user behavior events obtained based on the analysis of this sensor data, are the relevant operations used to trigger the target animation effect (such as the user's manual operation or eye gaze). Correspondingly, the second graphics task command obtained by the perception processing unit 102 after modifying the preset first graphics task command is the task command that can realize the target animation effect. That is, the processor 101 pre-generates the template part of the rendering task, and the perception processing unit 102 modifies the template online to obtain the rendering task command with user interaction feedback, so as to ensure the rendering effect of real-time interaction and at the same time meet the low power consumption requirements.

[0102] In one possible implementation, the processor 101 is further configured to enter a sleep state after generating the first graphics task command. The sleep state may include power-saving mode, low-power mode, standby state, or hibernation state. For example, the phone may enter a screen-off state or a locked screen state. That is, the processor 101 can generate the first graphics task command in advance, and then enter a sleep state under certain conditions or triggered by user operation. Furthermore, during subsequent interactions between the perception processing unit 102 and the graphics processor 103, or during interactions between the perception processing unit 102, the scheduler 106, and the graphics processor 103, the processor 101 does not need to be woken up and can continue to maintain the sleep state, i.e., the low-power state. For example, when the terminal device containing the graphics processing device 10 is in a screen-off display state and receives sensor data reported by the sensor, the processor 101 is in a sleep state, while the perception processing unit 102 and the graphics processor 103 are all in working state. In this embodiment, after generating the preset template task command, the processor 101 can enter a sleep state. During the subsequent rendering and generation of the first image by the graphics processor 103, it does not need to participate or interact. That is, the processor 101 can continue to be in a sleep state (such as hibernation, standby, or other low-power states, without being woken up), which can greatly save power consumption. At the same time, since there is no need to go through the process of waking up the processor or interact with the processor 101, the rendering efficiency of the GPU can be improved.

[0103] In one possible implementation, the perception processing unit 102 is always online. Since the perception processing unit 102 needs to acquire data reported by external sensors and process and analyze the sensor data in real time to determine specific user behavior events, it can always be online. That is, if the terminal device is powered on, the perception processing unit 102 is always working and does not need to be woken up, allowing it to wait in real time to process transactions. Optionally, when the perception processing unit 102 analyzes and obtains a user behavior event, the processor 101 can remain in sleep mode, while both the perception processing unit 102 and the graphics processor 103 remain working. Optionally, during the stage where the perception processing unit 102 determines the user behavior event and subsequently schedules the GPU for rendering, the processor 101 can also be in normal working mode. That is, if the processor 101 is not in sleep mode, it can remain in that state, but no interaction from the processor 101 is required during this stage, thus reducing power consumption during this stage. In this embodiment, the sensing processing unit 102 can always be online to acquire sensor data in real time and process the sensor data to obtain user behavior events. That is, the sensing processing unit 102, which is always online, can monitor user operations and other behaviors in real time to respond to user needs and improve user experience.

[0104] In one possible implementation, the processor 101, graphics processor 103, and perception processing unit 102 are integrated on the same chip. For example, the processor 101, graphics processor 103, and perception processing unit 102 are integrated on the same SoC. In this embodiment, the processor 101, graphics processor 103, and perception processing unit 102 are all integrated on the same chip, that is, they belong to the same system-on-a-chip (SoC), thereby further reducing signal transmission between components, improving data transmission efficiency, reducing overall power consumption, and reducing system complexity and cost, thus comprehensively improving the performance of the SoC.

[0105] In one possible implementation, the graphics processing device 10 further includes a display subsystem 104 for performing display processing on the first image to obtain a second image, which is then displayed on the display screen 20. The display processing operations performed by the display subsystem (DSS) 104 may specifically include one or more of the following pixel-domain operations: color and brightness conversion and compositing, rotation, scaling (or stretching), or layer overlay. Further, it may also include encoding the original pixel data into a standard display signal. It is understood that the display subsystem 104 also obtains an image sequence frame after processing, but it is an image sequence frame that has undergone dynamic range processing. In this embodiment, the graphics processing device 10 further includes a display subsystem 104, which is used to perform further display processing (e.g., rotation, scaling, layer overlay, etc.) on the first image rendered by the graphics processor to better and more appropriately display it on the display screen 20.

[0106] In one possible implementation, storage unit 105 may include a rendering task description area and a frame buffer; the rendering task description area can be used to store the main task descriptor and the GPU rendering task descriptor; the frame buffer can be used to store the first image sequence frames. That is, storage unit 105 can be logically divided into two areas, one for storing task descriptors and the other for storing GPU-rendered image sequence frames, so that the perception processing unit 102, graphics processor 103, or display subsystem 104 can more efficiently access data in the corresponding areas (GPU rendering task description area or frame buffer) of storage unit 105.

[0107] In one possible implementation, the first graphics task command includes multiple first graphics task commands, and the second graphics task command includes multiple second graphics task commands. The perception processing unit 102 is specifically used to schedule the current image task command among the multiple second graphics task commands to the graphics processor. The graphics processor 103 is specifically used to perform graphics rendering based on the current image task command to obtain the current image. In this embodiment, if the first graphics task command includes multiple first graphics task commands, and correspondingly the second graphics task command includes multiple second graphics task commands, then in the process of specifically sending the second graphics task commands to the graphics processor 103 for rendering, the perception processing unit 102 can schedule multiple second graphics task commands to the graphics processor 103 respectively, and the graphics processor 103 can then render the current graphics (such as the current first image) according to the image task command scheduled by the perception processing unit 102. Then, the perception processing unit 102 can schedule the next second graphics task command to the graphics processor 103 and repeat the above process until the graphics processor 103 has executed all the second graphics task commands to render all the first images corresponding to the target animation. Thus, the scheduling and execution of multiple second graphics task commands are realized through the interaction between the perception processing unit 102 and the graphics processor 103.

[0108] In one possible implementation, the first graphics task command includes multiple first graphics task commands, and the second graphics task command includes multiple second graphics task commands. The graphics processing device 10 further includes: a scheduler 106, used to schedule the current image task command among the multiple second graphics task commands to the graphics processor 103; the graphics processor 103 is specifically used to perform graphics rendering based on the current image task command to obtain the current image. In this embodiment, the first graphics task command includes multiple first graphics task commands, and correspondingly, the second graphics task command includes multiple second graphics task commands. Then, during the process of the perception processing unit 102 sending the second graphics task commands to the graphics processor 103 for rendering, the scheduler 106 can schedule multiple second graphics task commands to the graphics processor 103 respectively. The graphics processor 103 can then render the current graphics (such as the current first image) according to the image task commands scheduled by the scheduler 106. Then, scheduler 106 can schedule the next second graphics task command to graphics processor 103, repeating the above process until graphics processor 103 has executed all second graphics task commands to render all the first images corresponding to the target animation. Thus, the scheduling and execution of multiple second graphics task commands are achieved through the interaction between the added scheduler 106 and graphics processor 103. This avoids excessive involvement of the perception processing unit 102 in the scheduling and rendering process, saving the power consumption of the perception processing unit 102. In summary, by triggering GPU 103 to execute rendering through the added scheduler 106, the graphics software stack running on the perception processing unit 102 can be further streamlined.

[0109] For example, please refer to Figure 3B, which is a schematic diagram of the rendering and display process in an online rendering stage according to an embodiment of this application. Specifically, the simplified steps of the online rendering and display stage may include:

[0110] S34, User behavior triggers perception processing unit 102. For example, after acquiring sensor data, perception processing unit 102 analyzes the sensor data to obtain user behavior events, and these user behavior events are used to trigger target animation effects.

[0111] S35, the perception processing unit 102 generates an auxiliary task description. For example, the perception processing unit 102 generates an auxiliary task descriptor based on user behavior events, and reads the main task descriptor from the storage unit 105 based on the aforementioned first address information; it then modifies the main task descriptor based on the auxiliary task descriptor to generate a rendering task descriptor corresponding to the target animation effect, and stores it in the storage unit 105.

[0112] S36, the perception processing unit 102 submits the rendering task pointer set to the scheduler 106. For example, after storing the rendering task descriptor in the storage unit 105, the perception processing unit 102 sends the second address information (such as the rendering task pointer set) corresponding to the rendering task descriptor to the scheduler 106.

[0113] S37, scheduler 106 resolves dependencies and submits pending rendering task pointers to GPU 103. For example, scheduler 106 resolves the dependencies of the pointer set, generates multiple rendering task pointers corresponding to the rendering task descriptor, and sends them to graphics processor 103; each of the multiple rendering task pointers points to a subtask descriptor in the rendering task descriptor; scheduler 106 sends the current rendering task pointer from the multiple rendering task pointers to graphics processor 103.

[0114] S38, GPU 103 retrieves the task description based on the task pointer and performs rendering. For example, GPU 103 reads the sub-rendering task descriptor corresponding to the current rendering task pointer from storage unit 105; renders the image or image sequence frame corresponding to the sub-rendering task descriptor, and sends the corresponding feedback command to the scheduling unit. Based on the feedback command, scheduler 106 sends the next rendering task pointer from multiple rendering task pointers to GPU 103 until GPU 103 has completed all rendering tasks.

[0115] S39, the scheduler 106 notifies the perception processing unit 102 that rendering is complete. For example, after receiving feedback commands corresponding to all rendering task pointers, the scheduler 106 sends a notification command to the perception processing unit 102.

[0116] S310, the display subsystem 104 completes the rendering and display of the image. For example, in response to the notification command, the perception processing unit 102 sends a control command to the display subsystem 104; in response to the control command, the display subsystem 104 reads the first image sequence frame from the storage unit 105 and processes it to obtain the second image sequence frame, and then displays it based on the second image sequence frame.

[0117] In this embodiment, the rendering task is divided into offline configuration and online fine-tuning. The offline configuration is pre-generated by the processor; in the online phase, the perception processing unit generates auxiliary descriptors based on user interaction events; these are then overlaid with the GPU main task descriptor generated in the offline phase to collaboratively complete the GPU rendering task description. In summary, this embodiment proposes a high-efficiency rendering method for mobile SoCs that combines offline and online collaboration. This method can be applied to AoD applications such as mobile phones and tablets. In the offline phase, the processor generates the GPU main task description; during online rendering, the perception processing unit triggers the GPU to execute rendering without activating the processor (such as the main CPU), ensuring real-time interactive rendering effects while simultaneously meeting low-power requirements.

[0118] For example, please refer to Figure 4A. Figure 4A is a schematic diagram of the control path and data path architecture design within a graphics processing device according to an embodiment of this application. In the graphics processing device shown in Figure 4A, the processor 101 is taken as a CPU, and the storage unit 105 is taken as memory. For example, a complete graphics processing solution flow may include:

[0119] Step 1: The processor 101 generates a preset image task command template (i.e., including multiple first graphics task commands) in advance under normal working conditions; and stores the preset image task command template in memory through data path 1; further, the CPU also transmits the corresponding task description pointer set to the perception processing unit 102 through control path 0.

[0120] Step 2: The perception processing unit 102 captures user behavior events based on user behavior, generates an auxiliary task descriptor (e.g., filling in Uniform buffers with Data), and coordinates with the main task descriptor to generate a GPU rendering task descriptor 103 that matches the user behavior events. In other words, the perception processing unit 102 uses data path 2 to modify the main task descriptor stored in memory using the generated auxiliary task descriptor, thereby obtaining the GPU rendering task descriptor.

[0121] Step 3: The perception processing unit 102 controls the graphics processor 103 to perform graphics rendering through the control path 1. This includes sending the rendering task pointer to the GPU 103 through the control path 1, and receiving the rendering progress feedback from the GPU 103 (such as feedback interrupt signals, synchronization signals, etc.) through the control path 1.

[0122] Step 4: After receiving the rendering task pointer through control path 1, GPU 103 retrieves the GPU rendering task description stored in memory through data path 3 based on the pointer, and performs graphics rendering (e.g., parsing the task description to generate GPU rendering instructions, and performing calculations or rendering based on the GPU rendering instructions). Further, GPU 103 stores the rendered image in the frame buffer area in memory through data path 4; and further still, GPU 103 notifies the perception processing unit 102 through control path 1 that it has completed the graphics rendering task.

[0123] Step 5: After receiving the rendering completion notification sent by the GPU 103 through the control path 1, the perception processing unit 102 sends an instruction to the display subsystem 104 through the control path 2 to configure the display subsystem 104 to perform display.

[0124] Step 6: The display subsystem 104 reads the first image sequence frame (i.e., one or more first images) that has been rendered by the GPU 103 from the frame buffer area in memory through the data path 5, and performs display processing on it to obtain the second image sequence frame (i.e., one or more second images) for display on the display screen 20.

[0125] It is understood that the above steps are merely illustrative descriptions of how a graphics processing device implements a corresponding graphics processing scheme through the corresponding control and data paths. The specific implementation process can be adapted to different application scenarios or different needs, and will not be listed here.

[0126] For example, please refer to Figure 4B. Figure 4B is a schematic diagram of another control path and data path architecture design in a graphics processing device provided in an embodiment of this application. In the graphics processing device shown in Figure 4B, the processor is taken as a CPU, and the storage unit is taken as memory. For example, the graphics processing solution flow is as follows:

[0127] Step 1: The processor 101 generates a preset image task command template (i.e., including multiple first graphics task commands) in advance under normal working conditions; and stores the preset image task command template in memory through data path 1; further, the CPU also transmits the corresponding task description pointer set to the perception processing unit 102 through control path 0.

[0128] Step 2: The perception processing unit 102 captures user behavior events based on user behavior, generates an auxiliary task descriptor (e.g., filling in Uniform buffers with Data), and coordinates with the main task descriptor to generate a GPU rendering task descriptor 103 that matches the user behavior events. In other words, the perception processing unit 102 uses data path 2 to modify the main task descriptor stored in memory using the generated auxiliary task descriptor, thereby obtaining the GPU rendering task descriptor.

[0129] Step 3: The perception processing unit 102 sends the GPU rendering task pointer set to the scheduler 106 through control path 1. The scheduler 106 first resolves the dependencies of the rendering task pointer set, and then schedules the pending rendering task pointers to the GPU 103 through control path 2. Correspondingly, the GPU 103 feeds back the rendering progress to the scheduler 106 through control path 3. Furthermore, based on the rendering progress fed back by the GPU 103, the scheduler 106 schedules the next rendering task pointer through control path 2. The above process is repeated cyclically until the GPU 103 has finished rendering all GPU rendering tasks.

[0130] Step 4: After receiving the rendering task pointer through control path 2, GPU 103 retrieves the GPU rendering task description stored in storage unit 105 through data path 3 based on the pointer, and performs graphics rendering (e.g., parsing the task description to generate GPU rendering instructions, and performing calculations or rendering based on the GPU rendering instructions). Further, GPU 103 stores the rendered image in the frame buffer area in memory through data path 4; and further still, GPU 103 notifies scheduler 106 through control path 3 that it has completed graphics rendering.

[0131] Step 5: GPU 103 notifies scheduler 106 through data path 3 that it has completed all rendering tasks. Then scheduler 106 relays to perception processing unit 102 through control path 4 that GPU 103 has completed the rendering tasks.

[0132] Step 6: After receiving the notification from the scheduler 106 that the GPU 103 has completed the rendering task via the control path 4, the perception processing unit 102 sends an instruction to the display subsystem 104 via the control path 5 to configure the display subsystem 104 to perform display.

[0133] Step 7: The display subsystem 104 reads the first image sequence frame (i.e., one or more first images) that has been rendered by the GPU 103 from the frame buffer area in memory through the data path 5, and performs display processing on it to obtain the second image sequence frame (i.e., one or more second images) for display on the display screen 20.

[0134] It is understood that the above steps are merely illustrative descriptions of how a graphics processing device implements a corresponding graphics processing scheme through the corresponding control and data paths. The specific implementation process can be adapted to different application scenarios or different needs, and will not be listed here.

[0135] In one possible implementation, processor 101 is further configured to pre-construct the main task descriptor of the target animation effect through a first software stack; perception processing unit 102 is specifically configured to generate an auxiliary task descriptor of the target animation effect and a rendering task descriptor of the target animation effect through a second software stack; wherein the first software stack and the second software stack are different. This application embodiment not only separates offline and online rendering tasks but also separates the software stacks of processor 101 and perception processing unit 102, mainly in the generation of task descriptors. The software stack typically includes components such as an operating system, architecture layer, protocol, runtime environment, or database system. These technologies are layered to form a complete technology stack, supporting software development and operation.

[0136] Furthermore, this application provides a lightweight, discrete graphics software stack design to reduce the load on the perception processing unit running the graphics software stack.

[0137] In some embodiments of this application, the graphics software stack required for GPU 103 rendering is divided into two parts: one part is the graphics software stack running on processor 101 (such as CPU), and the other part is the graphics software stack running on perceptual processing unit 102. For example,

[0138] Please refer to Figure 5A, which is a schematic diagram of a graphics software stack running on a processor according to an embodiment of this application. The Always-on Display (AoD) application, based on the user's selection, has the processor calling the GPU user-mode graphics driver (GPU_DSA_DDK) to construct and store GPU tasks, while simultaneously using a compiler to compile the constructed task descriptions. Then, GPU_DSA_DDK calls the GPU kernel-mode graphics driver (GPU_KMD) to create / destroy contexts and allocate memory, etc.

[0139] Please refer to Figure 5B, which is a schematic diagram of a graphics software stack running on a perception processing unit according to an embodiment of this application. A user interaction event triggers the MCU in the perception processing unit 102 to call GPU_DSA_DDK to generate an auxiliary task description, and combines it with the main task description already generated by the processor 101 (e.g., CPU) to construct a GPU rendering task description that matches the user behavior event. The GPU user-mode graphics driver (GPU_DSA_DDK) running on the perception processing unit 102 calls the GPU kernel-mode graphics driver (GPU_KMD) to perform Memory Management Unit (MMU) address management, power-on / off management, exception handling, etc. GPU_DSA_DDK also issues tasks to the GPU, driving the GPU to perform rendering, computation, etc. Optionally, based on the scheduler 106, the graphics software stack running on the perception processing unit 102 integrates Schedule_core to further lightweight the graphics application programming interface (API). GPU_DSA_DDK calls the scheduler's driver (Schedule_core) to issue rendering tasks and de-dependencies on GPU tasks.

[0140] In this embodiment, a lightweight, separate graphics driver software stack is designed to run on the CPU and the perception processing unit respectively. Optionally, the perception processing unit triggers the GPU to perform rendering through a hardware scheduling unit, further lightweighting the graphics software stack running on the perception processing unit.

[0141] This application enables high-quality rendering effects with real-time interactivity at a lower cost. The beneficial effects of the technical solution in this application include at least one or more of the following:

[0142] 1. In some embodiments of this application, CPU wake-up can be avoided, creating a lightweight graphics software stack and reducing the software load on the perception processing unit.

[0143] 2. In some embodiments of this application, it can help HarmonyOS to have refined native features, and provide complex AoD animations such as 3D digital clock, text signature, and physics engine animations without waking up the CPU, thereby improving the visual playability and refinement experience through user interaction.

[0144] 3. In some embodiments of this application, the always-on display experience can be improved by triggering the GPU through the perception processing unit to complete high-end and sophisticated motion rendering, thereby saving CPU power consumption. Based on an AoD user triggering motion effects for 1 hour per day, approximately 50mAh of CPU wake-up power consumption can be saved.

[0145] Please refer to Figure 6, which is a schematic flowchart of a graphics processing method provided in an embodiment of this application. This graphics processing method can be applied to a graphics processing device, a terminal device including the graphics processing device, etc. The graphics processing device may include a processor, a graphics processor, and a perception processing unit. The method may include the following steps S601-S603:

[0146] Step S601: Generate a preset first graphics task command through the processor.

[0147] Step S602: Obtain sensor data through the perception processing unit, process the sensor data to obtain user behavior events, and modify the first graphics task command according to the user behavior events to obtain the second graphics task command.

[0148] Step S603: The graphics processor performs graphics rendering based on the second graphics task command to obtain a first image.

[0149] In one possible implementation, the apparatus further includes a storage unit; the method further includes: storing the first graphics task command in the storage unit via the processor; and modifying the first graphics task command according to the user behavior event via the perception processing unit to obtain a second graphics task command, which includes: modifying the first image task instruction in the storage unit according to the user behavior event via the perception processing unit to obtain a second graphics task command.

[0150] In one possible implementation, the preset first graphic task command includes a preset image task command template, the user behavior event is used to trigger the target animation effect, and the second graphic task command is an image task command capable of realizing the target animation effect.

[0151] In one possible implementation, the method further includes: the processor entering a sleep state after generating the first graphics task command.

[0152] In one possible implementation, the apparatus further includes a display subsystem; the method further includes: performing display processing on the first image through the display subsystem to obtain a second image, the second image being displayed on a display screen.

[0153] In one possible implementation, the first graphics task command includes a plurality of first graphics task commands, and the second graphics task command includes a plurality of second graphics task commands; the method further includes: scheduling the current image task command among the plurality of second graphics task commands to the graphics processor through the perception processing unit; the step of performing graphics rendering based on the second graphics task command through the graphics processor to obtain a first image includes: performing graphics rendering based on the current image task command through the graphics processor to obtain a current image.

[0154] In one possible implementation, the apparatus further includes a scheduler; the first graphics task command includes a plurality of first graphics task commands, and the second graphics task command includes a plurality of second graphics task commands; the method further includes: scheduling the current image task command among the plurality of second graphics task commands to the graphics processor through the scheduler; the step of performing graphics rendering based on the second graphics task command through the graphics processor to obtain a first image includes: the graphics processor is specifically configured to perform graphics rendering based on the current image task command to obtain a current image.

[0155] It should be noted that the specific flow of the graphics processing method described in the embodiments of this application can be found in the relevant descriptions in the embodiments of the application shown in Figures 1A-5B above, and will not be repeated here.

[0156] This application also provides a computer-readable storage medium, wherein the computer-readable storage medium may store a program, which, when executed by a computing device, includes some or all of the steps described in any of the above method embodiments.

[0157] This application also provides a computer program product including instructions that, when executed by a computing device, enable the computing device to perform some or all of the steps of any artificial intelligence processing method.

[0158] In the above embodiments, the descriptions of each embodiment have different focuses. For parts not described in detail in a certain embodiment, please refer to the relevant descriptions in other embodiments.

[0159] It should be noted that, for the sake of simplicity, the foregoing method embodiments are all described as a series of actions. However, those skilled in the art should understand that this application is not limited to the described order of actions, as some steps may be performed in other orders or simultaneously according to this application. Furthermore, those skilled in the art should also understand that the embodiments described in the specification are preferred embodiments, and the actions and modules involved are not necessarily essential to this application.

[0160] In the several embodiments provided in this application, it should be understood that the disclosed apparatus can be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative; for instance, the division of the units described above is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be through some interfaces; the indirect coupling or communication connection between devices or units may be electrical or other forms.

[0161] The units described above as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0162] Furthermore, the functional units in the various embodiments of this application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit.

[0163] If the integrated units described above are implemented as software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or all or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which can be a personal computer, server, or network device, specifically a processor in the computer device) to execute all or part of the steps of the methods described in the various embodiments of this application. The aforementioned storage medium may include various media capable of storing program code, such as a USB flash drive, portable hard drive, magnetic disk, optical disk, read-only memory (ROM), or random access memory (RAM).

[0164] The above-described embodiments are only used to illustrate the technical solutions of this application, and are not intended to limit them. Although this application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of this application.

Claims

1. A graphics processing device, characterized in that, The device includes; The processor is used to generate the preset first graphics task command; A perception processing unit is configured to: acquire sensor data, process the sensor data to obtain user behavior events, and modify the first image task command according to the user behavior events to obtain a second graphics task command. A graphics processor for performing graphics rendering based on the second graphics task command to obtain a first image.

2. The apparatus according to claim 1, characterized in that, The device also includes a storage unit; The processor is further configured to store the first graphics task command to the storage unit; The perception processing unit is specifically used to modify the first image task instruction in the storage unit according to the user behavior event to obtain a second image task command.

3. The apparatus according to claim 1 or 2, characterized in that, The preset first graphic task command includes a preset image task command template, the user behavior event is used to trigger the target animation effect, and the second graphic task command is an image task command that can achieve the target animation effect.

4. The apparatus according to any one of claims 1-3, characterized in that, The processor is also configured to enter a sleep state after generating the first graphics task command.

5. The apparatus according to any one of claims 1-4, characterized in that, The sensing and processing unit is always online.

6. The apparatus according to any one of claims 1-5, characterized in that, The processor, the graphics processor, and the perception processing unit are integrated on the same chip.

7. The apparatus according to any one of claims 1-6, characterized in that, The device further includes a display subsystem for processing the first image to obtain a second image, the second image being displayed on a display screen.

8. The apparatus according to any one of claims 1-7, characterized in that, The first graphics task command includes multiple first graphics task commands, and the second graphics task command includes multiple second graphics task commands; The perception processing unit is specifically used to schedule the current image task command among the plurality of second graphics task commands to the graphics processor; The graphics processor is specifically used to perform graphics rendering based on the current image task command to obtain the current image.

9. The apparatus according to any one of claims 1-7, characterized in that, The first graphics task command includes multiple first graphics task commands, and the second graphics task command includes multiple second graphics task commands; The device further includes: a scheduler, configured to schedule the current image task command among the plurality of second graphics task commands to the graphics processor; The graphics processor is specifically used to perform graphics rendering based on the current image task command to obtain the current image.

10. A graphics processing method, characterized in that, Applied to a graphics processing device, the device including a processor, a graphics processor, and a perception processing unit; the method includes: The processor generates a preset first graphics task command; The sensor processing unit acquires sensor data, processes the sensor data to obtain user behavior events, and modifies the first graphics task command based on the user behavior events to obtain a second graphics task command. The graphics processor performs graphics rendering based on the second graphics task command to obtain a first image.

11. The method according to claim 10, characterized in that, The device further includes a storage unit; the method further includes: The processor stores the first graphics task command into the storage unit. The step of modifying the first graphics task command based on the user behavior event through the perception processing unit to obtain the second graphics task command includes: The perception processing unit modifies the first image task instruction in the storage unit according to the user behavior event to obtain a second image task command.

12. The method according to claim 10 or 11, characterized in that, The preset first graphic task command includes a preset image task command template, the user behavior event is used to trigger the target animation effect, and the second graphic task command is an image task command that can achieve the target animation effect.

13. The method according to any one of claims 10-12, characterized in that, The method further includes: The processor enters a sleep state after generating the first graphics task command.

14. The method according to any one of claims 10-13, characterized in that, The device further includes a display subsystem; the method further includes: The first image is processed by the display subsystem to obtain a second image, which is then displayed on the display screen.

15. The method according to any one of claims 10-14, characterized in that, The first graphics task command includes a plurality of first graphics task commands, and the second graphics task command includes a plurality of second graphics task commands; the method further includes: The perception processing unit schedules the current image task command among the plurality of second graphics task commands to the graphics processor. The step of executing graphics rendering based on the second graphics task command through the graphics processor to obtain the first image includes: The graphics processor performs graphics rendering based on the current image task command to obtain the current image.

16. The method according to any one of claims 10-14, characterized in that, The device further includes a scheduler; the first graphics task command includes a plurality of first graphics task commands, and the second graphics task command includes a plurality of second graphics task commands; The method further includes: The scheduler schedules the current image task command from the plurality of second graphics task commands to the graphics processor. The step of executing graphics rendering based on the second graphics task command through the graphics processor to obtain the first image includes: The graphics processor is specifically used to perform graphics rendering based on the current image task command to obtain the current image.

17. A terminal device, characterized in that, Includes the graphics processing device and display screen as described in any one of claims 1-9.

18. A computer-readable storage medium, characterized in that, The computer-readable medium is used to store program code that, when executed by a graphics processing device, implements the method described in any one of claims 10-16.

19. A computer program product, characterized in that, The computer program product includes instructions that, when executed by a graphics processing device, cause the graphics processing device to perform the method as described in any one of claims 10-16.

Images