Video processing method and apparatus, computer device, storage medium and computer program product

Abstract

A video processing method, which is applied to a client. A main thread and a plurality of background threads in communication connection with the main thread are deployed in the client. The method comprises: calling the main thread to acquire image frames from a server by way of a data stream, and successively obtaining a plurality of image frame sequences (S302); for a sub-video generation task of each image frame sequence, calling the main thread to determine, from the background threads and on the basis of respective working states of the background threads, a selected background thread for responding to the sub-video generation task (S304); calling the selected background thread to encode and encapsulate the image frame sequence corresponding to the sub-video generation task to generate a sub-video corresponding to the image frame sequence (S306); and calling the main thread to successively display the sub-videos corresponding to the image frame sequences according to an order of the image frame sequences in the data stream (S308).

Description

Video processing methods, apparatus, computer equipment, storage media and computer program products

[0001] Related applications

[0002] This application claims priority to Chinese patent application filed on November 18, 2024, application number 202411648286.4, entitled "Video Processing Method, Apparatus, Computer Equipment, Storage Medium and Computer Program Product", the entire contents of which are incorporated herein by reference. Technical Field

[0003] This application relates to the field of computer technology, and in particular to a video processing method, apparatus, computer equipment, computer-readable storage medium, and computer program product. Background Technology

[0004] With the rapid development of computer technology and the improvement of people's living standards, watching videos has become an important entertainment activity in people's daily lives.

[0005] In traditional technology, the server generates the video and distributes it to the client in response to a video request. However, this traditional approach can easily lead to high server load and performance degradation when dealing with a large number of videos to be generated or a large number of clients requesting videos simultaneously. Summary of the Invention

[0006] Therefore, it is necessary to provide a video processing method, apparatus, computer equipment, computer-readable storage medium, and computer program product that can improve service performance in response to the above-mentioned technical problems.

[0007] In a first aspect, this application provides a video processing method applied to a client, wherein the client has a main thread and multiple background threads communicatively connected to the main thread; the method includes:

[0008] The main thread is invoked to obtain image frames from the server in the form of a data stream, and multiple image frame sequences are obtained sequentially.

[0009] For each sub-video generation task of the image frame sequence, the main thread is invoked to determine the selected background thread for responding to the sub-video generation task based on the working status of each background thread.

[0010] The selected background thread is invoked to encode and encapsulate the image frame sequence corresponding to the sub-video generation task, thereby generating the sub-video corresponding to the image frame sequence.

[0011] The main thread is invoked to sequentially display the sub-videos corresponding to each of the image frame sequences in the data stream, according to the order of each image frame sequence.

[0012] Secondly, this application also provides a video processing apparatus for a client, wherein the client has a main thread and multiple background threads communicatively connected to the main thread; the apparatus includes:

[0013] The image frame acquisition module is used to call the main thread to acquire image frames from the server in the form of a data stream, and obtain multiple image frame sequences in sequence.

[0014] The background thread selection module is used to, for each of the image frame sequences, call the main thread to determine the selected background thread for responding to the sub-video generation task based on the working status of each of the background threads.

[0015] The sub-video generation module is used to call the selected background thread to encode and encapsulate the image frame sequence corresponding to the sub-video generation task, and generate the sub-video corresponding to the image frame sequence.

[0016] The video display module is used to call the main thread to sequentially display the sub-videos corresponding to each of the image frame sequences in the data stream according to the order of each image frame sequence.

[0017] Thirdly, this application also provides a computer device. The computer device includes a memory and a processor, the memory storing a computer program, and the processor executing the computer program to implement the steps of the above-described method.

[0018] Fourthly, this application also provides a computer-readable storage medium. The computer-readable storage medium stores a computer program thereon, which, when executed by a processor, implements the steps of the above-described method.

[0019] Fifthly, this application also provides a computer program product. The computer program product includes a computer program that, when executed by a processor, implements the steps of the above-described method.

[0020] Details of one or more embodiments of this application are set forth in the following drawings and description. Other features, objects, and advantages of this application will become apparent from the specification, drawings, and claims. Attached Figure Description

[0021] To more clearly illustrate the technical solutions in the embodiments of this application, the drawings used in the embodiments will be briefly introduced below. Obviously, the drawings described below are only embodiments of this application. For those skilled in the art, other drawings can be obtained based on the published drawings without creative effort.

[0022] Figure 1 is an application environment diagram of a video processing method in one embodiment;

[0023] Figure 2 is a schematic diagram of the principle of a video processing method in one embodiment;

[0024] Figure 3 is a flowchart illustrating a video processing method in one embodiment;

[0025] Figure 4 is a schematic diagram of the allocation principle of sub-video generation tasks in one embodiment;

[0026] Figure 5 is a schematic diagram of the video processing process in one embodiment;

[0027] Figure 6 is a schematic diagram of the video processing process in another embodiment;

[0028] Figure 7 is a flowchart illustrating the video processing method in another embodiment;

[0029] Figure 8 is a schematic diagram of the process of generating a complete video in one embodiment;

[0030] Figure 9 is a structural block diagram of a video processing device in one embodiment;

[0031] Figure 10 is an internal structure diagram of a computer device in one embodiment. Detailed Implementation

[0032] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those of ordinary skill in the art without creative effort are within the scope of protection of this application.

[0033] The video processing method provided in this application embodiment can be applied to the application environment shown in Figure 1. The client 101 communicates with the server 102 via a network. This communication network can be a wired network or a wireless network. Therefore, the client 101 and the server 102 can be directly or indirectly connected via wired or wireless communication. For example, the client 101 can be indirectly connected to the server 102 via a wireless access point, or the client 101 can be directly connected to the server 102 via the Internet; this application does not impose any limitations on this.

[0034] The client 101 can run on a terminal. This terminal can be, but is not limited to, various desktop computers, laptops, smartphones, tablets, IoT devices, and portable wearable devices. IoT devices can include smart speakers, smart TVs, smart air conditioners, and smart in-vehicle systems. Portable wearable devices can include smartwatches, smart bracelets, and head-mounted devices. This embodiment can be applied to video display scenarios, where the video can be, for example, a short video, news video, or trending video. The terminal has a client that hosts the video display page. This client can be software (e.g., instant messaging clients, shopping clients, live streaming clients), or it can be a webpage, a mini-program, etc.

[0035] Furthermore, server 102 can be a standalone physical server, a server cluster or distributed system composed of multiple physical servers, or a cloud server providing basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communication, middleware services, domain name services, security services, CDN (Content Delivery Network), and big data and artificial intelligence platforms. The data storage system can store the data that server 102 needs to process. The data storage system can be set up independently, integrated into server 102, or placed in the cloud or on other servers.

[0036] As shown in Figure 1, client 101 can deploy a main thread 1011 and multiple background threads 1012 that communicate with the main thread 1011. During video processing, client 101 can call the main thread 1011 to obtain image frames from server 102 in the form of a data stream, thereby obtaining multiple image frame sequences. Client 101 can also, for each image frame sequence's respective sub-video generation task, call the main thread 1011 to determine, based on the working status of each background thread 1012, a selected background thread to respond to the sub-video generation task, and call the selected background thread to encode and encapsulate the image frame sequence corresponding to the sub-video generation task, generating the sub-video corresponding to the image frame sequence. Finally, client 101 can call the main thread to sequentially display the sub-videos corresponding to each image frame sequence according to the order of each image frame sequence in the data stream.

[0037] Taking the case where every 100 image frames constitute an image frame sequence as an example. For instance, as shown in Figure 2, the client can call the main thread to obtain image frames 201 from the server via a data stream. During the acquisition of each image frame 201, the main thread sequentially obtains multiple image frame sequences, following the pattern of every 100 image frames forming an image frame sequence. For instance, as shown in Figure 2, the first 100 image frames constitute image frame sequence A. For the sub-video generation task corresponding to image frame sequence A, the client calls the main thread, based on the working status of each background thread, to select background thread 1 as the selected background thread to respond to the sub-video generation task, and calls this selected background thread to encode and encapsulate image frame sequence A to generate the sub-video a corresponding to the image frame sequence. Meanwhile, the main thread continues to receive image frames as a data stream, quickly obtaining another image frame sequence B containing 100 images. Similarly, for the sub-video generation task corresponding to image frame sequence B, the client calls the main thread, based on the working status of each background thread, to select background thread 2 as the chosen background thread to respond to the sub-video generation task, and calls this selected background thread to encode and encapsulate image frame sequence B to generate the corresponding sub-video b. This process continues, with the main thread continuously receiving image frames as a data stream and allocating tasks, and each background thread responsible for generating the sub-videos corresponding to the image frame sequences. As shown in Figure 2, each generated sub-video is sent back to the main thread by its corresponding background thread, allowing the client to call the main thread to sequentially display the sub-videos corresponding to each image frame sequence according to their order in the data stream. This process is equivalent to encapsulating image frames into videos and displaying them on the client side using a distributed processing method, effectively reducing the server load and improving service performance.

[0038] In one embodiment, as shown in FIG3, a video processing method is provided. Taking the application of this method to client 101 in FIG1 as an example, the method includes the following steps:

[0039] Step S302: Call the main thread to obtain image frames from the server in the form of a data stream, and obtain multiple image frame sequences in sequence.

[0040] The client application consists of a main thread and multiple background threads that communicate with and connect to the main thread. The main thread is the core of the client program and is typically created and run immediately by the operating system when the client starts. In multithreaded tasks, the main thread is usually the first thread created and has the highest priority; once it terminates, the entire client program also terminates. Optionally, the main thread is primarily responsible for handling interactions between the client and the external environment, including processing user input, drawing and updating the user interface, and exchanging information with the server during interactions between the client and the server.

[0041] To avoid blocking the main thread, the client can also deploy multiple background threads. A background thread is a thread that provides services in the background while the program is running. Unlike foreground threads, background threads do not prevent thread termination. When all foreground threads terminate, the program will automatically end, regardless of whether any background threads are still running. Background threads are typically used to handle tasks that do not require direct user interaction, such as data loading, data processing, and logging. Optionally, the main thread can be responsible for creating and managing background threads and distributing tasks to them. Furthermore, the main thread and background threads can communicate with each other through mechanisms such as handlers, signals and slots, or event-driven mechanisms.

[0042] In an optional embodiment, the background thread may specifically be a Web Worker thread. Web Workers are a technology introduced in HTML5 designed to address the performance issues caused by the single-threaded model of JavaScript. They allow the creation of independent background threads within the browser to perform time-consuming computational tasks, thereby avoiding blocking the main thread and improving webpage responsiveness and user experience.

[0043] In an optional embodiment, the background thread may specifically be a Service Worker thread. Service Worker is an extension based on Web Worker. Compared to Web Worker, Service Worker is more independent, can continue running after the client's page is closed, and can be shared across pages.

[0044] In an optional embodiment, the background thread may specifically be a Shared Worker thread. Shared Worker is a variant of Web Worker that allows the same worker instance to be shared across multiple windows or tabs. This approach reduces resource consumption and is suitable for scenarios requiring data sharing across multiple pages.

[0045] Specifically, the client can invoke the main thread to retrieve image frames from the server in a data stream manner, obtaining multiple image frame sequences sequentially. Each image frame sequence can include multiple image frames, and the number of image frames contained in different image frame sequences can be the same or different. Furthermore, the image frame sequences can be non-overlapping sequences; for example, frames 1 to 100 constitute the first image frame sequence, and frames 101 to 200 constitute the second image frame sequence. Alternatively, the image frame sequences can overlap; for example, frames 1 to 100 constitute the first image frame sequence, and frames 91 to 190 constitute the second image frame sequence. It can be understood that when there is no overlap between image frame sequences, a corresponding sequence identifier needs to be added to each image frame sequence to represent its order in the data stream. When there is overlap between image frame sequences, the order of the image frame sequences in the data stream can be determined by performing similarity matching on the overlapping image frames at the beginning or end of the sequence. For example, if the overlapping image frames at the end of image frame sequence A match the overlapping image frames at the beginning of image frame sequence B, it means that the order of these two image frame sequences in the data stream is image frame sequence A first, and image frame sequence B second.

[0046] Streaming image frames received from the server eliminates the need to wait for all frames to be received before processing begins, significantly improving video processing efficiency. For example, if a video consists of 1000 frames, traditional processing methods might require waiting for all 1000 frames to be received before processing can begin. This approach has drawbacks: poor network conditions can lead to long waiting times, and network interruptions during reception may necessitate restarting the process, resulting in very low efficiency. Streaming avoids these problems; only the first batch (e.g., 100 frames) is received, and processing can begin immediately. This method allows processing to occur simultaneously with frame reception, greatly improving system response speed and efficiency. Furthermore, it effectively reduces memory usage, as only the frames currently being processed need to be stored, rather than storing all frames at once.

[0047] Streaming processing is a data processing mode. In this application, it refers to a mode where, when a client receives image frames from a server as a data stream, it does not need to wait for all image frames to be received before processing begins. As soon as a certain number of image frames are received to form an image frame sequence, the sequence is processed immediately. This processing mode can improve the efficiency of video processing, reduce memory usage, and avoid long waiting times and re-reception issues caused by network problems.

[0048] Step S304: For each sub-video generation task of each image frame sequence, the main thread is called to determine the selected background thread to respond to the sub-video generation task based on the working status of each background thread.

[0049] In this context, the image frame sequence is part of the complete data stream. Therefore, the video generation task corresponding to the image frame sequence is called a sub-video generation task, and the video generated based on the image frame sequence is called a sub-video. It can be understood that a sub-video is a part of the complete video corresponding to the complete data stream. For each image frame sequence obtained, the client can generate a sub-video generation task for that sequence. The generator of this sub-video generation task can be the main thread or the client's operating system; this is not limited here. After obtaining the sub-video generation task, the main thread allocates tasks for it.

[0050] The working states of a background thread can include busy, idle, and error states. A busy state means the background thread is currently executing a task and cannot immediately respond to other requests or tasks; an idle state means the background thread currently has no tasks to execute and is waiting for new tasks to arrive; an error state means that an error occurred during the execution of a task, causing the thread to terminate or crash.

[0051] Specifically, for each image frame sequence, the client can sequentially generate sub-video generation tasks corresponding to that image frame sequence. For each sub-video generation task of each image frame sequence, the client can call the main thread to obtain the working status of each background thread, and based on the working status of each background thread, determine the selected background thread to respond to the sub-video generation task.

[0052] In an optional embodiment, the main thread and background threads can communicate via a communication mechanism. Thus, the main thread can communicate with each background thread individually, obtain communication messages from each background thread, and determine the working status of each background thread based on these messages. For example, the main thread can monitor message and error events of the background threads. If the main thread designates a particular background thread as the selected background thread for responding to a sub-video generation task, that background thread is in a busy state. After completing the task, the background thread can send a task completion message to the main thread via `postMessage`. Upon receiving the task completion message, the main thread can determine that the background thread has completed its task and is currently in an idle state. Similarly, if an error occurs during the execution of a task, it can send an error message to the main thread. Upon receiving the error message, the main thread can determine that an error has occurred in the background thread and it is currently in an error state.

[0053] In an optional embodiment, the main thread can determine the working status of the background thread through an improved heartbeat mechanism. Specifically, the background thread can periodically send heartbeat packets to the main thread, and the frequency of sending heartbeat packets during busy states can be lower than the frequency during idle states, in order to reduce the resource consumption of the background thread during heartbeat packet sending and ensure the task processing efficiency of the background thread. Thus, the main thread can determine whether the background thread is in an idle state based on the frequency of the received heartbeat packets. If no heartbeat packet is received from the background thread within a set time period, the main thread can determine that the background thread is in an error state.

[0054] Furthermore, the specific method for determining the selected background thread to respond to the sub-video generation task is not unique, based on the working status of each background thread. Optionally, the client can randomly select any background thread in an idle state to be used for responding to the sub-video generation task. Optionally, for each idle thread, the client can determine its historical efficiency in processing historical sub-video generation tasks and select the idle thread with the highest historical efficiency as the selected background thread to respond to the sub-video generation task. Optionally, if no idle thread exists, the client can determine the expected remaining processing time for each background thread for its respective task and select the background thread with the shortest expected remaining processing time as the selected background thread to respond to the sub-video generation task.

[0055] Step S306: Call the selected background thread to encode and encapsulate the image frame sequence corresponding to the sub-video generation task, and generate the sub-video corresponding to the image frame sequence.

[0056] As mentioned earlier, the main thread is responsible for retrieving image frames from the server and assigning sub-video generation tasks. Background threads respond to these tasks, completing the process from image frames to sub-videos. Specifically, the client can assign sub-video generation tasks to selected background threads via the main thread, enabling calls to those specific threads. During sub-video generation, the selected background thread compresses the image sequence using encoding techniques and packages the compressed data into a playable video file using a specific encapsulation format, thus achieving the conversion from static images to dynamic video.

[0057] In an optional embodiment, the main thread can send each image frame contained in the image frame sequence to a selected background thread, so that the selected background thread can respond to the sub-video task corresponding to the image frame sequence, encode and encapsulate each received image frame, and generate a sub-video.

[0058] In an optional embodiment, the main thread can send the storage address of each image frame contained in the image frame sequence to a selected background thread, so that the selected background thread can respond to the sub-video task corresponding to the image frame sequence, access each image frame contained in the image frame sequence through each storage address, encode and encapsulate each image frame, and generate a sub-video.

[0059] The specific process of selecting a background thread to generate the sub-video corresponding to the image frame sequence is not unique.

[0060] In an optional embodiment, a selected background thread can use video editing tools or libraries to encode the image sequence within the image frame sequence to reduce the space required for storage and transmission. For example, an I-frame is a keyframe containing complete image information; a P-frame is a prediction frame generated based on the previous frame; and a B-frame is a bidirectional prediction frame generated by combining information from the preceding and following frames. After encoding, the selected background thread then encapsulates the encoded video data into a standard format file for easy parsing and playback by the player. This standard format can be, for example, MP4 (MPEG-4 Part 14) or MKV (Matroska Video, Matroska Multimedia Container) format, etc., and is not limited thereto.

[0061] In an optional embodiment, a selected background thread can invoke WebAssembly to perform image transcoding, encoding, and encapsulation processes, generating a sub-video corresponding to the image frame sequence. WebAssembly is a new code format that can run with near-native performance in modern web browsers. In this process, the selected background thread decodes the image frames in the image frame sequence, performs color space conversion, and finally encodes and encapsulates them to obtain the sub-video corresponding to the image frame sequence. This process involves a significant amount of computation, but due to the efficiency of WebAssembly, it can be completed in a very short time.

[0062] It's important to note that since image frames are acquired via a data stream, the client can generate a corresponding sub-video generation task for each image frame sequence it receives, and then invoke the main thread to allocate tasks, selecting a background thread from among the background threads to respond to that sub-video task. During this process, communication between the main thread and the background threads is asynchronous. This means that after the main thread completes task allocation, it doesn't need to wait for a response from the background thread but can continue executing other tasks. These other tasks could be allocating tasks for other sub-video generation tasks, or rendering and displaying already generated sub-videos, etc. Using asynchronous communication avoids blocking the main thread, improving system response speed and efficiency. Furthermore, since there are multiple background threads, each can respond to different sub-video generation tasks, thus achieving parallel processing of multiple sub-video generation tasks.

[0063] In one embodiment, the video processing method further includes: determining an encoding and encapsulation algorithm that matches the client. In this embodiment, step S306 includes: invoking a selected background thread to encapsulate the image frame sequence corresponding to the sub-video generation task according to the encoding and encapsulation algorithm, thereby generating a sub-video corresponding to the image frame sequence.

[0064] As mentioned above, this application transfers the video generation process from the server to the client. Since different clients may have different hardware configurations and user needs, during the generation of sub-videos, an encoding and encapsulation algorithm that matches the client can be determined, and a selected background thread can be called to encapsulate the image frame sequence corresponding to the sub-video generation task according to the encoding and encapsulation algorithm to generate the sub-video corresponding to the image frame sequence.

[0065] In an optional embodiment, the encoding and encapsulation algorithm matching the client can be determined based on the hardware configuration of the display device corresponding to the client. For example, the display resolution of the display device corresponding to the client can be determined as the resolution configured during the encoding of the image frame; the color space supported by the display device can be determined as the target color space for color space conversion during the image encoding and encapsulation process; and the frame rate configured during the encoding of the image frame can be determined based on the refresh rate of the display device.

[0066] For example, the encoding and encapsulation algorithm matching the client can be determined based on the hardware configuration of the client's display device. Specifically, for display resolution, if the display device has H horizontal pixels and V vertical pixels, when using the H.264 encoding standard, if H×V is greater than 1920×1080 (high-definition resolution), a larger GOP (Group of Pictures) size, such as 60 frames, should be used during encoding. This is because high-definition video has rich details, and a larger GOP can better utilize inter-frame correlation, improving encoding efficiency and video quality. If H×V is less than this value, the GOP size can be set to 30 frames to balance encoding efficiency and processing resources. For color space, if the display device supports the RGB color space, a color space conversion algorithm is used to convert the image frames from the common YUV color space to the RGB color space before encoding and encapsulation. The conversion process can be implemented through matrix operations, first linearly combining the YUV color components according to a specific conversion matrix to obtain the corresponding RGB color components. For refresh rate, if the display device's refresh rate is F Hz, the video frame rate is set to F during encoding. This ensures that the smoothness of the video playback on the display device matches the refresh rate of the display device, avoiding problems such as stuttering or flickering.

[0067] In an optional embodiment, a stylistic feature matching the user type logged in by the client can be determined, and a corresponding encoding and encapsulation algorithm can be further determined. This allows a selected background thread to incorporate the stylistic feature during the encoding and encapsulation process, resulting in a sub-video containing the stylistic feature. For example, a user watermark or icon can be added during the encoding and encapsulation process, or the image information in the image frame can be stylized to obtain a stylized image frame, which can then be further encapsulated to obtain the sub-video. This stylized image frame can be, for example, in a comic book style, an oil painting style, or a black and white style, etc., and is not limited thereto.

[0068] Specifically, based on the user type logged into the client, the stylistic features and corresponding encoding / encapsulation algorithms matching that user type can be determined as follows: Establish a mapping relationship between user type and stylistic features. If the user type is a regular user, the stylistic feature is to add a simple watermark. When adding a watermark, first determine the watermark position, which can be the lower right corner of the image frame as a fixed position. Then, overlay the watermark image with the image frame using the overlay formula I. new =α×I watermark +(1-α)×I frame , where I new For the superimposed image frames, I watermark For watermarked images, I frameThe image frame is the original image frame, and α is the transparency of the watermark image. If the user type is advanced user, the stylization feature is comic style conversion. Comic style conversion can be achieved through the following steps: First, the image frame is converted to grayscale, using the formula I... gray = 0.299×R + 0.587×G + 0.114×B, where I gray The image is a grayscale image, with R, G, and B being the three channels of a color image. Edge detection is then performed using the Canny edge detection algorithm to detect edges and obtain the image's contour information. Next, color quantization is applied to the image, reducing the number of colors to a limited set, giving the image a comic book style. Finally, the edge information and the color-quantized image are merged to obtain a comic book-style image frame. Color quantization refers to the operation of reducing the number of colors in an image to a limited set. In the comic book style conversion process, color quantization is used to give the image a comic book style color effect, making the image colors simpler and more vivid.

[0069] In the above embodiments, determining an encoding and encapsulation algorithm that matches the client, and then calling a selected background thread to encapsulate the image frame sequence corresponding to the sub-video generation task according to this algorithm, generates the sub-video corresponding to the image frame sequence. This ensures that the generated sub-video matches the client's needs, improving the compatibility between the sub-video and the client. Furthermore, since the client only needs to generate the video to be displayed on its own device, and the sub-video is generated through parallel processing by multiple background threads, the generation of personalized videos does not cause the main front-end thread to be blocked. Therefore, the above solution can generate sub-videos with high client compatibility without affecting processing efficiency, thus improving video processing results.

[0070] Step S308: Call the main thread to display the sub-videos corresponding to each image frame sequence in the order of their positions in the data stream.

[0071] The order of the image frame sequence in the data stream can be determined based on the sequence identifier carried in the image frame sequence, or based on the matching between the head or tail image frames of each image frame sequence. In some optional embodiments, since the sub-video generation tasks are allocated using streaming processing, the order of the image frame sequence in the data stream can also be determined based on the task allocation time of the sub-video generation task corresponding to the image frame sequence.

[0072] In some optional embodiments, if the image frame sequence does not carry a sequence identifier or the sequence identifier is unclear, since the sub-video generation task is allocated using streaming processing, the order of the image frame sequence in the data stream can be determined with the help of the task allocation time of the sub-video generation task corresponding to the image frame sequence. However, the sequence identifier carried by the image frame sequence shall prevail in the end.

[0073] Specifically, the client can invoke the main thread to display the sub-videos corresponding to each image frame sequence sequentially according to their order in the data stream. Optionally, the client can invoke the main thread to combine the sub-videos into a complete video according to their order in the data stream, and then display the complete video to achieve sequential display of the sub-videos. Alternatively, the client can choose not to combine the sub-videos, but instead use distributed processing, rendering the next sub-video while displaying the previous one according to their order in the data stream, to reduce display latency caused by video processing and ensure timely video display.

[0074] The aforementioned video processing method involves the client calling the main thread to acquire image frames from the server via a data stream, sequentially obtaining multiple image frame sequences. For each image frame sequence's sub-video generation task, the main thread, based on the working status of each background thread, determines a selected background thread to respond to the sub-video generation task. The selected background thread encodes and encapsulates the image frame sequence corresponding to the sub-video generation task, generating the corresponding sub-video. The main thread then displays the sub-videos corresponding to each image frame sequence in the order they appear in the data stream. In this process, on the one hand, the server only needs to send image frames to the client instead of the complete video, effectively reducing server bandwidth pressure and improving service performance. On the other hand, the client, through the cooperation of the main thread and multiple background threads, completes the generation and display of multiple sub-videos, shifting the video generation time to after the data stream is distributed and transferring the video generation from the server to the client, achieving distributed video generation on the client side. This fundamentally solves the problem of high server load while ensuring video processing efficiency, thus improving service performance.

[0075] In one embodiment, based on the working state of each background thread, the selected background thread for responding to the sub-video generation task is determined from the background threads, including: obtaining the working state of each background thread; if there is an idle thread in an idle state among the background threads, the idle thread is determined as the selected background thread for responding to the sub-video generation task; if all background threads are in a busy state, the sub-video generation task is placed into the task queue as an unprocessed task.

[0076] The working status includes a busy state and an idle state. A task queue is a mechanism for managing and executing tasks, widely used in scenarios such as concurrent computing, asynchronous programming, task scheduling, and task management. Unprocessed tasks refer to sub-video generation tasks built by the client that, due to the lack of idle threads to respond to these tasks, are placed in the task queue awaiting further processing.

[0077] Specifically, the client's main thread can obtain the working status of each background thread. If an idle thread exists among the background threads, it is selected as the designated background thread to respond to the sub-video generation task, without needing to put the sub-video generation task into a task queue. This ensures timely response to the sub-video generation task by omitting enqueue and dequeue operations. If all background threads are busy, the sub-video generation task is added to the task queue as an unprocessed task. The main thread can then retrieve unprocessed tasks from this queue and assign them to the corresponding background thread for processing.

[0078] In an optional embodiment, the processing priority of unprocessed tasks in the task queue is higher than that of sub-video generation tasks not placed in the task queue. That is, as shown in Figure 4, for the latest obtained current image frame sequence X, after constructing the current sub-video generation task Rx corresponding to the current image frame sequence X, the client needs to first determine whether the task queue is empty. If the task queue is empty, it means that the sub-video generation tasks corresponding to each historical image frame sequence before the current image frame sequence have been assigned. At this time, the client can directly assign tasks based on the working status of each background thread. If there are unprocessed tasks in the task queue 401 (such as R1 in Figure 4), it means that there are unassigned sub-video generation tasks corresponding to historical image frame sequences before the current image frame sequence. To ensure that the image frame sequences with higher sorting can be processed first, the client can put the current sub-video generation task Rx as an unprocessed task into the task queue. Subsequently, if there are idle threads, the unprocessed tasks can be taken from the task queue for assignment and processing.

[0079] In the above embodiments, when there is an idle thread in the background threads, the idle thread is selected as the background thread to respond to the sub-video generation task. By omitting the enqueue and dequeue operations, it can be ensured that the sub-video generation task can be responded to in a timely manner. When all background threads are busy, the sub-video generation task is placed in the task queue as an unprocessed task, which can facilitate subsequent task scheduling and allocation.

[0080] In one specific embodiment, the number of idle threads is multiple. In this embodiment, the video processing method further includes: for each idle thread, determining the historical efficiency of the idle thread in responding to historical tasks; and based on the historical efficiency of each idle thread, determining a selected background thread from the idle threads to respond to the sub-video generation task.

[0081] Specifically, for each background thread, its historical efficiency in responding to historical tasks is determined. This historical task could be, for example, a sub-video generation task or other tasks. Optionally, for the same task type, the client can record the start and end times of each background thread when executing tasks belonging to that task type, obtaining the execution duration of each background thread for that task type, and determining the historical efficiency inversely correlated with that execution duration. For example, the client can record the start time t of each background thread when executing tasks belonging to that task type. start and task end time t end The execution time Δt = t for each background thread corresponding to this task type is obtained. end -t start Let the historical efficiency be E, then it can be calculated using the formula... (where k is a positive constant that can be adjusted according to the actual situation) to determine the historical efficiency that is inversely related to the execution time.

[0082] Optionally, the client can determine the historical efficiency of each background thread by statistically analyzing its throughput when executing historical tasks. Throughput refers to the number of tasks completed per unit of time, and it is positively correlated with historical efficiency. For example, if the statistical period is T, and a background thread completes N tasks within that period, then the throughput of that background thread is R = N / T. The historical efficiency, denoted as E, can be determined using the formula E = m × R (where m is a positive constant that can be adjusted according to actual conditions).

[0083] Given the historical efficiency of each idle thread, the client can select the idle thread with the highest historical efficiency as the designated background thread for responding to the sub-video generation task; alternatively, the client can select an idle thread whose historical efficiency has reached a set level as the designated background thread. If there are multiple idle threads with the highest historical efficiency or that have reached a set level, the client can randomly select one of these idle threads as the designated background thread, or it can select the thread with the longest idle time as the designated background thread.

[0084] In the above embodiments, when there are multiple idle threads, the background thread is selected from among the idle threads based on their historical efficiency in responding to historical tasks. This allows for the priority calling of idle threads with high work efficiency, which is beneficial to further improve the work efficiency of the video processing process.

[0085] In one specific embodiment, the video processing method further includes: during the process of encoding and encapsulating the image frame sequence corresponding to the sub-video generation task by the selected background thread, if an error message is detected from the selected background thread, the sub-video generation task is placed in the task queue as an unfinished task; the selected background thread is terminated, and if there is an idle thread among the remaining background threads, the target task is retrieved from the task queue according to the task processing priority, and the idle thread is called to execute the target task.

[0086] An incomplete task refers to a sub-video generation task that was not completed due to an error during execution. This task will be placed in the task queue and wait to be re-executed later.

[0087] In practical applications, during the process of encoding and encapsulating the image frame sequence corresponding to the sub-video generation task using a selected background thread, exceptions may occur that lead to execution errors. Therefore, the client can monitor the running status of each selected background thread by calling the main thread and take appropriate action when error messages are detected.

[0088] Specifically, during the encoding and encapsulation of the image frame sequence corresponding to the sub-video generation task by a selected background thread, if an error message is detected from the selected background thread, the client can add the sub-video generation task as an incomplete task to the task queue and terminate the selected background thread that malfunctioned. Optionally, the main thread can obtain the error message from the selected background thread by monitoring the error events of each selected background thread. At this point, the task queue includes unprocessed tasks that have not yet started processing and incomplete tasks that have been partially processed. The main thread can continuously monitor the working status of each background thread, and if there is an idle thread among the remaining background threads, it can retrieve the target task from the task queue according to the task processing priority and call the idle thread to execute the target task.

[0089] In an optional embodiment, the task processing priority is positively correlated with the time when the task is placed in the task queue. That is, the main thread can retrieve the target task from the task queue and schedule and respond to it in a first-in-first-out order.

[0090] In an optional embodiment, the task processing priority is positively correlated with the order of the image frame sequence corresponding to the task in the data stream. That is, the main thread can reorder the tasks in the task queue according to the order of the image frame sequence corresponding to each task in the data stream to ensure that the tasks with higher order can be retrieved and processed first.

[0091] The target task is either an unprocessed or incomplete task. If multiple idle threads exist, the main thread can further refer to the filtering method described in the above embodiment to determine the thread to respond to the target task from among the idle threads.

[0092] In the above embodiments, by monitoring the feedback information of the selected background thread, the selected background thread that malfunctions can be terminated in a timely manner, and the task can be put back into the task queue to wait for the next execution. This can reduce the impact of a single thread error on the entire system and help improve the stability and efficiency of the client.

[0093] In a specific implementation, the video processing method further includes: for multiple unprocessed tasks in the task queue, determining the task processing priority of each unprocessed task according to the enqueue time of each unprocessed task; for multiple unfinished tasks in the task queue, determining the task processing priority of each unfinished task according to the first execution time of each unfinished task.

[0094] Among them, the task processing priority is positively correlated with the queuing time; the task processing priority is positively correlated with the first execution time; the task processing priority of any unfinished task is higher than that of any unprocessed task.

[0095] In practical applications, upon receiving a sequence of image frames, the client constructs a sub-video generation task for that sequence. If no idle thread responds to the task, this sub-video generation task is placed in the task queue as an unprocessed task. In other words, for an unprocessed task, its enqueue time typically represents its construction time, which is positively correlated with the reception time of the image frame sequence it corresponds to. Therefore, the enqueue time of a processing task can, to some extent, represent the order of the image frame sequence corresponding to that unprocessed task in the data stream. Based on this, for multiple unprocessed tasks in the task queue, task processing priorities are determined according to a positive correlation with enqueue time. Since earlier enqueue times result in higher priority, this ensures that earlier-ordered image frame sequences are processed first.

[0096] Similarly, unfinished tasks might be tasks that were partially processed after being enqueued, or sub-video tasks that were not enqueued due to idle threads but encountered errors during execution. In either case, the initial execution time of an unfinished task can, to some extent, characterize the order of the corresponding image frame sequence in the data stream. Based on this, for multiple unfinished tasks in the task queue, their respective processing priorities are determined according to their initial execution times. Since earlier initial execution times result in higher processing priority, this also ensures that earlier-ordered image frame sequences are processed first.

[0097] Furthermore, compared to unprocessed tasks, on the one hand, since unprocessed tasks have already been partially executed, the subsequent execution time is relatively short. If they are processed first, the time occupied by background threads can be reduced, allowing background threads to quickly complete one task and start processing the next, which is beneficial to improving the utilization rate of background threads. On the other hand, since unprocessed tasks have already completed one scheduling, it means that the image frame sequence corresponding to them is earlier in the data stream than unprocessed tasks. If they are processed first, it can be ensured that the image frame sequence that is earlier in the sequence can be processed first.

[0098] In the above embodiments, when the task queue includes unprocessed tasks and incomplete tasks, different task processing priority determination methods are adapted according to the different types of tasks. This can improve the utilization of background threads while ensuring that the image frame sequences with earlier sequences can be processed first, thereby reducing the delay in the front-end display process and improving the overall performance of the client in the video processing process.

[0099] In a specific implementation, calling an idle thread to execute the target task includes: if the target task is an incomplete task, obtaining the task progress recorded for the incomplete task; and according to the task progress, calling the idle thread to execute the incomplete part of the target task and generating the sub-video corresponding to the target task.

[0100] Specifically, during task execution, the client can record the task's progress. In practice, different tools and platforms offer various methods for recording task progress, allowing the client to choose the appropriate method based on actual needs. For example, the task's progress can be displayed in real-time using Session (stage) storage, enabling clients to query or retrieve it immediately. Alternatively, in a Redis message queue, the `set_task_progress()` function can be called to record the task's progress percentage and save the progress information to Redis for real-time querying or retrieval by the client.

[0101] When the target task is incomplete, the client can obtain the task progress record for the incomplete task and, based on the task progress, call an idle thread to execute the incomplete part of the target task, generating the sub-video corresponding to the target task. For example, if a background thread encounters an error while processing the 50th image frame in an image frame sequence, creating an incomplete task and adding it to the task queue, then after retrieving the incomplete task from the task queue, it is only necessary to call an idle thread to re-execute the part starting from the 50th frame, without needing to re-execute the previously successfully executed parts.

[0102] In the above embodiments, for incomplete tasks, by recording the task progress, only the incomplete part can be executed in the subsequent execution process, realizing incremental retry for incomplete tasks, which is beneficial to improving the system's work efficiency.

[0103] It should be noted that the background threads communicating with the main thread can be pre-deployed or deployed in real-time. In real-time deployment, after calling the main thread to acquire image frames as a data stream and obtaining the first image frame sequence, the client can generate a sub-video generation task corresponding to that image frame sequence and deploy background threads to respond to that sub-video generation task, and so on. Over time, if the number of deployed background threads reaches the client's capacity limit, or if there are idle threads that have completed tasks and are in an idle state, then no new background threads need to be deployed; instead, parallel processing can be performed based on the already deployed background threads. In pre-deployment, the client can obtain the expected data volume of the complete data stream from the server and deploy a corresponding number of background threads on the client based on this expected data volume. The number of pre-deployed background threads is positively correlated with the expected volume of the complete data stream.

[0104] It's understandable that the number of background threads isn't constant, whether deployed pre-emptively or in real-time. For example, when faced with complete data streams of varying volumes, the client can adaptively enable or disable some background threads based on those deployed in the previous data stream processing to match the processing needs of the current data stream. For the same data stream, if the parallel processing capabilities of the current background threads cannot meet the actual requirements, the client can also enable new background threads.

[0105] In one embodiment, the video processing method further includes: when the number of tasks in the task queue reaches a threshold, obtaining the client's performance parameters; determining a thread count threshold that matches the performance parameters; and if the number of background threads maintaining communication with the main thread is less than the thread count threshold, starting a new background thread.

[0106] The client's performance parameters can refer to the performance parameters of the terminal to which the client belongs, or the performance parameters corresponding to the resources allocated to the client by the terminal. These performance parameters may include, for example, the number of CPU (Central Processing Unit) cores, memory size, etc.

[0107] Specifically, when the number of tasks in the task queue reaches a threshold, it indicates that the current client's parallel processing capability is insufficient to meet application requirements. At this point, the client can obtain its performance parameters by calling the main thread and further determine the appropriate thread count threshold. Then, it compares the number of deployed background threads that maintain communication with the main thread with this thread count threshold. If the number of background threads maintaining communication with the main thread is less than the threshold, it means the client has the capability to start new threads and can create new background threads to respond to tasks queued in the task queue.

[0108] For example, suppose the number of CPU cores in the client is C, and the memory size is M. The appropriate thread count threshold to match the performance parameters can be determined using the formula threshold = a × C + b × M (where a and b are positive constants that can be determined based on actual testing and experience). If the number of background threads maintaining communication with the main thread is less than this threshold, then a new background thread is started.

[0109] In the above embodiments, when the number of tasks in the task queue reaches a threshold, a thread number threshold adapted to the client's performance parameters is determined. If the number of background threads maintaining communication with the main thread is less than the thread number threshold, a new background thread is started. This ensures that an appropriate number of background threads are used for video processing within the client's processing capacity, thereby improving video processing efficiency while avoiding waste of computing resources.

[0110] In one embodiment, the main thread is invoked to obtain image frames from the server in the form of a data stream, and multiple image frame sequences are obtained sequentially. This includes: invoking the main thread to obtain image frames from the server in the form of a data stream; when the number of obtained image frames reaches a set number, constructing an image frame sequence containing each image frame; and returning to the step of invoking the main thread to obtain image frames from the server in the form of a data stream, thereby obtaining multiple image frame sequences sequentially.

[0111] Each image frame sequence carries a corresponding sequence identifier. This sequence identifier indicates the order of the image frame sequence in the data stream, ensuring that the subsequent sub-videos are displayed in the correct order. This identifier can be a keyframe representing the order among the image frames in the sequence, or it can be additional identifier information, such as a timestamp, a combination of letters or numbers, etc.

[0112] Specifically, the client can invoke the main thread to sequentially retrieve multiple image frames from the server as a data stream. Once the number of retrieved image frames reaches a predetermined limit, a sequence of image frames is constructed. Simultaneously, the main thread continues to continuously retrieve image frames from the server until the predetermined limit is reached again, at which point the next image frame sequence is constructed. Furthermore, to ensure the correct sequential display of sub-videos, the client can mark each image frame sequence with a sequence identifier representing its order within the data stream. This sequence identifier can be a keyframe representing the order of the image frames within the sequence, such as an overlapping image frame between adjacent sequences. Alternatively, the sequence identifier can be additional information added to each image frame, such as a timestamp or a combination of letters and numbers.

[0113] In the above embodiments, the image frame sequence also carries a sequence identifier to characterize the position of the images contained in the image frame sequence in the data stream. Thus, it is not necessary to strictly follow the order of sub-video generation tasks to respond, and the sub-videos can be displayed in sequence, which is beneficial to improving the flexibility of the video processing process.

[0114] In an optional embodiment, the sub-videos corresponding to each image frame sequence are displayed sequentially according to the order of each image frame sequence in the data stream, including: determining the playback order of the sub-videos corresponding to each image frame sequence according to the order of each image frame sequence in the data stream; and displaying each sub-video sequentially according to the playback order.

[0115] Specifically, the playback order of each sub-video corresponding to each image frame sequence can be determined according to the order of each image frame sequence in the data stream, as follows: If the image frame sequence carries a timestamp as a sequence identifier, first collect the timestamps of all image frame sequences to obtain a timestamp set T = {t1, t2, ..., t...} n Then, these timestamps are sorted using a bubble sort algorithm, resulting in an ordered set T of timestamps. sorted ={t s1 ,t s2 ,…,t snBased on the sorted timestamp order, the playback order of the sub-videos corresponding to each image frame sequence is determined, that is, the sub-video corresponding to the image frame sequence with the earliest timestamp is played first, and so on.

[0116] Bubble sort is a simple sorting algorithm used in this application to sort the timestamps carried by a sequence of image frames to determine the playback order of the corresponding sub-videos. It repeatedly traverses the column of elements to be sorted, comparing two adjacent elements in turn, swapping them if they are in the wrong order, until no adjacent elements need to be swapped, meaning the column of elements has been sorted.

[0117] As shown in Figure 6, after generating sub-video 'a' corresponding to image frame sequence A, background thread 1 can send sub-video 'a' back to the main thread. On one hand, the main thread can continue to acquire image frames via data stream and distribute tasks so that background thread 2 can synchronously generate sub-video 'b' corresponding to image frame sequence B and send it back to the main thread. On the other hand, the main thread can render sub-video 'a' and send the rendered sub-video 'a' to the corresponding display device on the client for display. During the display of sub-video 'a', the main thread can render sub-video 'b' and send the rendered sub-video 'b' to the corresponding display device on the client for display.

[0118] It should be noted that the client can also first stitch multiple sub-videos together into a parent video, and then, following the synchronous processing method described above, render the next parent video while displaying the previous one, thereby further improving the background preparation time during the video display process and ensuring continuous video display.

[0119] In the video display process described above, the user viewing the display device sees a series of sub-videos, providing the same viewing experience as directly displaying the complete video. Furthermore, since each sub-video can be processed in parallel during the video display process, the delay between image frame acquisition and the final video display can be reduced. Therefore, the above video display processing method can improve work efficiency without affecting the display effect.

[0120] In an optional embodiment, the sub-videos corresponding to each image frame sequence are displayed sequentially according to the order of each image frame sequence in the data stream, including: determining the playback order of the sub-videos corresponding to each image frame sequence according to the order of each image frame sequence in the data stream; and splicing the sub-videos sequentially according to the playback order to obtain and display a complete video containing each sub-video.

[0121] As mentioned earlier, after the background thread generates the sub-videos corresponding to the image frame sequence, it sends the generated sub-videos back to the main thread. For example, in Figure 6, background thread 1 sends sub-video 'a' back to the main thread, and background thread 2 sends sub-video 'b' back to the main thread. Thus, the main thread can determine the playback order of the sub-videos corresponding to each image frame sequence based on their order in the data stream. Then, it concatenates the sub-videos in the playback order to obtain the complete video corresponding to the complete data stream. This complete video, for example, could be the video FMV containing sub-videos 'a' and 'b' in Figure 6. Finally, the client can call the main thread to render this complete video and display it.

[0122] In the above embodiments, the sub-videos are first spliced ​​together in the order of playback to obtain a complete video containing all the sub-videos, and then the complete video is displayed. This ensures that the data stream can be displayed completely, which is beneficial to improving the display effect.

[0123] In one embodiment, the video processing method further includes: storing the obtained image frames in a storage space; and deleting each image frame contained in the image frame sequence from the storage space when a selected background thread sends the sub-video corresponding to the image frame sequence back to the main thread.

[0124] The storage space can be the client's application cache space or the device memory space of the client's terminal device; there is no limitation here. Specifically, when the client calls the main thread to obtain image frames from the server in the form of a data stream, the obtained image frames can be stored in the storage space for subsequent processing. When a background thread is selected to send the sub-video corresponding to the image frame sequence back to the main thread, the client can delete each image frame contained in that image frame sequence from the storage space to release storage resources.

[0125] Optionally, for each sub-video generation task of each image frame sequence, after determining the corresponding selected background thread, the main thread can send the storage location information of the image frame sequence in the storage space to the selected background thread. This allows the selected background thread to access the image frame sequence through the storage location information during its invocation, encode and encapsulate the image frame sequence, and generate the corresponding sub-video. If the selected background thread returns the sub-video corresponding to the image frame sequence to the main thread, the client can delete the storage location information of the image frame sequence along with each image frame contained in the image frame sequence from the storage space.

[0126] In the above embodiments, when a background thread sends the sub-video corresponding to the image frame sequence back to the main thread, deleting each image frame contained in the image frame sequence from the storage space is equivalent to not needing to store all image frames at the same time, which can reduce the storage pressure of the video processing process and help to further improve service performance.

[0127] In one embodiment, a video processing method is provided, which can be applied to a client that has a main thread and multiple background threads communicating with the main thread. As shown in Figure 7, in this embodiment, the video processing method includes the following steps:

[0128] Step S701: Call the main thread to obtain image frames from the server in the form of a data stream. If the number of obtained image frames reaches a set number, construct an image frame sequence containing each image frame.

[0129] Each image frame sequence carries a sequence identifier; this sequence identifier is used to represent the order of the image frame sequence in the data stream.

[0130] Step S702: Store the obtained image frames in the storage space and construct a sub-video generation task for the image frame sequence;

[0131] Step S703: In the case that there are multiple idle threads in the background threads, determine the historical efficiency of each idle thread in the process of responding to historical tasks.

[0132] Step S704: Based on the historical efficiency of each idle thread, determine the selected background thread from the idle threads to respond to the sub-video generation task; return to step S701 and continue to execute step S705.

[0133] Step S705: Call the selected background thread to encode and encapsulate the image frame sequence corresponding to the sub-video generation task according to the encoding and encapsulation algorithm matched with the client.

[0134] In step S706, if the selected background thread encodes and encapsulates the image frame sequence corresponding to the sub-video generation task, and the task completion information fed back by the selected background thread is detected, it means that the sub-video corresponding to the image frame sequence has been generated.

[0135] In step S707, if an error message is detected from the selected background thread during the process of encoding and encapsulating the image frame sequence corresponding to the sub-video generation task in the selected background thread, the sub-video generation task is placed in the task queue as an unfinished task and the selected background thread is terminated.

[0136] Step S708: When all background threads are busy, add the sub-video generation task as an unprocessed task to the task queue; return to step S701 and continue to execute step S709.

[0137] Step S709: When the number of tasks in the task queue reaches the threshold, obtain the client's performance parameters and determine the thread count threshold that matches the performance parameters.

[0138] Step S710: If the number of background threads that maintain communication with the main thread is less than the thread count threshold, then start a new background thread.

[0139] Step S711: If there are idle threads among the remaining background threads, retrieve the target task from the task queue according to the task processing priority.

[0140] Wherein, the target task is an unprocessed task or an incomplete task; if the target task is an incomplete task, the idle thread is selected as the background thread, and the process returns to step S705.

[0141] Step S712: If the target task is an incomplete task, then obtain the task progress record for the incomplete task.

[0142] Step S713: Based on the task progress, call the idle thread to execute the unfinished part of the target task and generate the sub-video corresponding to the target task;

[0143] Step S714: If the selected background thread sends the sub-video corresponding to the image frame sequence back to the main thread, delete each image frame contained in the image frame sequence in the storage space.

[0144] Step S715: Determine the playback order of each sub-video corresponding to each image frame sequence based on the order of each image frame sequence in the data stream;

[0145] Step S716: Concatenate the sub-videos in the order of playback to obtain and display the complete video containing all the sub-videos.

[0146] In one embodiment, this application also provides an application scenario in which the above-described video processing method is applied. This application scenario could be, for example, a video communication scenario. In this scenario, the client can be a client capable of providing video communication services. Specifically, in a video communication scenario, the sender can upload captured image frames to the server as a data stream through the sender's corresponding client. The receiver can then obtain the image frames sent by the sender from the server as a data stream through the receiver's corresponding client. The receiver's client is equipped with a main thread and multiple background threads connected to the main thread. During video processing, the receiver: calls the main thread to obtain image frames from the server as a data stream, sequentially obtaining multiple image frame sequences; for each image frame sequence's sub-video generation task, the main thread, based on the working state of each background thread, determines the selected background thread to respond to the sub-video generation task; the selected background thread encodes and encapsulates the image frame sequence corresponding to the sub-video generation task, generating the sub-video corresponding to the image frame sequence; the main thread then sequentially displays the sub-videos corresponding to each image frame sequence according to their order in the data stream. Thus, video transmission and display from the sender to the receiver are completed, realizing video communication.

[0147] In one embodiment, this application also provides another application scenario where the video processing method described above is applied. This application scenario could be, for example, a short video browsing scenario. In this scenario, the client can be a client capable of providing video browsing services. Specifically, in the video browsing scenario, as shown in Figure 8, in the initial stage of video generation, image frames are received from the server using streaming processing. This eliminates the need to wait for all image frames to be received before processing begins, which improves the efficiency of the video processing process. For example, if a video consists of 1000 image frames, only the first batch (e.g., 100 frames) needs to be received to obtain the image frame sequence, and processing of that sequence can begin immediately.

[0148] Upon receiving the first batch of image frames, the main thread immediately assigns tasks and starts a WebWorker (background thread), here named WebWorker_No1. The main thread can send these 100 image frames to WebWorker_No1 using the `postMessage` method. After receiving the image frames, WebWorker_No1 calls WebAssembly, where it performs the image transcoding, encoding, and encapsulation process. During this process, WebWorker_No1 decodes the image frames, performs color space conversion, and finally encodes them. This process involves a large amount of computation, but due to the efficiency of WebAssembly, it can be completed in a very short time. After the video is generated, WebWorker_No1 sends the generated video back to the main thread using the `postMessage` method. Upon receiving the video, the main thread can display it on the user interface or perform other processing. In the above process, the communication between the main thread and WebWorker_No1 is asynchronous. This means that after sending the 100 image frames to WebWorker_No1, the main thread can continue executing other tasks without waiting for a response from WebWorker_No1. This avoids blocking the main thread and improves the system's response speed and efficiency.

[0149] To further improve video generation speed, WebWorkers can be used for multi-threaded processing on the client side. This fully utilizes the multi-core processors of modern computers to achieve parallel processing, significantly increasing video generation speed. For example, when the main thread receives the second batch of 100 frames, the client can start a second WebWorker to process this new task in parallel. Let's call this new WebWorker_No2. The main thread sends these 100 image frames to WebWorker_No2 via the postMessage method. Then, WebWorker_No2, like WebWorker_No1, will call WebAssembly to perform the image transcoding, encoding, integration, and encapsulation process. In this way, WebWorker_No1 and WebWorker_No2 can work simultaneously, achieving parallel processing.

[0150] The advantage of this approach is that it significantly increases video generation speed because it eliminates the need to wait for one task to complete before starting the next; multiple tasks can be performed simultaneously. Furthermore, this approach effectively utilizes the computing power of multi-core CPUs, improving overall system performance. Since each WebWorker runs in an independent thread, they can perform calculations on different CPU cores simultaneously, achieving parallel processing. Another advantage of using WebWorkers for parallel processing is that it avoids blocking the main thread. Because all computations occur within the WebWorkers, and the main thread is only responsible for receiving frame images and assigning tasks, even with very heavy computational loads, the main thread's operation remains unaffected, ensuring a smooth user interface.

[0151] In a specific implementation, this application also designs a task allocation strategy during multi-threaded processing to achieve load balancing among multiple WebWorkers. The aim is to ensure that each WebWorker has tasks to perform, avoiding situations where some WebWorkers are idle while others are overloaded. First, the client can monitor the working status of each WebWorker in real time by calling the main thread. This is achieved by monitoring the WebWorker's message and error events. When a WebWorker completes a task, it sends a message to the main thread via the `postMessage` method. Upon receiving this message, the main thread knows that the WebWorker has completed its current task. Similarly, if a WebWorker encounters an error during task execution, it will also send an error message to the main thread.

[0152] If a WebWorker has completed its current task, the main thread can assign a new task to that WebWorker instead of creating a new one. This avoids frequent creation and destruction of WebWorkers, reducing system overhead. It also ensures that each WebWorker is fully utilized, improving system efficiency. Conversely, if a WebWorker crashes, the main thread can promptly terminate the task and restart a new WebWorker and its corresponding task. This is achieved by monitoring the WebWorker's error event. When a WebWorker encounters an error, it sends an error message to the main thread. Upon receiving this message, the main thread knows that the WebWorker has crashed. Then, the main thread terminates the old WebWorker and starts a new one to restart the task.

[0153] This application also designs a mechanism for error retry and incremental retry, aiming to minimize the impact of errors on the system and improve its stability and efficiency. Incremental retry is a handling strategy for incomplete tasks. For tasks that are not completed due to errors during execution, only the incomplete part is re-executed based on the recorded task progress, without re-executing the previously successfully completed part, which helps improve the system's efficiency.

[0154] Specifically, if a WebWorker encounters an error while processing a frame image, the client can re-add the task to the task queue, waiting for the next execution. This is achieved by monitoring the WebWorker's error event. As mentioned earlier, when a WebWorker encounters an error, it sends an error message to the main thread. Upon receiving this message, the main thread knows that an error has occurred. Then, the main thread will re-add the task to the task queue, waiting for the next execution. Upon receiving the error message, the entire task is not re-executed; only the part that malfunctioned is re-executed. This is achieved by recording the execution progress in each task. For example, if a WebWorker encounters an error while processing the 50th frame image, only the part starting from frame 50 needs to be re-executed, without re-executing the previously successfully executed parts. This avoids repeatedly executing successfully executed parts, improving system efficiency.

[0155] The above technical solution can bring at least the following beneficial effects:

[0156] 1. Improve video generation efficiency

[0157] By breaking down the video generation task into multiple smaller tasks and processing them in a distributed manner on the client side, the efficiency of video generation can be greatly improved. This is because each client only needs to process a portion of the task, rather than the entire video generation task, thus reducing the processing time of a single task.

[0158] 2. Improve system stability and reliability

[0159] By using WebWorkers to implement multi-threaded processing on the client side, the speed of video generation can be further improved. At the same time, through reasonable task allocation and scheduling strategies, load balancing can be achieved among multiple clients, preventing overload of a single client and thus improving system stability and reliability.

[0160] 3. Save server resources

[0161] Since most computing tasks are performed on the client side, server computing resources can be significantly saved. This not only reduces server hardware costs but also operational costs, such as electricity and maintenance costs.

[0162] 4. Improve user experience

[0163] Because video generation speed has been improved, users can receive the generated video faster, thus enhancing the user experience. Furthermore, since the task is processed on the client side, users can continue other operations while waiting for the video to be generated without experiencing lag or delays, further improving the user experience.

[0164] 5. Improve system scalability

[0165] Due to the design of task decomposition and distributed processing, the system's processing capacity can increase linearly as the client's capabilities improve.

[0166] It should be understood that although the steps in the flowcharts of the above embodiments are shown sequentially according to the arrows, these steps are not necessarily executed in the order indicated by the arrows. Unless explicitly stated herein, there is no strict order restriction on the execution of these steps, and they can be executed in other orders. Moreover, at least some steps in the flowcharts of the above embodiments may include multiple steps or multiple stages. These steps or stages are not necessarily completed at the same time, but can be executed at different times. The execution order of these steps or stages is not necessarily sequential, but can be performed alternately or in turn with other steps or at least some of the steps or stages of other steps.

[0167] Based on the same inventive concept, this application also provides a video processing apparatus for implementing the video processing method described above. The solution provided by this apparatus is similar to the implementation scheme described in the above method; therefore, the specific limitations in one or more video processing apparatus embodiments provided below can be found in the limitations of the video processing method described above, and will not be repeated here.

[0168] In one embodiment, a video processing apparatus is provided, applied to a client, the client having a main thread and multiple background threads communicatively connected to the main thread. In this embodiment, as shown in FIG9, the video processing apparatus includes an image frame acquisition module 901, a background thread selection determination module 902, a sub-video generation module 903, and a video display module 904, wherein:

[0169] Image frame acquisition module 901 is used to call the main thread to acquire image frames from the server in the form of a data stream, and obtain multiple image frame sequences in sequence.

[0170] The background thread selection module 902 is used to, for each sub-video generation task of each image frame sequence, call the main thread to determine the selected background thread to respond to the sub-video generation task based on the working status of each background thread.

[0171] The sub-video generation module 903 is used to call a selected background thread to encode and encapsulate the image frame sequence corresponding to the sub-video generation task, and generate the sub-video corresponding to the image frame sequence.

[0172] The video display module 904 is used to call the main thread to display the sub-videos corresponding to each image frame sequence in the order of the image frame sequence in the data stream.

[0173] In one embodiment, the background thread selection module 902 is specifically used to: obtain the working status of each background thread; the working status includes a busy state and an idle state; and, if there is an idle thread in the idle state among the background threads, determine the idle thread as the selected background thread for responding to the sub-video generation task. In this embodiment, the video processing device also includes a task queue management module, used to place the sub-video generation task as an unprocessed task into the task queue when all background threads are in a busy state.

[0174] In one embodiment, the number of idle threads is multiple. In this embodiment, the background thread selection module 902 is further configured to: determine the historical efficiency of each idle thread in responding to historical tasks; and, based on the historical efficiency of each idle thread, determine the selected background thread from among the idle threads for responding to the sub-video generation task.

[0175] In one embodiment, the task queue management module is specifically used to: during the process of encoding and encapsulating the image frame sequence corresponding to the sub-video generation task by the selected background thread, if an error message is detected from the selected background thread, the sub-video generation task is placed in the task queue as an incomplete task. In this embodiment, the sub-video generation module 903 is further used to: terminate the selected background thread, and if there is an idle thread among the remaining background threads, retrieve the target task from the task queue according to the task processing priority, and call the idle thread to execute the target task. The target task is either an unprocessed task or an incomplete task.

[0176] In one embodiment, the task queue management module includes a task priority determination unit, configured to: determine the task processing priority of each unprocessed task according to its enqueue time for multiple unprocessed tasks in the task queue; the task processing priority is positively correlated with the enqueue time; and determine the task processing priority of each unfinished task according to its first execution time for multiple unfinished tasks in the task queue; the task processing priority is positively correlated with the first execution time; and the task processing priority of any unfinished task is higher than that of any unprocessed task.

[0177] Specifically, for unprocessed tasks, the enqueue time is used as the priority criterion. The earlier the enqueue time, the higher the task processing priority. A sorting algorithm can be used to sort unprocessed tasks according to their enqueue time, placing tasks with earlier enqueue times at the top. For incomplete tasks, the first execution time is used as the priority criterion. The earlier the first execution time, the higher the task processing priority. Similarly, a sorting algorithm is used to sort incomplete tasks according to their first execution time. When processing tasks, incomplete tasks are retrieved from the task queue first, because the processing priority of any incomplete task is higher than that of any unprocessed task. This ensures that tasks already being processed are completed first, reducing task interruptions and duplicate processing, and improving overall processing efficiency.

[0178] In one embodiment, the sub-video generation module 903 is specifically used to: when the target task is an incomplete task, obtain the task progress recorded for the incomplete task; and according to the task progress, call an idle thread to execute the incomplete part of the target task and generate a sub-video corresponding to the target task.

[0179] In one embodiment, the video processing apparatus further includes a thread management module, configured to: obtain client performance parameters when the number of tasks in the task queue reaches a threshold; determine a thread number threshold that matches the performance parameters; and start a new background thread if the number of background threads maintaining communication with the main thread is less than the thread number threshold.

[0180] In one embodiment, the image frame acquisition module 901 is specifically used to: call the main thread to acquire image frames from the server in the form of a data stream; when the number of acquired image frames reaches a set number, construct an image frame sequence containing each image frame; return to the step of calling the main thread to acquire image frames from the server in the form of a data stream, and sequentially obtain multiple image frame sequences; wherein, each image frame sequence carries a corresponding sequence identifier; the sequence identifier is used to characterize the order of the image frame sequence in the data stream.

[0181] In one embodiment, the video display module 904 is specifically used to: determine the playback order of each sub-video corresponding to each image frame sequence according to the order of each image frame sequence in the data stream; and sequentially splice the sub-videos according to the playback order to obtain and display a complete video containing each sub-video.

[0182] In one embodiment, the video processing apparatus further includes a storage space management module, configured to: store the obtained image frames in the storage space; and delete each image frame contained in the image frame sequence in the storage space when a selected background thread sends the sub-video corresponding to the image frame sequence back to the main thread.

[0183] In one embodiment, the video processing apparatus further includes an algorithm determination module for determining an encoding and encapsulation algorithm that matches the client. In this embodiment, the sub-video generation module 903 is specifically used to: call a selected background thread to encapsulate the image frame sequence corresponding to the sub-video generation task according to the encoding and encapsulation algorithm, thereby generating a sub-video corresponding to the image frame sequence.

[0184] Each module in the aforementioned video processing device can be implemented entirely or partially through software, hardware, or a combination thereof. These modules can be embedded in the processor of a computer device in hardware form or independent of it, or stored in the memory of a computer device in software form, so that the processor can call and execute the operations corresponding to each module.

[0185] In one embodiment, a computer device is provided, which may be a terminal, and its internal structure diagram is shown in Figure 10. The computer device includes a processor, memory, input / output interface, communication interface, display unit, and input device. The processor, memory, and input / output interface are connected via a system bus, and the communication interface, display unit, and input device are also connected to the system bus via the input / output interface. The processor of the computer device provides computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and internal memory. The non-volatile storage medium stores an operating system and computer programs. The internal memory provides an environment for the operation of the operating system and computer programs in the non-volatile storage medium. The input / output interface of the computer device is used for exchanging information between the processor and external devices. The communication interface of the computer device is used for wired or wireless communication with external terminals; wireless communication can be achieved through Wi-Fi, mobile cellular networks, NFC (Near Field Communication), or other technologies. When the computer program is executed by the processor, it implements a video processing method. The display unit of the computer device is used to form a visually visible image. It can be a display screen, a projection device, or a virtual reality imaging device. The display screen can be an LCD screen or an e-ink screen. The input device of the computer device can be a touch layer covering the display screen, or buttons, trackballs, or touchpads set on the casing of the computer device, or external keyboards, touchpads, or mice, etc.

[0186] Those skilled in the art will understand that the structure shown in Figure 10 is merely a block diagram of a portion of the structure related to the present application and does not constitute a limitation on the computer device to which the present application is applied. Specific computer devices may include more or fewer components than those shown in the figure, or may combine certain components, or may have different component arrangements.

[0187] In one embodiment, a computer device is provided, including a memory and a processor, wherein the memory stores a computer program, and the processor executes the computer program to implement the steps in the above-described method embodiments.

[0188] In one embodiment, a computer-readable storage medium is provided having a computer program stored thereon that, when executed by a processor, implements the steps in the above method embodiments.

[0189] In one embodiment, a computer program product is provided, including a computer program that, when executed by a processor, implements the steps in the above method embodiments.

[0190] In summary, this application provides a video processing method, apparatus, computer device, computer-readable storage medium, and computer program product. The client calls the main thread to obtain image frames from the server in a data stream manner, sequentially obtaining multiple image frame sequences. This process employs streaming processing, eliminating the need to wait for all image frames to be received before processing begins. As soon as a certain number of image frames are received to form an image frame sequence, the sequence is immediately processed, reducing long waiting times and re-reception issues caused by network problems, thus improving the efficiency of the video processing process. It also effectively reduces memory usage, as it does not require storing all image frames at once, only the currently processed image frames. For each sub-video generation task of each image frame sequence, the main thread, based on the working status of each background thread, determines a selected background thread to respond to the sub-video generation task. The selected background thread encodes and encapsulates the image frame sequence, generating the corresponding sub-video. The main thread then sequentially displays the sub-videos corresponding to each image frame sequence according to their order in the data stream. This method of shifting video generation from the server to the client reduces the amount of data processing and transmission on the server, thus lowering the server's bandwidth pressure. The client achieves distributed video generation through multi-threaded collaboration, improving the system's parallel processing capabilities, solving the problem of high server load, and enhancing the overall system's resource utilization and processing efficiency.

[0191] Furthermore, when selecting a background thread, the main thread obtains the working status of each background thread, including busy and idle states. If any background thread is idle, it is selected as the background thread to respond to the sub-video generation task. This avoids the enqueue and dequeue operations of putting tasks into a task queue, reducing the time overhead of task processing and ensuring that the sub-video generation task can be responded to in a timely manner, thus improving the timeliness of task processing. If all background threads are busy, the sub-video generation task is placed in the task queue as an unprocessed task, facilitating subsequent task scheduling and allocation based on thread status, ensuring the orderly processing of tasks.

[0192] When there are multiple idle threads, for each idle thread, its historical efficiency in responding to historical tasks is determined. Based on the historical efficiency of each idle thread, a selected background thread for responding to the sub-video generation task is determined from among the idle threads. Historical efficiency reflects the processing capacity and performance of a thread. Selecting an idle thread with high historical efficiency to execute the task can fully leverage the advantages of high-efficiency threads, reduce task processing time, and improve the overall work efficiency of the video processing process.

[0193] During the encoding and encapsulation of image frame sequences by a selected background thread, if an error message is detected from that selected background thread, the sub-video generation task is added to the task queue as an incomplete task, the selected background thread is terminated, and if there are idle threads among the remaining background threads, the target task is retrieved from the task queue according to task processing priority, and the idle thread is invoked to execute the target task. By monitoring thread feedback to promptly detect and handle errors, preventing erroneous threads from continuing to occupy resources and affect other tasks, and by re-adding the task to the task queue for subsequent processing, the stability of the system and the continuity of task processing are ensured. This reduces the impact of a single thread error on the entire system and improves client stability and work efficiency.

[0194] For multiple unprocessed tasks in the task queue, the processing priority of each unprocessed task is determined according to its enqueue time, with a positive correlation between processing priority and enqueue time. For multiple incomplete tasks in the task queue, the processing priority of each incomplete task is determined according to its first execution time, with a positive correlation between processing priority and first execution time, and the processing priority of any incomplete task is higher than that of any other unprocessed task. The enqueue time and first execution time are related to the order of the image frame sequence in the data stream. This prioritization ensures that image frames that appear earlier in the sequence are processed first, guaranteeing the correct order of video display. At the same time, prioritizing incomplete tasks, since they have already been partially executed, results in relatively shorter subsequent execution times, reducing the time occupied by background threads, improving the utilization of background threads, reducing latency in the front-end display process, and ultimately improving the overall performance of the client in the video processing process.

[0195] When an idle thread is invoked to execute a target task that is incomplete, the system retrieves the task progress recorded for the incomplete task. Based on the progress, the idle thread executes the incomplete portion of the target task, generating the corresponding sub-video. By recording task progress, subsequent executions only execute the incomplete portion, avoiding repeated execution of already successful parts. This enables incremental retries for incomplete tasks, reducing wasted computing resources and improving system efficiency.

[0196] When the number of tasks in the task queue reaches a threshold, the system retrieves the client's performance parameters and determines a thread count threshold that matches these parameters. If the number of background threads maintaining communication with the main thread is less than this threshold, a new background thread is started. Dynamically adjusting the number of background threads based on the client's performance parameters allows the system to allocate resources rationally according to the client's actual processing capacity, avoiding resource shortages or waste. This improves video processing efficiency and enhances the system's adaptability and scalability while avoiding wasted computing resources.

[0197] The main thread retrieves image frames from the server as a data stream. Once a set number of image frames are obtained, a sequence of image frames is constructed, and the process returns to the steps where the main thread retrieved image frames from the server as a data stream. Multiple image frame sequences are obtained sequentially, each carrying a corresponding sequence identifier. This sequence identifier indicates the order of the image frames in the data stream. The sequence identifier allows the display of sub-videos to be independent of the order in which the tasks were generated, improving the flexibility of the video processing while ensuring that sub-videos are displayed in the correct order.

[0198] When displaying the sub-videos corresponding to each image frame sequence sequentially according to their order in the data stream, the playback order of the sub-videos is determined based on their order in the data stream. The sub-videos are then concatenated in this playback order to obtain and display the complete video containing all the sub-videos. This method ensures that the data stream is displayed completely, improving the integrity and coherence of the video presentation and enhancing the overall video display effect.

[0199] The acquired image frames are stored in storage space. When a selected background thread sends the sub-video corresponding to the image frame sequence back to the main thread, all image frames contained in that sequence are deleted from storage space. By promptly deleting processed image frames, it is not necessary to store all image frames simultaneously, reducing storage pressure during video processing, improving storage resource utilization, and ultimately enhancing service performance.

[0200] A matching encoding and encapsulation algorithm is determined for the client. A selected background thread is then invoked to encapsulate the image frame sequence corresponding to the sub-video generation task according to this algorithm, generating the sub-video corresponding to the image frame sequence. Considering the differences in hardware configuration and user needs among different clients, using a matching encoding and encapsulation algorithm ensures that the generated sub-video is compatible with the client's display device and user requirements, improving the sub-video's compatibility with the client. Furthermore, the multi-threaded parallel processing method does not cause blocking of the front-end main thread, improving video processing quality without affecting processing efficiency.

[0201] The main thread and background threads communicate through mechanisms such as handlers, signal-slot mechanisms, or event-driven mechanisms. These mechanisms provide stable and efficient channels for information exchange between threads, ensuring that critical information such as task allocation, progress feedback, and error messages can be transmitted between the main thread and background threads in a timely and accurate manner. The main thread can effectively manage and schedule background threads based on this information, enabling collaborative work among multiple threads and guaranteeing the efficiency and stability of multi-threaded cooperation. This, in turn, improves the accuracy and timeliness of task allocation and execution during video processing.

[0202] The background thread uses Web Worker threads, which allow the creation of independent background threads within the browser to perform time-consuming computational tasks. Because Web Worker threads run independently of the main thread, they do not block the main thread when performing video processing tasks, ensuring the webpage remains responsive during video processing, thus improving page speed and user experience. Furthermore, multiple Web Worker threads can process different tasks in parallel, enhancing the client's parallel processing capabilities and improving video processing efficiency.

[0203] The background thread uses a Service Worker thread, which is more independent than a Web Worker. It can continue running after the client page is closed and can be shared across pages. This means that video processing tasks can continue even after the page is closed, avoiding task interruption caused by the user closing the page. At the same time, Service Workers can share processing results between different pages, reducing redundant calculations and data transfer, and improving the system's resource utilization efficiency and data sharing capabilities.

[0204] The background thread uses a Shared Worker thread, which allows the same worker instance to be shared across multiple windows or tabs. Multiple windows or tabs can share a single Shared Worker thread for video processing tasks, reducing redundant resource usage and lowering system resource consumption. This sharing mechanism is suitable for scenarios where data needs to be shared across multiple pages, improving system resource reusability and reducing resource waste.

[0205] During video display, the client first concatenates multiple child videos into a parent video, and then renders the next parent video while displaying the previous one. This pre-rendering method increases the background preparation time during video display, ensuring that the next video is processed and prepared while the current video is being displayed. This guarantees continuous video display, reduces stuttering and latency during video display, and improves the smoothness of video display.

[0206] In multi-threaded processing, the working status of WebWorkers is monitored in real time, and new tasks are assigned to them when a WebWorker completes a task. This task allocation strategy avoids the frequent creation and destruction of WebWorker threads, reducing the overhead of thread creation and destruction, ensuring that each WebWorker can fully utilize its processing power, and improving the overall efficiency of the system. Simultaneously, when a WebWorker crashes, the task can be terminated promptly and a new WebWorker and task can be started, ensuring the normal execution of tasks and improving the stability and reliability of the system.

[0207] Error retry and incremental retry mechanisms re-add erroneous tasks to the task queue for execution, and execute the unfinished parts of incomplete tasks according to their progress. This mechanism can promptly handle errors that occur during task execution, preventing erroneous tasks from being ignored or repeated. For incomplete tasks, only the unfinished parts are executed, avoiding redundant calculations, saving computing resources, and improving the system's efficiency and stability.

[0208] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, data stored, data displayed, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties, and the collection, use and processing of the relevant data must comply with relevant regulations.

[0209] Those skilled in the art will understand that all or part of the processes in the methods of the above embodiments can be implemented by a computer program instructing related hardware. The computer program can be stored in a non-volatile computer-readable storage medium, and when executed, it can include the processes of the embodiments of the above methods. Any references to memory, databases, or other media used in the embodiments provided in this application can include at least one of non-volatile and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical memory, high-density embedded non-volatile memory, resistive random access memory (ReRAM), magnetic random access memory (MRAM), ferroelectric random access memory (FRAM), phase change memory (PCM), graphene memory, etc. Volatile memory can include random access memory (RAM) or external cache memory, etc. By way of illustration and not limitation, RAM can take many forms, such as Static Random Access Memory (SRAM) or Dynamic Random Access Memory (DRAM). The databases involved in the embodiments provided in this application may include at least one type of relational database and non-relational database. Non-relational databases may include, but are not limited to, blockchain-based distributed databases. The processors involved in the embodiments provided in this application may be general-purpose processors, central processing units, graphics processing units, digital signal processors, programmable logic devices, quantum computing-based data processing logic devices, etc., and are not limited to these.

[0210] The technical features of the above embodiments can be combined in any way. For the sake of brevity, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0211] The embodiments described above are merely illustrative of several implementation methods of this application, and while the descriptions are relatively specific and detailed, they should not be construed as limiting the scope of the invention patent. It should be noted that those skilled in the art can make various modifications and improvements without departing from the concept of this application, and these all fall within the protection scope of this application. Therefore, the protection scope of this patent application should be determined by the appended claims.

Claims

1. A video processing method applied to a client, wherein the client has a main thread and multiple background threads communicatively connected to the main thread; the method includes: The main thread is invoked to obtain image frames from the server in the form of a data stream, and multiple image frame sequences are obtained sequentially. For each sub-video generation task of the image frame sequence, the main thread is invoked to determine the selected background thread for responding to the sub-video generation task based on the working status of each background thread. The selected background thread is invoked to encode and encapsulate the image frame sequence corresponding to the sub-video generation task, thereby generating the sub-video corresponding to the image frame sequence. and The main thread is invoked to sequentially display the sub-videos corresponding to each of the image frame sequences in the data stream, according to the order of each image frame sequence.

2. The method according to claim 1, wherein determining the selected background thread for responding to the sub-video generation task from the background threads based on the respective working state of each background thread includes: Obtain the working status of each of the background threads; The working status includes a busy status and an idle status; If there is an idle thread in the background threads, the idle thread is determined as the selected background thread for responding to the sub-video generation task; When all the background threads are busy, the sub-video generation task is placed in the task queue as an unprocessed task.

3. The method according to claim 2, wherein the number of idle threads is multiple; the method further includes: For each of the aforementioned idle threads, determine the historical efficiency of the idle thread in responding to historical tasks; Based on the historical efficiency of each of the idle threads, a selected background thread for responding to the sub-video generation task is determined from the idle threads.

4. The method according to claim 2 or 3, further comprising: If an error message is detected by the selected background thread during the encoding and encapsulation of the image frame sequence corresponding to the sub-video generation task, the sub-video generation task will be placed in the task queue as an incomplete task. Terminate the selected background thread. If there is an idle thread among the remaining background threads, retrieve the target task from the task queue according to the task processing priority, and call the idle thread to execute the target task; the target task is an unprocessed task or an incomplete task.

5. The method according to claim 4, further comprising: For multiple unprocessed tasks in the task queue, the task processing priority of each unprocessed task is determined according to the enqueue time of each unprocessed task. The task processing priority is positively correlated with the queuing time; For multiple unfinished tasks in the task queue, the task processing priority of each unfinished task is determined according to its first execution time; the task processing priority is positively correlated with the first execution time; the task processing priority of any unfinished task is higher than that of any unprocessed task.

6. The method according to claim 4 or 5, wherein invoking the idle thread to execute the target task includes: If the target task is an incomplete task, obtain the task progress recorded for the incomplete task; Based on the task progress, the idle thread is invoked to execute the unfinished part of the target task and generate the sub-video corresponding to the target task.

7. The method according to any one of claims 2 to 6, wherein the method further comprises: When the number of tasks in the task queue reaches a threshold, obtain the client's performance parameters; Determine the thread count threshold that is compatible with the performance parameters; If the number of background threads that maintain communication with the main thread is less than the thread count threshold, then a new background thread is started.

8. The method according to any one of claims 1 to 7, wherein calling the main thread to obtain image frames from the server in the form of a data stream, and sequentially obtaining a plurality of image frame sequences, includes: The main thread is invoked to obtain image frames from the server in the form of a data stream. When the number of obtained image frames reaches a set number, an image frame sequence containing each of the image frames is constructed. Returning to the step of calling the main thread to obtain image frames from the server in the form of a data stream, multiple image frame sequences are obtained sequentially; Each of the aforementioned image frame sequences carries a corresponding sequence identifier; The sequence identifier is used to characterize the order of the image frame sequence in the data stream.

9. The method according to claim 8, wherein displaying the sub-video corresponding to each of the image frame sequences sequentially according to their order in the data stream comprises: Based on the order of each image frame sequence in the data stream, determine the playback order of the corresponding sub-video for each image frame sequence; By sequentially piecing together the sub-videos according to the playback order, a complete video containing all the sub-videos is obtained and displayed.

10. The method according to any one of claims 1 to 9, wherein the method further comprises: Store the obtained image frames to the storage space; When the selected background thread sends the sub-video corresponding to the image frame sequence back to the main thread, each image frame contained in the image frame sequence is deleted from the storage space.

11. The method according to any one of claims 1 to 10, wherein the method further comprises: Determine the encoding and encapsulation algorithm that matches the client; The step of calling the selected background thread to encode and encapsulate the image frame sequence corresponding to the sub-video generation task, and generating the sub-video corresponding to the image frame sequence, includes: The selected background thread is invoked to encode and encapsulate the image frame sequence corresponding to the sub-video generation task according to the encoding and encapsulation algorithm, thereby generating the sub-video corresponding to the image frame sequence.

12. A video processing apparatus, applied to a client, the client having a main thread and multiple background threads communicatively connected to the main thread; the apparatus comprising: The image frame acquisition module is used to call the main thread to acquire image frames from the server in the form of a data stream, and obtain multiple image frame sequences in sequence. The background thread selection module is used to, for each of the image frame sequences, call the main thread to determine the selected background thread for responding to the sub-video generation task based on the working status of each of the background threads. The sub-video generation module is used to call the selected background thread to encode and encapsulate the image frame sequence corresponding to the sub-video generation task, and generate the sub-video corresponding to the image frame sequence. The video display module is used to call the main thread to sequentially display the sub-videos corresponding to each of the image frame sequences in the data stream according to the order of each image frame sequence.

13. A computer device comprising a memory and a processor, the memory storing a computer program, the processor executing the computer program to implement the steps of the method according to any one of claims 1 to 11.

14. A computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the steps of the method according to any one of claims 1 to 11.

15. A computer program product comprising a computer program that, when executed by a processor, implements the steps of the method according to any one of claims 1 to 11.

Images