Subject person image segmentation method, apparatus and device, and medium

Abstract

A subject person image segmentation method, apparatus and device, and a medium, relating to the technical field of network live streaming. The method comprises: detecting an image region where each person is located in an image frame of a live stream, and determining a region anchor box corresponding to the image region where each person is located; on the basis of a preset significance determination condition, determining, from among all the region anchor boxes, as a subject anchor box, a region anchor box corresponding to an image region where a subject person is located; performing tracking and smoothing processing on the subject anchor box to stabilize position information of the subject anchor box; and performing image segmentation on the image frame in the live stream on the basis of the position information of the subject anchor box to obtain a human body image of the subject person. The present application overcomes the limitations of conventional portrait segmentation technology in multi-person scenarios, and achieves accurate recognition and stable segmentation of subject persons by means of innovative technical means, thereby providing a more efficient and accurate technical solution for related application scenarios.

Description

Subject Image Segmentation Methods, Apparatus, Equipment, and Media

[0001] This application claims priority to Chinese Patent Application No. 202510234751.8, filed on February 28, 2025, entitled "Method and Apparatus, Device and Medium for Segmenting Subject Image", the entire contents of which are incorporated herein by reference. Technical Field

[0002] This application relates to live streaming technology, and more particularly to a method, apparatus, and medium for segmenting images of main figures. Background Technology

[0003] In the fields of computer vision and image processing, human segmentation technology has always been a research hotspot, widely used in image editing, virtual reality, augmented reality, video conferencing, and many other scenarios. However, existing traditional technologies have some obvious limitations when handling multi-person scenes, and these problems are particularly prominent in practical applications.

[0004] First, traditional portrait segmentation techniques typically treat all people in an image as segmentation targets, failing to effectively distinguish between main figures in the foreground and irrelevant figures in the background. For example, in video conferencing or live streaming scenarios, it is often necessary to exclude irrelevant figures in the background from the segmentation range to achieve better visual effects, such as background blurring or replacement. However, existing segmentation techniques cannot meet this requirement, resulting in segmentation results that may include unnecessary background figures, affecting the final visual effect.

[0005] Secondly, traditional technologies have significant shortcomings in terms of real-time performance. Most existing face segmentation algorithms have high computational complexity, making it difficult to meet the low-latency requirements of real-time applications, especially on devices with lower computing power, such as mobile devices or PCs with low-end graphics cards. This poor real-time performance limits the widespread application of face segmentation technology in practical scenarios, particularly in live video streaming and video conferencing where high real-time performance is required.

[0006] Furthermore, traditional technologies also suffer from instability in multi-person scenarios. When there are multiple moving targets in the scene, existing tracking and segmentation technologies often struggle to accurately distinguish and continuously track the main person, and are easily affected by interference from other people or the background, leading to unstable segmentation results, or even misjudgment or loss of targets.

[0007] These problems severely limit the application of traditional portrait segmentation technology in multi-person scenarios. Therefore, there is an urgent need for a new technological solution that can effectively address these issues, enabling accurate segmentation of the most prominent figures in an image while meeting real-time and stability requirements, thus better adapting to the needs of various practical application scenarios. Summary of the Invention

[0008] The purpose of this application is to solve the above-mentioned problems by providing a method for segmenting a subject image, as well as corresponding apparatus, devices, and non-volatile readable storage media.

[0009] According to one aspect of this application, a method for segmenting a subject image is provided, comprising the following steps: detecting the image region where each person is located in an image frame of a live stream, and determining the region anchor box corresponding to the image region where each person is located; determining, according to a preset saliency criterion, the region anchor box corresponding to the image region where the subject is located as the subject anchor box from all region anchor boxes; performing tracking and smoothing processing on the subject anchor box to stabilize the position information of the subject anchor box; and performing image segmentation on the image frames in the live stream according to the position information of the subject anchor box to obtain a human image of the subject.

[0010] According to another aspect of this application, a subject image segmentation apparatus is provided, comprising: an anchor frame detection module, configured to detect the image region where each person is located in an image frame of a live stream, and determine the region anchor frame corresponding to the image region where each person is located; a subject determination module, configured to determine the region anchor frame corresponding to the image region where the subject is located from all region anchor frames as the subject anchor frame according to a preset saliency determination condition; a tracking and smoothing module, configured to track and smooth the subject anchor frame to stabilize the position information of the subject anchor frame; and a segmentation processing module, configured to perform image segmentation on the image frames in the live stream according to the position information of the subject anchor frame to obtain a human image of the subject.

[0011] According to another aspect of this application, a subject image segmentation device is provided, including a central processing unit and a memory, wherein the central processing unit is used to invoke and run a computer program stored in the memory to perform the steps of the subject image segmentation method described in this application.

[0012] According to another aspect of this application, a non-volatile readable storage medium is provided, which stores a computer program implemented according to the subject image segmentation method in the form of computer-readable instructions, wherein the computer program, when invoked by a computer, executes the steps included in the method.

[0013] This application addresses the limitations of traditional portrait segmentation techniques in multi-person scenes. By detecting the region anchor boxes of each person in the live stream and selecting the region anchor boxes of the main person based on saliency criteria, it can accurately identify and stably track the main person. Combining tracking and smoothing processing ensures the stability and accuracy of the main person's anchor boxes in consecutive frames, and the main person's image is accurately extracted through image segmentation.

[0014] Accordingly, this application achieves significant beneficial effects, including but not limited to: First, it solves the problem that traditional technologies cannot distinguish between the main subject and the background, improving the accuracy and practicality of segmentation; Second, it achieves efficient and stable real-time segmentation on low-performance devices, meeting the needs of scenarios with high real-time requirements such as video conferencing and live streaming; Third, through tracking and smoothing processing, it solves the problem of unstable segmentation results in multi-person scenarios, ensuring stable tracking and accurate segmentation of the main subject. Attached Figure Description

[0015] Figure 1 shows an exemplary network architecture suitable for applying the subject image segmentation method of this application;

[0016] Figure 2 is a flowchart illustrating an embodiment of the main character image segmentation method of this application;

[0017] Figure 3 is a schematic diagram of the main figure image segmentation device of this application;

[0018] Figure 4 is a schematic diagram of the structure of a subject image segmentation device used in this application. Detailed Implementation

[0019] Before detailing the specific embodiments of the technical solution of this application, we will first disclose the network architecture and application scenarios suitable for supporting the implementation of the technical solution of this application.

[0020] The technical solution of this application is applicable to various video processing fields, and is particularly suitable for scenarios that require accurate identification and segmentation of main characters in video streams, such as live streaming, video conferencing, virtual reality, and augmented reality. In this context, the technical solution of this application can be applied in a typical live streaming platform, as shown in Figure 1. This platform consists of a live streaming server 83, a media server 85, terminal devices 90 for broadcasters, and terminal devices 92 for viewers.

[0021] The live streaming server 83 is responsible for managing the creation of live streaming rooms and the reception and distribution of live streams. As the central node of the live stream, it receives the live stream from the broadcaster's user terminal device 90 and distributes it to the media server 85 and the viewer's user terminal device 92. The live streaming server 83 is also responsible for handling live streaming room management tasks, such as user authentication and live streaming room settings.

[0022] Media server 85 is responsible for encoding, transcoding, and storing the live stream. It performs necessary transcoding based on the viewer's network conditions and device capabilities to ensure the live stream is transmitted to the viewer in a suitable format and quality. Media server 85 is also responsible for storing the live stream for playback or other subsequent processing.

[0023] The broadcaster's terminal device 90 is the source of the live broadcast content. The broadcaster captures video through the terminal device 90 and sends the raw video stream to the live broadcast server 83. The broadcaster's terminal device 90 can be a smartphone, tablet, laptop, or professional camera equipment, which is connected to the live broadcast server 83 via the Internet.

[0024] The viewer's terminal device 92 is the receiving end of the live broadcast content. Viewers watch the live broadcast through terminal devices 92, which can be smartphones, tablets, personal computers, or smart TVs, etc. The viewer's terminal device 92 connects to the live broadcast server 83 via the Internet to receive and play the live broadcast stream.

[0025] The live stream transmission process is as follows: The broadcaster's terminal device 90 captures video and notifies the live streaming server 83. The live streaming server 83 distributes the live stream to the media server 85. The media server 85 encodes and transcodes the live stream, then sends the processed live stream's network address back to the live streaming server 83. The live streaming server 83 then transmits the live stream's network address to the viewer's terminal device 92, so that the viewer's terminal device 92 can load and play the live stream based on the network address.

[0026] The subject image segmentation method of this application can be programmed into a computer program product. This computer program product can be selectively deployed on nodes such as the broadcaster's terminal device 90, the media server 85, or the viewer's terminal device 92, to identify the subject image corresponding to the subject in the image frames of the live stream processed by that node. The program product includes the algorithms and logic required to implement the technical solution of this application, and can accurately segment the subject in the video frames of the live stream to enable various subsequent applications, including but not limited to background blurring, background replacement, character beautification, and interactive functions such as virtual gift-giving to specific subjects.

[0027] Please refer to Figure 2. According to the subject image segmentation method provided in this application, it can be implemented as a computer program product, installed and run on various node devices of a live streaming platform, such as the terminal devices of broadcasters, media servers, or viewer users, to be responsible for performing subject image segmentation processing in image frames within the live stream. In some embodiments of this method, the method includes the following steps:

[0028] Step S3100: Detect the image region where each person is located in the image frame of the live stream, and determine the region anchor box corresponding to the image region where each person is located;

[0029] Taking the exemplary live streaming scenario of this application as an example, when it is necessary to identify the main figures in the live stream and segment human images, the image region where each person is located in the image frame of the live stream is first detected, and the corresponding region anchor box of each person's image region is determined. The purpose is to accurately identify the position and range of all people from each frame of the live stream, providing basic data for subsequent main figure identification and image segmentation. In practical application scenarios, multiple people may exist in the live stream at the same time, so it is necessary to be able to accurately distinguish and identify each person to ensure the accuracy and effectiveness of subsequent processing.

[0030] Specifically, region anchor boxes can be implemented in various ways. For example, deep learning-based object detection models such as the YOLO (You Only Look Once) series, SSD (Single Shot MultiBox Detector), or Faster R-CNN can be used. These models can quickly and accurately detect people in images and generate a bounding box, or region anchor box, for each person. These models learn the features of people through a large amount of training data, thus enabling them to accurately locate people in complex backgrounds. In practical applications, the appropriate detection model can be selected based on the resolution, frame rate, and computing power of the live stream. For example, on mobile devices with limited computing resources, lightweight models such as YOLOv5 or NanoDet can be preferred. These models can run at high speed while ensuring detection accuracy, meeting real-time requirements.

[0031] Furthermore, to further improve the accuracy and robustness of detection, multi-scale detection techniques can be incorporated. Multi-scale detection effectively addresses the issue of inconsistent person sizes by sampling and detecting images at different scales, especially in live streaming scenarios where people may appear different sizes depending on their distance from the camera. For example, pyramid downsampling can be performed on the image to generate image copies at different resolutions. Person detection can then be performed separately on each copy, and the results can be merged to obtain a more comprehensive and accurate detection outcome.

[0032] The region anchor boxes in this application refer to the rectangular boxes output by the object detection model used to identify the location and extent of a person. The coordinates of the region anchor boxes are typically centered at the top-left corner of the image and represented by four parameters: the x-coordinate of the top-left corner, the y-coordinate of the top-left corner, the width of the box, and the height of the box. These parameters uniquely determine the position and size of a person in the image. For example, assuming a person is detected in a 1920×1080 resolution image frame with a region anchor box of (500, 300, 200, 400), this means the person is located slightly to the right of the center of the image, with their head roughly in the upper half of the image.

[0033] In multi-person scenarios, object detection models can generate multiple region anchor boxes, each corresponding to a person. These region anchor boxes provide precise location information for subsequent main person identification, ensuring that the main person can be accurately distinguished from multi-person scenarios.

[0034] In some embodiments, target detection can be performed at intervals of a certain number of image frames to determine region anchor boxes in these image frames, and then the subject anchor box can be determined in step S3200. For image frames that are not detected in the middle, the subject anchor box obtained after tracking and smoothing the determined subject anchor box in step S3300 of this application can be used instead, until a new subject anchor box is determined in the next interval, and so on, iterating continuously. This saves system overhead and improves the efficiency of subject and person recognition.

[0035] Step S3200: Based on the preset saliency determination conditions, determine the region anchor frame corresponding to the image region where the main subject is located from all region anchor frames as the main anchor frame;

[0036] After the positions and ranges of all people in each frame of the live stream are accurately identified by region anchor boxes, the region anchor box corresponding to the image area where the main person is located can be determined from all region anchor boxes according to the preset saliency judgment conditions. This is to accurately identify the most prominent person, i.e. the main person, in a multi-person scene, so that subsequent image segmentation processing can focus on the main person, thereby achieving accurate image segmentation results.

[0037] The criteria for determining salience can be predefined based on various factors, such as the position, size, and movement of a person in the image. In one embodiment, the salience criteria are determined based on the position and size of the person in the image. For example, a salience criterion can be preset, considering the person closest to the image center and with the largest area as a salience person. The rationale for this condition is that the anchor is usually located in the center of the frame and occupies a large proportion of the image. In practical applications, salience people can be determined by calculating the distance between the center point of each region's anchor frame and the image center point, and comparing the area sizes of each region's anchor frame.

[0038] Besides salience criteria based on position and size, another embodiment that better supports motion scenarios can also incorporate the motion state of the characters to set salience criteria. For example, a salience criterion can be preset to consider characters with faster movement speeds as salience characters. The rationale for this criterion is that the broadcaster may have many actions and movements during the live stream. In practical applications, salience characters can be determined by tracking the region anchor boxes in consecutive frames and calculating the movement speed of each character.

[0039] In another embodiment, multiple action information reflected by the image of a person in the live stream can be combined to set saliency determination conditions. For example, these conditions can be set according to any one or more of the person's facing angle, eye focus duration, etc., so as to determine the main anchor frame from multiple regional anchor frames.

[0040] In some complex live streaming scenarios, it may be necessary to consider multiple factors to determine saliency criteria. For example, factors such as a person's position, size, facing angle, eye focus duration, and motion state can be considered simultaneously. By assigning weights, these factors can be combined for a comprehensive judgment. For instance, a pre-defined comprehensive saliency criterion can be used, where the weight of position is 0.4, the weight of size is 0.3, and the weight of motion state is 0.3. This approach can more accurately identify salient figures, thereby improving the accuracy and robustness of subject image segmentation.

[0041] Therefore, this application can accurately determine the corresponding region anchor frame indicating the location of the main person from all region anchor frames in each frame of a live stream, based on preset saliency determination conditions. This process not only considers the possibility of multiple people being present in a live stream, but also improves the accuracy and robustness of salient person recognition through the flexible application of various saliency determination conditions, providing accurate location information of the main person for subsequent image segmentation processing.

[0042] Step S3300: Track and smooth the main anchor frame to stabilize its position information;

[0043] After determining the anchor frame of the main subject in the live stream, tracking and smoothing of the anchor frame are necessary to ensure stable and accurate positional information of the main subject across consecutive image frames. This addresses the issue of anchor frame position fluctuations or discontinuities in multi-person scenes caused by factors such as subject movement, camera shake, or detection noise.

[0044] Specifically, tracking and smoothing techniques can be implemented using various methods. In one embodiment, a Kalman filter can be used to track and smooth the position information of the subject's anchor frame. The Kalman filter is a highly efficient self-recursive filter capable of estimating the dynamic state of a system from a series of noisy measurements. In this application, the Kalman filter first predicts the subject's anchor frame's position in the next frame based on its historical position information, and then corrects the predicted position by combining it with the actual detection results of the current frame, thereby updating the subject's anchor frame's position information. This process not only effectively reduces positional jitter caused by detection noise but also provides a stable predicted position in image frames where no subject is detected.

[0045] Besides the Kalman filter, other smoothing techniques can be used, such as moving average filtering or median filtering. Moving average filtering smooths position information by averaging the anchor frame positions over several consecutive frames, while median filtering takes the median of the positions over several consecutive frames. Both of these implementations can effectively reduce jitter in position information.

[0046] Furthermore, to further improve the robustness of tracking, some embodiments may incorporate multi-target tracking algorithms, such as SORT (Simple Online and Realtime Tracking) or DeepSORT. These algorithms, by jointly considering the target's appearance features and motion information, can more accurately track and associate the main subject in consecutive frames, maintaining stable tracking performance even in cases of occlusion or target loss.

[0047] By tracking and smoothing the subject's anchor frame, the position information of the subject's anchor frame in consecutive frames can be ensured to be stable and accurate. This not only improves the robustness of subject image segmentation but also provides high-quality input data for subsequent image segmentation processing, thereby achieving accurate subject image segmentation results.

[0048] Step S3400: Perform image segmentation on the image frames in the live stream based on the position information of the main anchor frame to obtain the human body image of the main figure.

[0049] After the above steps, the subject anchor frame corresponding to the main figure in each image frame of the live stream is determined. When the subject anchor frame is detected from the image frame using an interval frame method, this detected subject anchor frame can be used. For image frames that were not detected, the subject anchor frame obtained by tracking and smoothing the previously detected subject anchor frame can be used. After determining the subject anchor frame corresponding to each image frame, image segmentation can be performed on the image frames in the live stream based on the position information of the subject anchor frame to obtain the human body image of the main figure.

[0050] Specifically, image segmentation can be achieved in various ways. One approach is to use deep learning-based segmentation models, such as Mask R-CNN, the DeepLab series, or U-Net. These models learn the facial features of people through extensive training data, enabling them to accurately segment the contours of people in complex backgrounds. For example, the Mask R-CNN model can not only detect the location of people but also generate a segmentation mask for each person. This mask is a binary image with the same resolution as the original image, where pixel values ​​for the person region are 1 and pixel values ​​for the background region are 0. By multiplying the segmentation mask by the original image, the human body image of the subject and the corresponding background image can be extracted.

[0051] In another embodiment, to further improve the efficiency and accuracy of segmentation, an attention-based image segmentation model can be employed. The attention mechanism allows the model to focus more on human figures in the image, thereby improving segmentation accuracy. For example, the SeaTopFormer architecture can be used. This architecture, by introducing an attention mechanism, can better handle human figure segmentation in multi-person scenes while remaining lightweight. Through optimized attention design, it improves the model's ability to recognize human contours, especially showing more significant segmentation results at human edges and in complex backgrounds.

[0052] In practical applications, image segmentation based on the position information of the subject's anchor frame can be implemented as follows: First, based on the position information of the subject's anchor frame (such as the coordinates of the top left corner, width, and height), the image region containing the subject is cropped from the image frames of the live stream. Then, the cropped image region is input into a pre-trained image segmentation model. The image segmentation model generates a segmentation mask by learning the features of the subject. The segmentation mask is a binary image used to distinguish between the subject and the background. Next, the segmentation mask is multiplied pixel-by-pixel with the original image frame to extract the human image of the subject. Simultaneously, the background image can also be extracted as needed for subsequent background processing. Furthermore, the extracted subject image and background image are further processed, such as applying background blur, background replacement, or subject effects, to enhance the visual effect of the live stream. Finally, taking a live streaming scenario as an example, the processed image frame can replace the original image frame in the live stream, making the live stream include the corresponding processing effects, and then output to the live streaming room for playback.

[0053] Furthermore, to improve the robustness and adaptability of segmentation, multi-scale segmentation techniques can be combined. Multi-scale segmentation, by segmenting the image at different resolutions, can better handle situations where people are of varying sizes. For example, pyramid downsampling can be performed on the image to generate image copies at different resolutions, then segmentation can be performed separately on each copy, and finally the segmentation results can be merged to obtain a more comprehensive and accurate segmentation result.

[0054] Accordingly, this application can accurately segment image frames in a live stream based on the position information of the main anchor frame, extracting the human body image of the main person. This process not only considers the possibility of multiple people appearing simultaneously in a live stream, but also improves the accuracy and real-time performance of segmentation through various technical means, providing high-quality images of the main person for subsequent image processing and applications.

[0055] Through the above embodiments, this application achieves accurate segmentation of the main character image in a live stream, gaining significant technical advantages, including but not limited to:

[0056] First, by detecting the image region where each person is located in the image frame of the live stream and determining the region anchor box, this application can accurately identify the position and range of all people, providing a solid foundation for subsequent main person identification. Combining preset saliency criteria, the main person anchor box is selected from all region anchor boxes, further ensuring accurate identification of the most salient person, i.e., the main person, in multi-person scenes. This process not only considers various factors such as the person's position, size, and motion state in the image, but also improves the accuracy and robustness of salient person identification by flexibly applying various saliency criteria.

[0057] Secondly, regarding real-time performance, this application significantly reduces computational resource consumption and improves processing speed by employing a lightweight object detection model and an interval frame detection strategy, meeting the needs of applications with high real-time requirements, such as live streaming. Furthermore, by using a Kalman filter with low computational overhead to track and smooth the subject's anchor frame, this application not only stabilizes the anchor frame's position information but also further enhances the tracking effect in consecutive frames. Even in scenes with fast-moving figures or complex backgrounds, accurate segmentation is maintained, ensuring the real-time performance and stability of the subject image segmentation.

[0058] The technological advantages of this application are also reflected in its rich application capabilities. Through precise positioning and stable tracking of the main anchor frame, and efficient segmentation of image frames, this application can be widely applied to various video processing scenarios, such as live streaming, video conferencing, and virtual reality. In these scenarios, this application can not only achieve effects such as background blurring and background replacement, but also provide high-quality main character images for interactive functions such as character effects and virtual gift giving, greatly enhancing the user experience and visual effects.

[0059] Based on any embodiment of the method in this application, according to a preset salience determination condition, the region anchor frame corresponding to the image region where the main subject is located is determined from all region anchor frames as the main subject anchor frame, including:

[0060] Step S3211: Calculate the distance between the anchor frame of each region and the image center of its corresponding image frame;

[0061] Calculating the distance between each region anchor frame and the image center of its corresponding image frame quantifies the positional information of the region anchor frame corresponding to each detected person in the image. The image center typically refers to the geometric center of the image, while the center point of the region anchor frame can be calculated using its top-left corner coordinates, width, and height. By calculating the Euclidean distance between these two center points, the positional deviation of each region anchor frame relative to the image center can be obtained. For example, in a 1920×1080 resolution image frame, if the center point coordinates of a region anchor frame are (800, 500), and the image center coordinates are (960, 540), then the distance between the region anchor frame and the image center can be calculated using the formula... The calculation yielded the result.

[0062] Step S3212: Calculate the area of ​​the anchor frame in each region;

[0063] Calculating the area of ​​each region's anchor frame is an effective way to measure the proportion of each person's image within the frame. The area of ​​a region's anchor frame can be calculated by multiplying its width and height; this metric reflects the amount of space a person occupies in the image. For example, if a region's anchor frame is 200 pixels wide and 400 pixels high, its area is 80,000 square pixels. The size of this area can serve as an important criterion for judging the salience of a person, because in most cases, the main subject tends to occupy a large proportion of the image.

[0064] Step S3213: Sort the region anchor frames according to the distance and area;

[0065] Based on the calculated distances and areas, sorting all region anchor frames can be achieved in various ways. For example, in one embodiment, they can be first sorted by area from largest to smallest, and then, for region anchor frames with the same area, sorted by distance from smallest to largest. The rationale for this sorting method is that region anchor frames with larger areas are more likely to correspond to the main subject, while in the case of similar areas, region anchor frames closer to the image center are more likely to correspond to the main subject.

[0066] In another embodiment, a weighted approach can be used to comprehensively consider distance and area factors. For example, the area weight can be set to 0.6 and the distance weight to 0.4, and the area anchor frames can be sorted by weighted sum.

[0067] Step S3214: Using the highest ranking as the significance criterion, select the highest ranking region anchor frame as the main anchor frame.

[0068] To determine the most salient area anchor frame as the subject anchor frame, the area anchor frame ranked first based on distance and area is used as the subject anchor frame. This salience criterion is based on the assumption that in a multi-person scene, the most salient person is often the one located in the center of the frame and occupying a large proportion of the screen. Through the above ranking process, the area anchor frame ranked first is considered the anchor frame of the area containing the person that best meets this salience criterion. For example, in a live broadcast scene, the broadcaster is usually located in the center of the frame and occupies a large proportion of the screen; therefore, the subject anchor frame determined through the above steps can accurately indicate the broadcaster's location.

[0069] The above embodiments accurately determine the anchor frames of the main figures by calculating the distance and area of ​​each region's anchor frame from the image center and sorting them according to these quantitative indicators. This process is not only computationally lightweight but also highly efficient and rapid. First, by calculating the distance and area, the position and size of each person in the image can be quantitatively assessed, providing a precise basis for subsequent sorting. Second, the sorting process can employ various flexible methods, such as sorting by area first and then by distance, or using a weighted sum method to comprehensively consider distance and area; these methods can effectively improve the accuracy of sorting. Finally, using the anchor frame of the region with the highest sorting position as the main figure's anchor frame ensures accurate identification of the most prominent person, i.e., the main figure, in multi-person scenes. This process has low computational load and fast recognition speed, making it suitable for scenarios with high real-time requirements, such as live streaming, and can provide efficient and accurate technical support for related applications.

[0070] Based on any embodiment of the method in this application, according to a preset salience determination condition, the region anchor frame corresponding to the image region where the main subject is located is determined from all region anchor frames as the main subject anchor frame, including:

[0071] Step S3221: Continuously track the region anchor box of each person and extract the motion feature information of each person, including the motion speed, the frequency of change of motion direction and the dwell time in the image of the live stream.

[0072] By continuously tracking the bounding boxes of each person, motion feature information can be extracted from the live stream. Continuous tracking can be achieved through various techniques, such as using Kalman filters or the SORT (Simple Online and Realtime Tracking) algorithm. These algorithms can predict and correct the position information of a person based on changes in position across consecutive frames, thus achieving continuous tracking. Through continuous tracking, the motion trajectory of each person can be obtained, and their motion speed and the frequency of change in motion direction can be calculated. For example, motion speed can be obtained by calculating the rate of change of the person's position across consecutive frames, while the frequency of change in motion direction can be determined by analyzing the number of times the person's motion direction changes. In addition, the dwell time of each person in the live stream image can be counted, that is, the length of time a person remains stationary in a certain area. These motion feature information collectively constitute the dynamic behavioral characteristics of the person.

[0073] Step S3222: Based on the motion characteristic information of each person, apply the preset activity scoring model to determine the corresponding activity score for each person.

[0074] By obtaining the motion characteristic information of each individual, a pre-defined activity scoring model can be applied to determine the corresponding activity score for each individual. The activity scoring model can be designed according to different application scenarios and needs. In one embodiment, features such as motion speed, frequency of change in motion direction, and dwell time can be weighted and summed to obtain a comprehensive activity score. The weight allocation can be adjusted according to the degree of importance given to different features in the actual application. For example, in a live streaming scenario primarily focused on dynamic performance, the weights of motion speed and frequency of change in motion direction can be set higher, while in a live streaming scenario primarily focused on explanation, the weight of dwell time can be appropriately increased.

[0075] In another embodiment, a corresponding machine learning model or deep learning model can be pre-trained as an activity scoring model. Using motion feature information as input to the model, the activity score determined by the model after inference can be obtained.

[0076] This demonstrates that the activity scoring model can flexibly adapt to different live streaming scenarios and accurately reflect the activity level of each individual.

[0077] Step S3223: Using the highest activity score as the significance criterion, select the region anchor frame with the highest score as the main anchor frame.

[0078] Once the activity score is determined, the highest activity score can be used as the saliency criterion, and the anchor box with the highest score can be selected as the subject anchor box. This criterion is based on the assumption that in a multi-person scenario, the most active person is often the main subject of the live stream. The activity score calculated through the above steps effectively reflects the activity status of the person; therefore, the anchor box with the highest score is considered to be the anchor box of the person whose location best meets the saliency criterion. For example, in a live stream scenario, the host may move frequently during the presentation, with a high frequency of changes in movement speed and direction, while remaining still for a period of time when explaining a key point. These behavioral characteristics make the host's activity score higher than other people, thus accurately identifying them as the main subject.

[0079] The embodiments of this application significantly improve the accuracy and adaptability of subject identification by introducing dynamic behavioral features to determine the main character. First, by continuously tracking the region anchor boxes of each character, motion feature information such as movement speed, frequency of movement direction changes, and dwell time is extracted, comprehensively reflecting the character's dynamic behavior. Second, a preset activity scoring model is applied to calculate the activity score of each character based on these motion feature information. This model can be flexibly designed, such as using weighted summation or machine learning models, to adapt to the needs of different live streaming scenarios. Finally, the region anchor box with the highest activity score is used as the main character anchor box, ensuring accurate identification of the most active main character in multi-person scenarios. This process not only considers the dynamic behavior of the characters but also improves the accuracy and real-time performance of subject identification through flexible scoring models and saliency judgment conditions, making it applicable to various complex live streaming scenarios and providing high-quality main character location information for subsequent image segmentation processing.

[0080] Based on any embodiment of the method in this application, the main anchor frame is tracked and smoothed, including:

[0081] Step S3310: Predict the position information of the main anchor frame based on the Kalman filter;

[0082] For a set B of anchor frames consisting of multiple region anchor frames detected from an image frame, assuming that the distance of each region anchor frame from the center of the image and the size of the region anchor frame are used as saliency criteria, the region anchor frame with the most prominent saliency, i.e., the main anchor frame B, is selected. m Based on this, the anchor box B corresponding to the most salient subject can be obtained using Kalman filtering. m The corresponding prediction results and the results after smoothing.

[0083] The Kalman filter, a recursive filter based on linear dynamic systems theory, is widely used in target tracking. In this application, the Kalman filter predicts the position of the main anchor frame in the next frame by modeling its historical position information.

[0084] Before making predictions, the necessary initial settings must be configured. First, based on the main anchor frame obtained at time t-1, the corresponding state vector x at that time can be calculated. t-1 :

[0085] Where (x, y) are the coordinates of the center point of the main anchor frame, w and h are the width and height of the main anchor frame, respectively, and v x v y v w and v hThese are the velocity components corresponding to the variables mentioned above. These parameters collectively describe the state of the main anchor frame at the current moment.

[0086] The following state transition matrix F is defined to represent the temporal transformation of the state vector:

[0087] Simultaneously, a measurement matrix H is defined to map the state vector to the measurement space:

[0088] In addition, the noise covariance matrix Q = c is defined in advance. Q I8. Measurement noise covariance matrix R = c R I4. The initial state covariance matrix P t-1 =c P *I8, where c Q c R and c P The coefficients corresponding to each matrix can be set to 0.01, 10, 0.1, and I, respectively, for example. n This represents an identity matrix of size n×n. The process noise covariance matrix Q is used to simulate the uncertainty of model prediction, the measurement noise covariance matrix R is used to simulate the uncertainty of measurement, and the state covariance matrix P is used to simulate the uncertainty of predicted state.

[0089] During the prediction phase, the Kalman filter uses the state transition matrix F to transform the state vector x from the previous time step. t-1 Mapped to the state vector x at the next time step t|t-1 The state transition matrix F describes the change pattern of the state vector over time, reflecting the motion trend of the target. Through matrix operations x... t|t-1 =Fx t-1 The Kalman filter can predict the position information of the main anchor frame in the next frame, providing a prior estimate for subsequent measurement updates.

[0090] At the same time, the Kalman filter also needs to predict the state covariance matrix P at the next time step. t|t-1 To quantify the uncertainty of prediction. State covariance matrix P t-1 This represents the uncertainty of the state estimate at the previous time step, while the process noise covariance matrix Q reflects the uncertainty of the model prediction. This can be expressed by formula P. t|t-1 =FP t-1 F T +Q, the Kalman filter can comprehensively consider the uncertainties in the state transition process and the uncertainties in the model prediction, thus obtaining an uncertainty description of the state estimate at the next time step.

[0091] Therefore, this step, which predicts the position information of the main anchor frame using a Kalman filter, not only considers the target's motion trend but also quantifies the prediction uncertainty through the state covariance matrix. This process provides prior information for subsequent measurement updates and is a crucial foundation for achieving stable tracking of the main anchor frame.

[0092] Step S3320: Correct the predicted position information based on the position information of the main anchor frame determined in the current image frame to obtain the predicted position information;

[0093] Correcting the predicted position information within the framework of the Kalman filter hinges on calculating the measurement residual and the Kalman gain. The measurement residual reflects the difference between the actual measured value and the predicted value. Specifically, assuming the actual position information of the subject anchor box in the current image frame is z... t The position information predicted by the Kalman filter is x. t|t-1 Then measure the residual y t It can be represented as:

[0094] y t =z t -Hx t|t-1 ;

[0095] Where H is the measurement matrix, used to map the predicted state vector to the measurement space. Measurement residual y t It describes the deviation between the actual measured value and the predicted value, and is an important basis for correcting the predicted location information.

[0096] Next, calculate the Kalman gain K. t It is a key parameter used to weigh the weights of predicted and measured values ​​in the correction process. The formula for calculating the Kalman gain is:

[0097] K t =P t|t-1 H T HP t|t-1 H T +R) -1 ;

[0098] Among them, P t|t-1 R is the state covariance matrix obtained during the prediction phase, representing the uncertainty of the predicted state; R is the measurement noise covariance matrix, reflecting the uncertainty of the measurement value. Kalman gain K t By comprehensively considering the uncertainties of prediction and measurement, the relative importance of predicted and measured values ​​in the correction process was determined.

[0099] Finally, the predicted location information is corrected using Kalman gain and measurement residuals to obtain the corrected predicted location information x. t|T :

[0100] x t|t =x t|t-1 +K t y t ;

[0101] This correction process combines actual measurements with predicted values, dynamically adjusting their weights in the correction result using Kalman gain to obtain a more accurate target location estimate. The corrected predicted location information x t It not only takes into account the target's motion trend, but also incorporates the actual measurement information of the current frame, thus more accurately reflecting the target's actual position in the current frame.

[0102] Through the above correction process, the Kalman filter can effectively reduce prediction errors and improve the accuracy and stability of target tracking. This process has wide application value in the field of target tracking, especially when processing measurement data with noise and uncertainty, it can significantly improve system performance.

[0103] Step S3330: Update the state parameters of the Kalman filter based on the corrected predicted position information to reflect the latest position of the main anchor frame;

[0104] After obtaining the corrected predicted position information, the state parameters of the Kalman filter can be updated to reflect the latest position of the main anchor frame, so as to incorporate the corrected predicted position information into the filter state, thereby providing more accurate prior information for subsequent prediction and correction.

[0105] In the update phase of the Kalman filter, the state vector x needs to be updated first. t|t This has already been completed in step S3320. Next, the state covariance matrix P needs to be updated. t|t This is to reflect the uncertainty of the corrected state estimate. The update formula for the state covariance matrix is:

[0106] Where I is the identity matrix, K t It is the Kalman gain, H is the measurement matrix, and P is the measurement matrix. t|t-1 This is the state covariance matrix obtained during the prediction phase. This formula updates the state covariance matrix by subtracting the effect of the product of the Kalman gain and the measurement matrix on the prediction covariance matrix. The updated state covariance matrix P t|t It more accurately reflects the uncertainty of the corrected state estimate.

[0107] The purpose of updating the state covariance matrix is ​​to more accurately estimate the uncertainty of the target position in subsequent prediction stages. For example, when the Kalman gain is large, it indicates that the measured value contributes significantly to the correction process, and the updated state covariance matrix will decrease, indicating greater confidence in the corrected state estimate. Conversely, when the Kalman gain is small, it indicates higher reliability of the prediction, and the change in the state covariance matrix is ​​relatively small.

[0108] By updating the state covariance matrix, the Kalman filter can more accurately reflect the uncertainty of the target position in subsequent prediction stages, thereby improving the accuracy and stability of target tracking. This process not only considers the target's motion trend but also incorporates the actual measurement information of the current frame, enabling the Kalman filter to dynamically adjust the estimation accuracy of the target position.

[0109] In practical applications, the updated state parameters (including the state vector and state covariance matrix) are used in the prediction stage of the next frame, thereby achieving continuous tracking of the target position. For example, in a live broadcast scenario, if the broadcaster moves quickly in the frame, the Kalman filter can accurately track the broadcaster's position changes by continuously updating the state parameters, maintaining stable tracking performance even in the presence of noise or occlusion.

[0110] Therefore, by updating the state parameters of the Kalman filter, the filter can dynamically reflect the latest information and uncertainties regarding the target's position, providing a more accurate basis for subsequent predictions and corrections. This process is one of the key steps in achieving target tracking using the Kalman filter, and it is of great significance for improving the accuracy and stability of target tracking.

[0111] Step S3340: In the image frames that have not been detected, use the predicted location information as the position of the main anchor box.

[0112] In this application, an important function of the Kalman filter in target tracking is to ensure that when the subject anchor frame corresponding to the subject is not directly detected in some image frames, or in the case of an embodiment that uses interval frame detection to determine the subject anchor frame, the Kalman filter can still provide stable target position information through prediction to stabilize the position information of the subject anchor frame and thus maintain the continuity of target tracking.

[0113] In practical applications, especially when computational resources are limited or the detection algorithm cannot successfully detect the target in every frame, there may be instances where the main anchor box is not detected in some frames. In such cases, the prediction function of the Kalman filter becomes particularly important. By performing prediction based on the current state vector and state transition matrix in step S3310, and correcting the prediction results and updating the state parameters in steps S3320 and S3330, the Kalman filter can provide a reliable predicted location in frames where the target is not detected.

[0114] Specifically, when a frame is not detected or the detection fails, the Kalman filter uses the previously updated state vector x. t|t and state covariance matrix P t|t Make a prediction to obtain the state vector x at the next time step. t+1|t This predicted location information can be directly used as the main anchor box location for that frame, ensuring the continuity of target tracking. For example, suppose the main anchor box is not detected in a certain frame, the Kalman filter uses the state transition matrix F and the previous state vector x... t|t Make a prediction to obtain the predicted location information:

[0115] x t+1|t =Fx t|t .

[0116] This predicted location information will be used as the main anchor box position for that frame, and will be used for subsequent image segmentation or other processing. In this way, even if the target main anchor box is not detected or cannot be directly detected in some frames, the Kalman filter can still provide a reasonable target location estimate, thereby ensuring the stability and continuity of target tracking.

[0117] Furthermore, this prediction mechanism can effectively reduce target loss caused by detection noise or algorithm instability. For example, in live streaming scenarios, if the streamer moves quickly or is partially occluded, the detection algorithm may fail to accurately detect the main anchor frame in some frames. In this case, the prediction function of the Kalman filter can fill these detection gaps, ensuring that the position information of the main anchor frame remains stable across consecutive frames.

[0118] The above embodiments utilize a Kalman filter for subject anchor box tracking and smoothing, significantly improving the stability and accuracy of target tracking. First, through prediction and correction mechanisms, the Kalman filter can provide reliable predicted positions in image frames where subject anchor boxes are not detected or where subject anchor boxes have not been detected, effectively avoiding missed detections and ensuring the continuity of target tracking. Second, by using interval frame detection and the prediction function of the Kalman filter, the need for detection in every frame is reduced, significantly saving system resources and improving processing efficiency. Furthermore, the lightweight nature and efficient mathematical model of the Kalman filter enable it to be widely adaptable to devices with varying performance, including mobile devices with limited computing resources and low-end PCs, ensuring efficient operation on various devices. These technical advantages not only improve the robustness and real-time performance of subject image segmentation but also provide an efficient and stable technical solution for related application scenarios.

[0119] Based on any embodiment of the method in this application, image segmentation is performed on image frames in the live stream according to the position information of the main anchor frame to obtain a human body image of the main figure, including:

[0120] Step S3410: Based on the position information of the main anchor frame, crop the corresponding image region in the image frame to obtain the portrait area map;

[0121] The anchor frame's position information includes its top-left corner coordinates, width, and height. These parameters uniquely determine the subject's position and extent within the image. Using these parameters, a rectangular region containing the complete image of the subject can be cropped from the original image frame. For example, assuming the anchor frame's position information is top-left corner coordinates (x, y), width w, and height h, a rectangular region of size w*h can be cropped from the original image frame, with its top-left corner located at (x, y).

[0122] Cropping can be implemented using cropping functions in image processing libraries or frameworks. These functions typically accept the coordinates and dimensions of the cropping region as input parameters and output the cropped image region. For example, in the Python image processing library Pillow, the crop() function can be used to perform cropping operations, with the input parameters being the coordinates of the top-left and bottom-right corners of the cropping region.

[0123] The cropped portrait region is a reduced image area, called the portrait region map, which contains only the subject and some background information. This cropped image region will be used as input to the subsequent image segmentation model, thereby reducing the image size processed by the model and improving processing speed. For example, if the original image frame has a resolution of 1920×1080 pixels, and the subject anchor box size is 300×400 pixels, then the resolution of the cropped portrait region map will be 300×400 pixels, significantly reducing the amount of image data.

[0124] Furthermore, the cropping process can be combined with image preprocessing steps, such as scaling and normalization, to further optimize the input to the image segmentation model. For example, if the image segmentation model requires the input image to be a fixed size of 256×256 pixels, the image of the person can be scaled after cropping to meet the model's input requirements.

[0125] Step S3420: Input the cropped human image region into the attention-based image segmentation model, and generate a segmentation mask for the human body image of the main person through the image segmentation model;

[0126] The cropped human image region is input into an attention-based image segmentation model, which generates a segmentation mask for the subject's human image. This mask allows for the accurate extraction of the subject's outline and boundaries from the cropped human image region, providing high-quality segmentation results for subsequent image editing and applications.

[0127] Image segmentation models based on attention mechanisms are deep learning models that enhance the model's focus on target regions by introducing an attention mechanism, thereby improving the accuracy and efficiency of segmentation. The attention mechanism allows the model to automatically focus on the most important parts of the image, i.e., the main subject, while ignoring irrelevant information in the background. For example, the SeaTopFormer architecture can be used, a variant model combining SeaFormer and TopFormer. Through optimized attention design, it improves the model's ability to recognize human contours, especially showing significant segmentation results at human edges and in complex backgrounds.

[0128] In practice, image segmentation models typically require extensive training data to learn the appearance features and shape information of the subject. Training data usually includes images labeled with segmentation masks that clearly define the boundaries of the subject. Through training, the model learns how to distinguish the subject from the background in an input image and generates the corresponding segmentation mask.

[0129] After the cropped image of the person is input into the segmentation model, the model extracts image features and generates a segmentation mask step by step through a series of convolutional layers, attention modules, and decoder structures. For example, the SeaTopFormer model first obtains the global features of the image through the SeaFormer feature extraction module, then optimizes the feature map using the TopFormer attention mechanism, and finally restores the resolution of the segmentation mask through the decoder, outputting a segmentation mask of the same size as the image of the person.

[0130] Step S3430: Extract the human body image and background image of the main person from the image frame according to the segmentation mask;

[0131] The segmentation mask is a binary image with the same resolution as the input image, where pixels in the subject area have a value of 1, and pixels in the background area have a value of 0. By multiplying the segmentation mask pixel-by-pixel with the original image frame, the human image of the subject can be extracted. Specifically, for each pixel in the original image frame, if the corresponding segmentation mask pixel value is 1, the value of that pixel is retained; if the segmentation mask pixel value is 0, the value of that pixel is set to 0. In this way, an image containing only the subject—the human image of the subject—can be obtained.

[0132] Meanwhile, the background image can be extracted by multiplying the inverted segmentation mask (i.e., 1 becomes 0, 0 becomes 1) pixel-by-pixel with the original image frame. This process is also based on the binarization property of the segmentation mask, but the operation is reversed. For each pixel in the original image frame, if the corresponding segmentation mask pixel value is 0, the value of that pixel is retained; if the segmentation mask pixel value is 1, the value of that pixel is set to 0. In this way, an image containing only the background, i.e., the background image, can be obtained.

[0133] For example, assuming the original image frame has a resolution of 1920×1080 pixels, and the segmentation mask is also a binarized image of 1920×1080 pixels, the above operations will yield two images: one containing only the subject (human body), and the other containing only the background. These two images can be used for subsequent image editing processes, such as background blurring, background replacement, or adding special effects.

[0134] In practical applications, the process of extracting the subject and background images can be achieved using pixel-by-pixel manipulation functions in image processing libraries or frameworks. For example, in the Python image processing library Pillow, the Image.point() function can be used to perform pixel-by-pixel operations on an image, combined with a segmentation mask to extract the subject and background images.

[0135] Step S3440: After performing image editing processing on the human body image or background image, the live stream is transmitted to the live streaming room for playback.

[0136] By performing image editing processing on the extracted main character image or background image and transmitting the processed image frame to the live broadcast room for playback, the final application stage of this application can be realized. Image editing technology can enhance the visual effect of live broadcasts and meet the needs of different application scenarios.

[0137] Image editing can include various operations such as background blurring, background replacement, adding effects, and beautification. These operations are based on extracted images of the subject and background, and are implemented using different image processing algorithms. For example, background blurring can be achieved by applying a Gaussian blur algorithm to the background image, making the background blurred and thus highlighting the subject. Background replacement can replace the extracted background image with a preset virtual background, such as a virtual studio scene or a specific background image, thereby providing more creative visual effects for live broadcasts.

[0138] In addition, beautification effects can be applied to the main subject's image, such as skin smoothing, whitening, and face slimming, to enhance the broadcaster's visual appeal. These beautification effects are typically achieved through image processing algorithms, such as using bilateral filtering for skin smoothing or affine transformation for face slimming. These processes not only improve the visual quality of the live stream but also meet the personalized needs of different broadcasters and viewers.

[0139] After image editing is complete, the processed image frames need to be transmitted to the live streaming room for playback. For example, in a live streaming scenario, the main image of the streamer is accurately extracted and the background is blurred. The processed image frames are encoded into an H.264 format video stream and transmitted to the viewer's terminal device through a streaming media server. When watching the live stream, viewers can see the streamer highlighted and the background blurred, thus obtaining a better visual experience.

[0140] The embodiments described above achieve accurate segmentation and efficient editing of the main subject in a live stream through a series of optimized image processing steps. First, by cropping the area corresponding to the anchor frame of the main subject in the image frame, the amount of image data is significantly reduced, improving processing speed. Second, using an attention-based image segmentation model, the contours and boundaries of the main subject are accurately extracted, enhancing the accuracy and robustness of segmentation. Furthermore, by extracting the main subject and background images through segmentation masks, high-quality materials are provided for subsequent image editing. Finally, the main subject or background image is edited (e.g., background blurring, background replacement, beautification), and then efficiently transmitted to the live stream for playback, significantly improving the visual effects and user experience of the live stream. This process not only optimizes the efficiency of image segmentation and editing but also meets the diverse application scenario requirements through flexible image processing functions, is applicable to various devices and network conditions, and has wide applicability and high efficiency.

[0141] Based on any embodiment of the method in this application, the method further includes:

[0142] Step S31: Call the target detection model after pruning and channel compression, and determine the corresponding region anchor box of each person from the image frames of the live stream according to the preset interval. For image frames that have not been detected, the main anchor box is the main anchor box determined after corresponding tracking and smoothing.

[0143] By calling a pruned and channel-compressed object detection model, the corresponding region anchor boxes for each person can be determined from the image frames of the live stream at preset intervals. This can optimize the efficiency of object detection, enabling it to run efficiently on low-performance devices while reducing the consumption of computing resources.

[0144] Specifically, the target detection model in this embodiment is a lightweight model, such as the NanoDet model. NanoDet is a lightweight, single-stage target detection model optimized for mobile devices and low-performance hardware. Through model pruning and channel compression techniques, it significantly reduces the number of parameters and computational complexity while maintaining high detection accuracy. For example, the NanoDet model has only 1.01M parameters and is trained using the SimOTA label dynamic allocation strategy based on optimal transport theory, enabling fast detection on low-performance devices.

[0145] During implementation, the target detection model operates according to a preset interval frame call strategy. This means that it does not perform detection on every single frame of the live stream, but rather every few frames. For example, it can be set to perform detection every 3 or 5 frames, with the specific interval adjusted according to device performance and real-time requirements. This interval frame detection strategy further reduces computational load, while utilizing the Kalman filter algorithm in subsequent steps to predict and track the anchor boxes of the main subject in undetected frames, ensuring the continuity of the main subject tracking.

[0146] For image frames that were not detected, the subject anchor box is determined after tracking and smoothing. This mechanism combines the efficiency of the object detection model with the stability of the Kalman filter algorithm, ensuring stable tracking and accurate segmentation of the subject even on low-performance devices.

[0147] Step S32: Apply the Kalman filter algorithm to track and smooth the main anchor frame;

[0148] Applying the Kalman filter algorithm to track and smooth the subject's anchor frame is a key technique for ensuring stable tracking of the subject on low-performance devices. When the target detection model determines the subject's anchor frame from the image frames of the live stream at preset intervals, the position information of the subject's anchor frame in image frames that have not been detected needs to be predicted and corrected using the Kalman filter algorithm to ensure the continuity and stability of the subject's tracking.

[0149] Referring to the specific algorithm demonstration described above, the Kalman filter algorithm uses the motion information of the target in consecutive frames to predict the position of the subject's anchor frame. In frames where the subject's anchor frame is detected, the Kalman filter updates its internal state based on the detection result. In frames where the subject's anchor frame is not detected, the Kalman filter predicts based on the state and motion model of the previous frame, thus providing stable subject anchor frame position information. This prediction and correction mechanism ensures that the position information of the subject remains continuous and accurate even with large detection intervals.

[0150] For example, assuming the object detection model performs detection every 3 frames, in frames where no object is detected, the Kalman filter predicts the position of the subject's anchor frame in the current frame based on the previously detected subject's anchor frame position and its movement trend. When the subject's anchor frame is detected again, the Kalman filter corrects the predicted position based on the new detection result, thus ensuring that the subject's anchor frame position information always accurately reflects the actual position of the subject.

[0151] By applying the Kalman filter algorithm, stable tracking of the main subject can be achieved efficiently on low-performance devices. This process not only reduces the need for detection in every frame, thus lowering computational resource consumption, but also ensures the continuity and accuracy of the subject tracking through prediction and correction mechanisms. For example, on mobile devices or low-end PCs, this optimization strategy enables the subject image segmentation technology to run stably in applications with high real-time requirements, such as live streaming and video conferencing.

[0152] Step S33: Use an attention-based image segmentation model to segment the image frames in the live stream.

[0153] As revealed earlier, attention-based image segmentation models are a type of deep learning model that enhances the model's focus on the main subject by introducing an attention mechanism, thereby improving segmentation accuracy and efficiency. The attention mechanism allows the model to automatically focus on the most important part of the image, i.e., the main subject, while ignoring irrelevant information in the background. For example, the SeaTopFormer architecture can be used, a variant model combining SeaFormer and TopFormer. Through optimized attention design, it improves the model's ability to recognize human contours, especially showing more significant segmentation results at human edges and against complex backgrounds.

[0154] In practice, image segmentation models typically require extensive training data to learn the appearance features and shape information of the subject. Training data usually includes images labeled with segmentation masks that clearly define the boundaries of the subject. Through training, the model learns how to distinguish the subject from the background in an input image and generates the corresponding segmentation mask.

[0155] After the cropped image of the person is input into the segmentation model, the model extracts image features and generates a segmentation mask step by step through a series of convolutional layers, attention modules, and decoder structures. For example, the SeaTopFormer model first obtains the global features of the image through the SeaFormer feature extraction module, then optimizes the feature map using the TopFormer attention mechanism, and finally restores the resolution of the segmentation mask through the decoder, outputting a segmentation mask of the same size as the image of the person.

[0156] To adapt to the needs of low-performance devices, image segmentation models can be optimized using pruning and channel compression techniques to reduce the number of parameters and computational complexity. For example, the SeaTopFormer model can have its number of parameters significantly reduced after optimization while maintaining high segmentation accuracy. This optimization strategy enables the model to run efficiently on low-performance devices, meeting the needs of applications with high real-time requirements, such as live web streaming and video conferencing.

[0157] The above embodiments employ various optimization techniques to ensure that the subject image segmentation technology can run efficiently on low-performance devices while maintaining accurate recognition and smooth performance. First, a lightweight object detection model combined with an interval frame detection strategy significantly reduces computational load and resource consumption. Simultaneously, the Kalman filter algorithm is used to predict and track undetected frames, ensuring the continuity and stability of subject tracking. Second, an attention-based image segmentation model further optimizes segmentation accuracy, particularly excelling in handling complex backgrounds and subject edges. Finally, model pruning and channel compression techniques reduce the number of parameters and computational complexity of the image segmentation model, enabling it to run quickly on low-performance devices. The combination of these optimization techniques not only guarantees accurate subject recognition but also achieves efficient and smooth operation in real-time demanding applications such as live streaming and video conferencing, demonstrating significant technical advantages.

[0158] Based on any embodiment of the method in this application, before performing image segmentation on the image frames in the live stream using an attention-based image segmentation model, the method includes:

[0159] Step S2100: Obtain the foreground and background real-scene images of people and their corresponding segmentation masks and depth information of people; extract the corresponding foreground and background images of people based on the segmentation masks.

[0160] Foreground and background images refer to real-world scenes captured in actual settings, including both the main subject and background figures. These images can be acquired using professional photography equipment or ordinary cameras, ensuring image quality meets training requirements. A segmentation mask is a binarized image corresponding to these real-world images. Segmentation masks can be manually annotated or automatically generated using existing segmentation algorithms, clearly identifying the boundaries between foreground and background figures.

[0161] Depth information refers to the depth value of each pixel relative to the camera, typically obtained through a depth camera or stereo vision technology. Depth information is crucial for determining the layer order of foreground and background subjects in image composition space, as it reflects the relative distance between them. Pixels with smaller depth values ​​generally represent foreground subjects closer to the camera, while pixels with larger depth values ​​represent background subjects.

[0162] Based on the segmentation mask, foreground and background images of people can be extracted from a real-world image. The extraction process can be implemented pixel-by-pixel. For example, for the foreground image, regions with a pixel value of 1 in the segmentation mask are retained, while the remaining regions are set to transparent or the background color; for the background image, regions with a pixel value of 0 in the segmentation mask are retained. This extraction method ensures the independence of the foreground and background images, providing a foundation for subsequent image compositing.

[0163] Step S2200: Determine the layer order of the foreground and background figures in the image synthesis space based on the depth information of the figures, and synthesize an initial image sample that simultaneously contains foreground and background figures.

[0164] When creating image samples for training the image segmentation model, in a virtual image synthesis space defined by the depth information of the corresponding figures and the image plane, the layer order of the foreground and background figures is determined based on their depth information. According to their relative depth, the foreground figure image is placed as the top layer above the corresponding bottom layer of the background figure image, forming the initial image sample. For example, assuming the depth value range of the foreground figure is 1-5 meters and the depth value range of the background figure is 6-10 meters, then during image synthesis, the foreground figure image will be placed above the background figure image, thus simulating the situation where the foreground figure occludes the background figure in a real scene.

[0165] Step S2300: Based on the step adjustment of the relative positional relationship between the foreground figure and the background figure in the image synthesis space, synthesize multiple derived image samples, so that the initial image sample and each derived image sample reflect the temporal gradual change characteristics.

[0166] Based on the initial image samples, step adjustments can be made according to the relative positional relationship between foreground and background figures in the image synthesis space to simulate the relative positional changes of foreground and background figures at different time steps, thereby synthesizing multiple derived image samples. The resulting derived image samples not only enrich the diversity of the training data, but also provide training samples that are closer to real-world scenes for the image segmentation model, thus accelerating the convergence of the model and improving its generalization ability.

[0167] Specifically, the objects to be adjusted in steps can include various operations such as translation, rotation, and scaling. For example, by translating the foreground person image in the image synthesis space, the positional changes of the person at different time steps can be simulated; by rotating, the orientation of the person can be simulated; and by scaling, the distance between the person and the camera can be simulated. The combination of these operations can generate a series of image samples with visual continuity and gradation, thus more realistically reflecting the dynamic behavior of people in real-world scenes.

[0168] Taking translation as an example, suppose the depth of the foreground figure image at its initial position is 3 meters, and the depth of the background figure image is 7 meters. After synthesizing the initial image sample, the position of the foreground figure image can be gradually adjusted, moving it left or right in the synthesis space while keeping the position of the background figure image unchanged. In this way, multiple derived image samples can be generated, each reflecting the positional changes of the foreground figure at different time steps.

[0169] Similarly, rotation can be achieved by adjusting the orientation angle of the foreground person image. For example, starting from an initial frontal orientation, the foreground person image can be rotated by a certain angle (e.g., 5 degrees each time) to generate a series of derived image samples with different orientations. These samples can simulate the turning motion of a person in a real scene, providing richer training data for the image segmentation model.

[0170] Scaling operations can simulate changes in distance between a person and the camera by changing the size of the foreground image. For example, starting from an initial normal size, the foreground image can be gradually enlarged or reduced to generate a series of derived image samples of different sizes. These samples can simulate the dynamic process of a person moving closer to or further away from the camera in a real-world scene.

[0171] These step-by-step adjustments generate a series of derived image samples that exhibit visual continuity and gradation. These samples not only enrich the diversity of the training data but also provide the image segmentation model with training samples that more closely resemble real-world scenes. The derived image samples generated through translation, rotation, and scaling operations can simulate various dynamic behaviors of people in real-world scenes, thereby improving the adaptability and generalization ability of the image segmentation model to dynamic changes in people.

[0172] Step S2400: Train the image segmentation model using the synthesized image samples, and use the corresponding foreground human images as supervision samples to train the image segmentation model to a convergent state.

[0173] Initial image samples and their derived image samples were used to train the image segmentation model. By using corresponding foreground human images as supervised samples, the model can learn how to accurately segment the main subject from complex backgrounds, maintaining high segmentation accuracy even when the subject's position, orientation, and size change. This training sample synthesis method based on temporal gradual changes not only improves the training efficiency of the image segmentation model but also significantly enhances its robustness and accuracy in practical applications. Once the image segmentation model has been trained to convergence, it can be used to extract human images of the main subject in this application.

[0174] The above embodiments construct image samples with temporally varying characteristics through efficient synthesis for model training, significantly improving the image segmentation model's ability to accurately segment main figures in live streams. First, by acquiring real-world images of foreground and background figures, along with their segmentation masks and depth information, the foreground and background figure images can be accurately extracted. Next, based on the figure's depth information, the layer order is determined, and initial image samples containing foreground and background figures are synthesized to simulate occlusion relationships in real-world scenes. Furthermore, through step-by-step adjustments such as translation, rotation, and scaling, multiple derived image samples are generated. These samples not only enrich the diversity of the training data but also simulate the dynamic behavior of figures, enabling the model to learn the changes in the main figure under different temporal and spatial conditions. Finally, using these synthesized samples for training, the model can converge quickly and possess stronger generalization ability, thus efficiently and accurately segmenting the main figure's human image in live streams, meeting the needs of applications with high real-time requirements.

[0175] Please refer to Figure 3. A subject image segmentation apparatus according to one aspect of this application includes an anchor frame detection module 3100, a subject determination module 3200, a tracking and smoothing module 3300, and a segmentation processing module 3400. The anchor frame detection module 3100 is configured to detect the image region where each person is located in an image frame of a live stream and determine the corresponding region anchor frame for each person's image region. The subject determination module 3200 is configured to determine the corresponding region anchor frame indicating the image region where the subject is located from all region anchor frames according to a preset saliency judgment condition. The tracking and smoothing module 3300 is configured to track and smooth the subject anchor frame to stabilize its position information. The segmentation processing module 3400 is configured to perform image segmentation on the image frames in the live stream based on the position information of the subject anchor frame to obtain a human image of the subject.

[0176] Based on any embodiment of the device in this application, the subject determination module 3200 includes: a distance calculation module, configured to calculate the distance between each region anchor frame and the image center of its image frame; an area calculation module, configured to calculate the area of ​​each region anchor frame; a joint sorting module, configured to sort the region anchor frames according to the distance and area; and a sequence filtering module, configured to select the region anchor frame with the highest sorted position as the subject anchor frame, using the highest sorted position as the salience criterion.

[0177] Based on any embodiment of the device in this application, the subject determination module 3200 includes: an anchor frame tracking module, configured to continuously track the region anchor frame of each person and extract the motion feature information of each person, including the motion speed, the frequency of change of motion direction, and the dwell time in the image of the live stream; an activity scoring module, configured to determine the corresponding activity score of each person by applying a preset activity scoring model based on the motion feature information of each person; and a high score filtering module, configured to select the region anchor frame with the highest activity score as the subject anchor frame, using the highest activity score as a significant criterion.

[0178] Based on any embodiment of the apparatus in this application, the tracking processing module includes: a position prediction module, configured to predict the position information of the main anchor frame based on a Kalman filter; a prediction correction module, configured to correct the predicted position information according to the position information of the main anchor frame determined in the current image frame, to obtain the predicted position information; a parameter update module, configured to update the state parameters of the Kalman filter according to the corrected predicted position information to reflect the latest position of the main anchor frame; and a prediction application module, configured to use the predicted position information as the position of the main anchor frame in image frames that have not been detected.

[0179] Based on any embodiment of the device in this application, the segmentation processing module 3400 includes: a portrait cropping module, configured to crop the corresponding image region in the image frame according to the position information of the subject anchor frame to obtain a portrait region map; a segmentation execution module, configured to input the cropped portrait region map into an image segmentation model based on an attention mechanism, and generate a segmentation mask of the human body image of the subject through the image segmentation model; a separation and extraction module, configured to extract the human body image and background image of the subject from the image frame according to the segmentation mask; and an application processing module, configured to perform image editing processing on the human body image or background image, and then transmit the live stream to the live broadcast room for playback.

[0180] Based on any embodiment of the device in this application, the anchor frame detection module 3100 is further configured to call a pruned and channel-compressed target detection model to determine the region anchor frame corresponding to each person from the image frames of the live stream at preset intervals. For image frames that have not been detected, the main anchor frame is the main anchor frame determined after corresponding tracking and smoothing processing. The tracking and smoothing module 3300 is further configured to apply a Kalman filter algorithm to track and smooth the main anchor frame. The segmentation processing module 3400 is further configured to use an attention-based image segmentation model to perform image segmentation on the image frames in the live stream.

[0181] Based on any embodiment of the device in this application, prior to the segmentation processing module 3400, this device further includes: a material processing module, configured to acquire a foreground real-scene image of a person and a background real-scene image of a person, along with their corresponding segmentation masks and person depth information, and extract corresponding foreground and background person images based on the segmentation masks; a sample synthesis module, configured to determine the layer order of the foreground and background person images in the image synthesis space based on the person depth information, and synthesize an initial image sample that simultaneously contains foreground and background persons; a sample augmentation module, configured to synthesize multiple derived image samples based on a stepwise adjustment of the relative positional relationship between the foreground and background persons in the image synthesis space, so that the initial image sample and each derived image sample exhibit temporal gradual change characteristics; and a model training module, configured to train an image segmentation model using the synthesized image samples, and use the corresponding foreground person images as supervision samples to train the image segmentation model to a convergent state.

[0182] Another embodiment of this application provides a subject image segmentation device. Figure 4 shows a schematic diagram of the internal structure of the subject image segmentation device. This subject image segmentation device includes a processor, a computer-readable storage medium, a memory, and a network interface connected via a system bus. The computer-readable non-volatile storage medium of the subject image segmentation device stores an operating system, a database, and computer-readable instructions. The database may store information sequences, and when the computer-readable instructions are executed by the processor, the processor can implement a subject image segmentation method.

[0183] The processor of the subject image segmentation device provides computational and control capabilities, supporting the operation of the entire subject image segmentation device. The memory of the subject image segmentation device can store computer-readable instructions, which, when executed by the processor, cause the processor to perform the subject image segmentation method of this application. The network interface of the subject image segmentation device is used for communication with a terminal.

[0184] Those skilled in the art will understand that the structure shown in Figure 4 is merely a block diagram of a portion of the structure related to the solution of this application, and does not constitute a limitation on the subject image segmentation device to which the solution of this application is applied. The specific subject image segmentation device may include more or fewer components than shown in the figure, or combine certain components, or have different component arrangements.

[0185] In this embodiment, the processor executes the specific functions of each module in Figure 3, and the memory stores the program code and various types of data required to execute the above modules or sub-modules. The network interface is used to realize data transmission between user terminals or servers. The non-volatile readable storage medium in this embodiment stores the program code and data required to execute all modules in the subject image segmentation device of this application, and the server can call the server's program code and data to execute the functions of all modules.

[0186] This application also provides a non-volatile readable storage medium storing computer-readable instructions, which, when executed by one or more processors, cause the one or more processors to perform the steps of the subject person image segmentation method of any embodiment of this application.

[0187] This application also provides a computer program product, including a computer program / instructions that, when executed by one or more processors, implement the steps of the method described in any embodiment of this application.

[0188] Those skilled in the art will understand that all or part of the processes in the methods of the above embodiments of this application can be implemented by a computer program instructing related hardware. This computer program can be stored in a non-volatile readable storage medium, and when executed, it can include the processes of the embodiments of the above methods. The aforementioned storage medium can be a computer-readable storage medium such as a magnetic disk, optical disk, read-only memory (ROM), or random access memory (RAM).

[0189] In summary, this application significantly improves the efficiency, accuracy, and adaptability of subject image segmentation by integrating multiple innovative technologies. First, by optimizing the saliency judgment criteria, it can accurately identify subjects in complex multi-person scenes, avoiding misjudgments that occur in traditional methods. Second, the introduction of a Kalman filter for tracking and smoothing the subject's anchor box not only improves the stability of the segmentation results but also enhances the tracking effect in consecutive frames, maintaining accurate segmentation even in scenes with fast-moving subjects or complex backgrounds. Furthermore, this application employs a lightweight object detection and image segmentation model, significantly reducing computational resource consumption through pruning and channel compression techniques, enabling efficient operation on low-performance devices and meeting real-time requirements. Simultaneously, the training sample synthesis strategy based on depth information further improves the model's robustness and generalization ability, solving the problem of insufficient training data. The combined application of these technological advantages makes this application not only perform exceptionally well in multi-person scenes but also widely applicable in video conferencing, live streaming, virtual reality, and other fields, providing users with a higher quality visual experience while reducing hardware costs and improving overall performance and practicality.

Claims

1. A method for segmenting a subject image, characterized in that, include: Detect the image region where each person is located in the image frame of the live stream, and determine the region anchor box corresponding to the image region where each person is located. Based on the preset saliency determination criteria, the region anchor frame corresponding to the image region where the main subject is located is determined from all region anchor frames as the main subject anchor frame; The main anchor frame is tracked and smoothed to stabilize its position information; Based on the position information of the main anchor frame, image frames in the live stream are segmented to obtain the human body image of the main figure.

2. The method for segmenting a main character image according to claim 1, characterized in that, Based on preset saliency criteria, the region anchor frame corresponding to the image region where the main subject is located is determined from all region anchor frames as the main subject anchor frame, including: Calculate the distance between the anchor box of each region and the image center of its corresponding image frame; Calculate the area of ​​the anchor frame for each region; The region anchor frames are sorted according to the distance and area; Using the highest ranking as the significance criterion, the anchor frame of the region ranked highest is selected as the main anchor frame.

3. The method for segmenting a main character image according to claim 1, characterized in that, Based on preset saliency criteria, the region anchor frame corresponding to the image region where the main subject is located is determined from all region anchor frames as the main subject anchor frame, including: The region anchor box of each person is continuously tracked to extract the motion feature information of each person, including the motion speed, the frequency of change of motion direction, and the dwell time in the image of the live stream. Based on the motion characteristics of each person, a preset activity rating model is applied to determine the corresponding activity rating for each person. Using the highest activity score as the significance criterion, the region with the highest score is selected as the main anchor frame.

4. The method for segmenting a main character image according to claim 1, characterized in that, The tracking and smoothing process for the main anchor frame includes: Predicting the position information of the main anchor frame based on Kalman filter; The predicted position information is corrected based on the position information of the main anchor frame determined in the current image frame to obtain the predicted position information; Based on the corrected predicted position information, update the state parameters of the Kalman filter to reflect the latest position of the main anchor frame; In image frames that have not been detected, the predicted location information is used as the position of the main anchor box.

5. The method for segmenting a main character image according to claim 1, characterized in that, Based on the position information of the main anchor frame, image frames in the live stream are segmented to obtain the human body image of the main figure, including: Based on the position information of the main anchor frame, the corresponding image region in the image frame is cropped to obtain the human portrait area map; The cropped human image region is input into an attention-based image segmentation model, which generates a segmentation mask for the human body image of the main figure. After image editing processing is performed on the human body image or background image, the live stream is transmitted to the live streaming room for playback; Extract the human body image and background image of the main figure from the image frame based on the segmentation mask.

6. The method for segmenting a main character image according to any one of claims 1 to 5, characterized in that, The method further includes: The target detection model with pruned and channel compression is invoked to determine the corresponding region anchor box for each person from the image frames of the live stream at preset intervals. For image frames that are not detected, the main anchor box is the main anchor box determined after corresponding tracking and smoothing. An attention-based image segmentation model is used to segment image frames in the live stream. The Kalman filter algorithm is applied to track and smooth the main anchor frame.

7. The method for segmenting a main character image according to claim 6, characterized in that, Before performing image segmentation on the image frames in the live stream using an attention-based image segmentation model, the following steps are included: Acquire real-world images of foreground and background figures, along with their corresponding segmentation masks and figure depth information. Extract the corresponding foreground and background figures based on the segmentation masks. Based on the depth information of the figures, the layer order of the foreground figure image and the background figure image in the image synthesis space is determined, and an initial image sample containing both foreground and background figures is synthesized. Based on the step-by-step adjustment of the relative positional relationship between the foreground and background figures in the image synthesis space, multiple derived image samples are synthesized, so that the initial image sample and each derived image sample exhibit temporal gradation characteristics. The image segmentation model is trained using synthetic image samples, and corresponding foreground human images are used as supervision samples to train the image segmentation model to a convergent state.

8. A device for segmenting a main human figure image, characterized in that, include: The anchor frame detection module is set to detect the image region where each person is located in the image frame of the live stream and determine the region anchor frame corresponding to the image region where each person is located. The subject determination module is set to determine the corresponding region anchor frame indicating the image area where the subject person is located from all region anchor frames based on preset saliency judgment conditions; The tracking and smoothing module is configured to track and smooth the main anchor frame in order to stabilize the position information of the main anchor frame; The segmentation processing module is configured to perform image segmentation on the image frames in the live stream based on the position information of the main anchor frame to obtain the human body image of the main figure.

9. A subject-based image segmentation device, comprising a central processing unit and a memory, characterized in that, The central processing unit is used to invoke and run a computer program stored in the memory to perform the steps of the method as described in any one of claims 1 to 7.

10. A non-volatile readable storage medium, characterized in that, It stores, in the form of computer-readable instructions, a computer program implemented according to any one of claims 1 to 7, which, when invoked by a computer, executes the steps included in the corresponding method.

Images