Method and apparatus for multi-person identity recognition in live stream, and device and medium

Abstract

A method and apparatus for multi-person identity recognition in a live stream, and a device and a medium, which relate to the field of network live-streaming. The method comprises: determining data distances between each facial image in a live stream of a live-streaming room and a plurality of facial templates corresponding to a plurality of live streamers in a preset live streamer library corresponding to the live-streaming room, so as to obtain a distance matrix; on the basis of the distance matrix, performing solving to obtain an optimal matching between the facial images and the facial templates, and using an optimal solution to determine an identity identifier of a live streamer identity to which the facial template that uniquely matches each facial image belongs; and on the basis of mapping relationship data between position information of each facial image in a video frame and the identity identifier of the facial template that matches the facial image, responding to an operation instruction acting on a target live streamer identity. In the present application, a live streamer identity is determined by means of searching for an optimal matching, thereby significantly improving the accuracy of multi-person identity recognition in a live stream, and also improving the user experience.

Description

Methods, devices, equipment, and media for multi-person identity recognition in live streams

[0001] This application claims priority to Chinese Patent Application No. 202510107713.6, filed on January 23, 2025, entitled "Method, Apparatus, Device and Medium for Multi-Person Identity Recognition in Live Stream", the entire contents of which are incorporated herein by reference. Technical Field

[0002] This application relates to live streaming technology, and more particularly to a method, apparatus, device, and medium for identifying multiple individuals in a live stream. Background Technology

[0003] With the development of internet technology and the rise of social media, live streaming has become an indispensable part of people's daily lives. In the live streaming field, especially in scenarios where multiple streamers simultaneously broadcast through the same camera, users not only expect to watch the live content but also hope to have deeper interactions with any streamer. To achieve these functions, it is necessary to be able to identify each streamer in the live video in order to correctly respond to user commands directed at the streamers.

[0004] However, existing facial recognition technologies face numerous challenges in live video scenarios. First, the dynamic nature of live videos often results in issues such as profile views, head tilts, obstructions, exaggerated expressions, and motion blur, making it difficult for traditional facial recognition algorithms to accurately identify and match faces. Second, the multi-person scenes in live streams increase the complexity of recognition, as the faces of different streamers may overlap or intersect in the frame, making it difficult for recognition algorithms to distinguish and match. Furthermore, different streamers may require different beauty and makeup parameters, and traditional technologies cannot accurately apply these parameters to the correct streamer's face.

[0005] Traditional facial recognition technologies typically require high-quality facial images, such as clear, unobstructed, frontal photos. However, in live streaming environments, obtaining facial images that meet these criteria is difficult due to the aforementioned reasons. Consequently, existing algorithms often fail to achieve accurate facial recognition when processing low-quality images, leading to issues such as mismatched or duplicated identity markers. These problems not only impact the interactive experience of live streaming but also limit the design and innovation of live streaming features.

[0006] Traditional techniques for handling multi-person live streams often employ a simple nearest-neighbor matching method. This involves calculating the similarity between each detected face and all feature vectors in a template library, then selecting the template with the highest similarity as the matching result. This method is prone to errors when the face image quality is low because it doesn't consider the overall optimal matching, but only maximizes the value of a single matching result. This can lead to multiple different faces being incorrectly matched to the same template, or multiple expressions of the same person being matched to different templates, thus failing to achieve accurate one-to-one correspondence recognition.

[0007] In summary, traditional technologies suffer from low accuracy, inability to handle low-quality facial images, and inability to achieve globally optimal matching in multi-person identity recognition within live streams. These problems severely limit the improvement of the live streaming interactive experience and the further development of live streaming technology. Therefore, it is necessary to develop a new technical solution to address the problems existing in traditional technologies. Summary of the Invention

[0008] The purpose of this application is to solve the above-mentioned problems by providing a method for identifying multiple people in a live stream, as well as corresponding devices, equipment, and non-volatile readable storage media.

[0009] According to one aspect of this application, a method for multi-person identity recognition in a live stream is provided, comprising the following steps: determining the data distance between each face image in the live stream and multiple face templates corresponding to multiple anchors in a preset live streamer database corresponding to the live stream, thereby obtaining a distance matrix; solving for the optimal match between the face image and the face template based on the distance matrix, such that the sum of the data distances between the face image and the face template is the optimal solution, thereby determining the identity identifier of the anchor to which the uniquely matched face template of each face image belongs; and responding to operation commands applied to the target anchor identity based on the mapping relationship data between the position information of each face image in the video frame and the identity identifier of the face template that matches the face image.

[0010] According to another aspect of this application, a multi-person identity recognition device in a live stream is provided, comprising: a similarity calculation module, configured to determine the data distance between each face image in the live stream and multiple face templates corresponding to multiple anchors in a preset live stream host database corresponding to the live stream, to obtain a distance matrix; a face matching module, configured to solve for the optimal match between the face image and the face template based on the distance matrix, so that the sum of the data distances between the face image and the face template is the optimal solution, to determine the identity identifier of the anchor to which the uniquely matched face template of each face image belongs; and a mapping response module, configured to respond to operation commands acting on the identity of the target anchor based on the mapping relationship data between the position information of each face image in the video frame and the identity identifier of the face template matching the face image.

[0011] According to another aspect of this application, a device for identifying multiple people in a live stream is provided, including a central processing unit and a memory, wherein the central processing unit is used to invoke and run a computer program stored in the memory to perform the steps of the method for identifying multiple people in a live stream described in this application.

[0012] According to another aspect of this application, a non-volatile readable storage medium is provided, which stores a computer program implemented according to the method for identifying multiple people in a live stream in the form of computer-readable instructions, wherein the computer program, when invoked by a computer, executes the steps included in the method.

[0013] This application effectively addresses the challenges in existing technologies and achieves significant beneficial effects: by constructing a distance matrix and solving for optimal matching to process facial images in live videos, it improves recognition accuracy, can handle various complex scenarios, and reduces mismatches and duplications of anchor identities; at the same time, it achieves globally optimal matching based on the distance matrix, ensuring the stability and reliability of the matching results; in addition, by mapping relational data, it can accurately respond to user operation commands for specific anchors, such as sending virtual gifts or applying beauty effects, greatly enhancing the user's interactive experience. Attached Figure Description

[0014] Figure 1 shows an exemplary network architecture suitable for applying the multi-person identity recognition method in a live stream according to this application;

[0015] Figure 2 is a flowchart illustrating an embodiment of the multi-person identity recognition method in a live stream according to this application;

[0016] Figure 3 is a schematic diagram of the multi-person identity recognition device in the live stream of this application;

[0017] Figure 4 is a schematic diagram of the structure of a multi-person identity recognition device used in this application for live streaming. Detailed Implementation

[0018] Before detailing the specific embodiments of the technical solution of this application, we will first disclose the network architecture and application scenarios suitable for supporting the implementation of the technical solution of this application.

[0019] The technical solution of this application is applicable to the field of online live streaming, and is particularly suitable for scenarios where it is necessary to identify the identities of multiple people in a live stream. In this context, the technical solution of this application can be applied in a typical online live streaming platform, as shown in Figure 1. This platform consists of a live streaming server 83, a media server 85, terminal devices 90 for broadcasters, and terminal devices 92 for viewers.

[0020] The live streaming server 83 is responsible for managing the creation of live streaming rooms and the reception and distribution of live streams. As the central node of the live stream, it receives the live stream from the broadcaster's user terminal device 90 and distributes it to the media server 85 and the viewer's user terminal device 92. The live streaming server 83 is also responsible for handling live streaming room management tasks, such as user authentication and live streaming room settings.

[0021] Media server 85 is responsible for encoding, transcoding, and storing the live stream. It performs necessary transcoding based on the viewer's network conditions and device capabilities to ensure the live stream is transmitted to the viewer in a suitable format and quality. Media server 85 is also responsible for storing the live stream for playback or other subsequent processing.

[0022] The broadcaster's terminal device 90 is the source of the live broadcast content. The broadcaster captures video through the terminal device 90 and sends the raw video stream to the live broadcast server 83. The broadcaster's terminal device 90 can be a smartphone, tablet, laptop, or professional camera equipment, which is connected to the live broadcast server 83 via the Internet.

[0023] The viewer's terminal device 92 is the receiving end of the live broadcast content. Viewers watch the live broadcast through terminal devices 92, which can be smartphones, tablets, personal computers, or smart TVs, etc. The viewer's terminal device 92 connects to the live broadcast server 83 via the Internet to receive and play the live broadcast stream.

[0024] The live stream transmission process is as follows: The broadcaster's terminal device 90 captures video and notifies the live streaming server 83. The live streaming server 83 distributes the live stream to the media server 85. The media server 85 encodes and transcodes the live stream, then sends the processed live stream's network address back to the live streaming server 83. The live streaming server 83 then transmits the live stream's network address to the viewer's terminal device 92, so that the terminal device 92 can load and play the live stream based on the network address.

[0025] The method for identifying multiple individuals in a live stream according to this application can be programmed into a computer program product. This computer program product can be selectively deployed on the terminal device 90 of the broadcaster user, the media server 85, or the terminal device 92 of the viewer user. The program product includes the algorithms and logic required to implement the technical solution of this application, and is capable of identifying the identities of multiple broadcasters in the video frames of the live stream, so as to carry out various subsequent applications, including but not limited to beautification processing corresponding to beauty filters and makeup, and sending virtual gifts to specific broadcasters.

[0026] Please refer to Figure 2. According to the method for multi-person identification in a live stream provided in this application, it can be implemented as a computer program product, installed and run on various node devices of a network live streaming platform, such as the terminal devices of broadcasters, media servers, or the terminal devices of viewers, to be responsible for implementing the identification processing of multiple broadcasters in video frames within the live stream. In some embodiments of this method, the following steps are included:

[0027] Step S3100: Determine the data distance between each face image in the live stream of the live room and multiple face templates corresponding to multiple anchors in the preset live streamer database corresponding to the live room, and obtain the distance matrix;

[0028] When a live streamer invites multiple other live streamers to their location for a multi-person live broadcast, the live video feed is captured by the camera on the streamer's device, forming a live stream that is then transmitted to the live streaming room for simultaneous broadcasting. Viewers' devices receive this live stream, decode it, and play it, allowing them to see the live stream displayed in a graphical user interface, thus recognizing each streamer. These streamers, as users of the online live streaming platform, each possess their own user ID, which serves as their identifier. At the computer program level, these identifiers also allow for recognition of each streamer.

[0029] To accurately identify each streamer in a live stream, the streamer needs to prepare a streamer database in advance. This database stores facial images of each streamer participating in the live stream, which serve as facial templates for subsequent identity matching and recognition. In one embodiment, the facial templates can be directly uploaded to the streamer database by the streamer hosting the multi-person live stream. In another embodiment, each streamer participating in the multi-person live stream can provide their facial image as a facial template, linked to their account. When the streamer hosting the multi-person live stream configures their streamer database, they can select one or more streamers participating in the live stream from their list of followed friends. In response to the selection event of the current streamer, the facial template of the selected streamer is retrieved from the corresponding streamer's account and stored in the streamer database based on the selected streamer's identity identifier, which is convenient and efficient.

[0030] For each face template in the live stream personality database, a pre-trained feature extraction model can be used to extract image feature information of the face template, thereby achieving feature representation of each face template. This model can identify and extract key semantic features in face images and represent them as high-dimensional vectors, such as 512-dimensional or 1024-dimensional vectors. These high-dimensional vectors contain the feature information of the face image and are used for subsequent distance calculation and matching. The identity identifier of the anchor and its corresponding high-dimensional vector of the face template are stored together with the face template in the live stream personality database, providing a basis for calculating the data distance between face images in the live stream and preset face templates.

[0031] In practical implementation, the feature extraction model can be trained using a deep learning-based convolutional neural network (CNN) for classification to learn the ability to extract effective high-dimensional feature vectors from facial images. These feature vectors not only contain geometric information of the face but also detailed information such as texture and expression, making each facial template uniquely identifiable. In this way, the live stream person database can provide an accurate reference for each facial image in the live stream, enabling real-time identity recognition and matching during the live broadcast.

[0032] During live streaming, the video frames contain images of multiple presenters. The first step is to extract the facial images of each presenter from these consecutive video frames. Various mature human detection technologies can be used to first identify the human body images, and then face detection technology can be applied to identify the facial images within them, ultimately locating and segmenting the facial regions from the video frames. Specifically, the terminal device or media server using the method of this application receives the live stream in real time and processes each frame using preset human and face detection algorithms to identify and extract the faces, forming a real-time face set. These detected facial images are then compared with facial templates in a live stream participant database.

[0033] The extracted face images then need to be converted into a computationally usable format, namely high-dimensional vectors. This conversion process is accomplished using the feature extraction model revealed earlier, which can extract key semantic features from face images and encode them into vectors in a high-dimensional space, i.e., high-dimensional vectors. These high-dimensional vectors capture key facial features, such as the positions of the eyes, nose, and mouth, as well as facial expressions and contours.

[0034] After obtaining the high-dimensional vector representations of the face images, the next step is to calculate the data distance between these vectors and each face template in the live stream person database. Common methods for calculating data distance include cosine similarity, Euclidean distance, Minkowski distance, Jaccard similarity coefficient, Manhattan distance, Chebyshev distance, and Pearson correlation coefficient. Cosine similarity assesses the similarity between two vectors by measuring the angle between them, while Euclidean distance measures the straight-line distance between two points in Euclidean space. These calculations form a distance matrix, where each row represents a detected face image, each column represents a face template from the live stream person database, and each element in the matrix is ​​the distance value between the corresponding face image and the face template.

[0035] In the above data distance algorithms, the direction in which the distance values ​​represent the degree of similarity differs. Larger values ​​for cosine similarity, Jaccard similarity coefficient, Pearson correlation coefficient, and Spearman rank correlation coefficient indicate greater similarity between the face image and the face template; conversely, smaller values ​​for Euclidean distance, Manhattan distance, and Chebyshev distance indicate greater similarity. In practical applications, this representation can be converted as needed. For example, simply inverting the Euclidean distance value by adding a negative sign converts it to a value where a larger value indicates greater similarity between the face image and the face template.

[0036] Step S3200: Based on the distance matrix, solve for the optimal match between the face image and the face template, so that the sum of the data distances between the face image and the face template is the optimal solution, so as to determine the identity identifier of the anchor person to which the uniquely matched face template of each face image belongs;

[0037] In the process of multi-person identity recognition in a live stream, it is essential to ensure that each facial image identified from the video frame matches a unique facial template, and that the sum of the data distances of all matching pairs reaches the optimal solution. The definition of the optimal solution depends on the direction in which the sum of data distances is represented: if the larger the distance value indicates higher similarity, then the optimal solution is the one with the largest sum of data distances; if the smaller the distance value indicates higher similarity, then the optimal solution is the one with the smallest sum of data distances.

[0038] To achieve optimal matching, various algorithms can be employed, one of which is the minimum matching algorithm, capable of finding the optimal solution in multiple-choice problems. This algorithm constructs a cost matrix and then finds the minimum-cost matching scheme through row and column operations. In this application, the cost matrix is ​​a distance matrix composed of the data distances between the face image and the face template. Since this algorithm seeks the optimal match, when the direction represented by the distance values ​​in the distance matrix indicates that a larger distance value represents a higher similarity, it should be transformed, for example, by adding a negative sign to invert it, so that a smaller distance value represents a higher similarity. The algorithm iteratively updates the matching state by finding augmenting paths until the optimal match is found.

[0039] Another algorithm is the linear programming method, which solves the optimal matching problem by constructing a linear objective function and a set of linear constraints. The objective function aims to minimize or maximize the sum of data distances, while the constraints ensure that each face image matches only one face template. Linear programming problems can be solved using algorithms such as the simplex method or the interior point method.

[0040] Greedy algorithms are also a viable approach, taking the optimal choice at each step based on the current state, with the aim of leading to a globally optimal solution. In this application, a greedy algorithm can be used to select the minimum data distance for matching in each round until all face images have been matched.

[0041] In addition to the algorithms mentioned above, deep learning-based matching algorithms can also be used. These algorithms train a deep neural network to learn the complex matching relationships between face images and face templates, thereby achieving optimal matching. Deep learning models can automatically extract features and perform matching, reducing the need for manual feature engineering.

[0042] In practical applications, the choice of algorithm depends on the nature and scale of the specific problem, as well as the available computational resources. For example, for smaller problems, algorithms for finding the minimum matching and linear programming methods may be more suitable because they can guarantee finding the globally optimal solution. For large-scale problems, greedy algorithms or deep learning-based algorithms may be more practical because they are more computationally efficient. By applying these algorithms, optimal matching for multi-person identity recognition in live streams can be achieved, improving the accuracy and efficiency of the recognition process.

[0043] After determining the matching relationship between the face image and the face template, since the face template has been associated with the identity of the anchor to which it belongs, the relationship between the face image and the identity of the anchor to which it belongs is also determined.

[0044] Step S3300: Based on the mapping relationship data between the position information of each face image in the video frame and the identity identifier of the face template that matches the face image, respond to the operation command that acts on the identity of the target anchor.

[0045] Once the matching relationship between each facial image and its corresponding facial template is determined, a mapping relationship can be established between the positional information of the facial image in the video frame and its corresponding identity identifier, forming a mapping table. This mapping table is key to achieving precise operation command response; it allows for the execution of various operation commands for a specific anchor based on the position and identity identifier of the facial image.

[0046] For example, in one embodiment, when a viewer clicks on the face image of a broadcaster in the graphical user interface, the identity of the broadcaster corresponding to the clicked face image can be quickly determined through a mapping table based on the viewer's click location, thereby executing the operation that the viewer wants to perform, such as sending virtual gifts or initiating interaction, so that the user can have a convenient and fast experience.

[0047] In another embodiment, the viewer or broadcaster directly specifies the broadcaster's identity via operation commands. In this case, this identity can be directly used to query a mapping table to determine the location information of the corresponding face image. Based on this location information, appropriate image processing effects, such as beauty filters or dynamic stickers, are then composited into the broadcaster's face image within the video frame. The advantage of this method is that it does not rely on user clicks on the face image; instead, it directly associates the identity with a specific broadcaster, making the operation more flexible and direct.

[0048] In another embodiment, a viewer clicks on a streamer's face in the video. The system queries a mapping table based on the click location to determine the specified face and its identifier. The viewer then enters a comment to trigger an action command. In response to this command, the comment text is composited as a bullet screen around the streamer's face image, with corresponding directional markers added. This feature not only enhances the interactivity of the live stream but also provides viewers with a more personalized viewing experience, making the bullet screen's direction clearer.

[0049] The implementation of these operational instructions relies on precise identity recognition and the establishment of mapping relationship data. This ensures that operational instructions are accurately applied to the target broadcaster, whether through image clicks or identity markers. This precise response mechanism enhances the interactivity and audience engagement of live streams, while also providing live streaming platforms with more business opportunities, such as targeted advertising and personalized content recommendations associated with specific broadcasters.

[0050] Through the application of the above embodiments, the online live streaming platform can accurately identify multiple anchors in the live video, thereby significantly improving the interactivity and user experience of the live stream. Its technical advantages include, but are not limited to:

[0051] First, by building a live-stream participant database and employing real-time face detection technology, it was ensured that each person's face image could be quickly and accurately detected and extracted in multi-person live-stream scenarios. This process not only improved recognition efficiency but also provided accurate foundational data for subsequent identity matching.

[0052] Secondly, by utilizing facial feature vectors extracted using deep learning models, each facial image can be uniquely identified, significantly enhancing recognition accuracy. Compared to traditional facial recognition technologies, this method can handle low-quality facial images, such as those with profile views, heads tilted down, or occlusions, which is particularly important in dynamic live streaming environments. In this way, even if the anchor's position and expression change during the live stream, high recognition accuracy can be maintained.

[0053] Furthermore, by solving the optimal matching problem based on the distance matrix between face images and face templates, the best match between face images and face templates is achieved, optimizing the total data distance and thus ensuring that each face image can be matched with the correct face template. This globally optimized matching strategy avoids problems caused by local optima, such as identity mismatch or duplicate matching, and improves the stability and reliability of the matching.

[0054] Furthermore, by establishing a mapping between facial image location information and identity identifiers, various operational commands targeting specific livestreamers can be responded to. Whether it's a viewer's click or an instruction input based on their identity identifier, the system can quickly and accurately locate the target livestreamer and execute the corresponding action, such as sending virtual gifts, applying beauty effects, or displaying livestream comments. This response mechanism not only enhances viewer engagement but also creates more interactive opportunities and commercial value for the livestreaming platform.

[0055] Based on any embodiment of the method in this application, the data distance between each face image in the live stream of the live room and multiple face templates corresponding to multiple anchors in the preset live streamer database corresponding to the live room is determined to obtain a distance matrix, including:

[0056] Step S3111: Extract video frames from the live stream transmitted in real time in the live room, perform face tracking and detection on the video frames to obtain multiple face images and form a real-time face set.

[0057] During the real-time transmission of a live stream, the stream consists of continuous video signals captured and transmitted by the camera of the broadcaster's terminal device. To address this, video frame images can be extracted frame by frame from the live stream, and face tracking and detection can be performed on each frame. The aim is to identify and locate the facial regions of all broadcasters appearing in the frame, extract the images, and thus obtain the corresponding face images to form a real-time face set.

[0058] Depending on the device deployed using the method described in this application, the method for acquiring video frames from a live stream will vary accordingly. For example, for the broadcaster's terminal device, video frames can be directly acquired from the image data captured by the camera. However, at the media server and the viewer's terminal device, since they first receive and decode the live stream, the corresponding video frames can be extracted from the decoded image data.

[0059] Face tracking and detection technology typically relies on advanced computer vision algorithms that can accurately identify the location and features of faces within video frames. In one embodiment, a pre-trained face detection model can be used. This model can handle various complex situations, including changes in facial angle, expression, and lighting conditions. By applying the face detection model, multiple face images can be extracted from each video frame, and these images can be aggregated to form a real-time face set.

[0060] In practical implementation, various face detection techniques can be employed, such as cascaded classifiers based on Haar features, multi-task cascaded convolutional neural networks (MT-CNN) based on deep learning, or single-shot detector enhancement (SSD). Each method has its advantages; for example, cascaded classifiers offer faster processing speeds, while deep learning methods offer greater accuracy. The choice of appropriate method depends on the specific application scenario and performance requirements.

[0061] Step S3112: Calculate the data distance between the semantic features of each face image in the real-time face set and each face template in the live broadcast character database, and represent the data distance relationship between each face image and each face template as a distance matrix;

[0062] Once the real-time face set is obtained, the data distance between these real-time face images and the preset face templates in the live stream personality database can be calculated. To perform this calculation, the feature vector of each face image in the real-time face set needs to be extracted first. These feature vectors are high-dimensional representations automatically learned from face images using deep learning models, such as convolutional neural networks (CNNs). These vectors can capture key facial features, including geometric structure, texture, and expression. Similarly, each face template in the live stream personality database has also been converted into a corresponding feature vector and stored in association with the identity identifier of the broadcaster.

[0063] Various metrics can be used to calculate data distance, such as cosine similarity, Euclidean distance, Manhattan distance, and Chebyshev distance. Cosine similarity measures the similarity between two vectors by calculating the angle between them, suitable for scenarios where direction is more important than magnitude. Euclidean distance measures the straight-line distance between two points in Euclidean space, suitable for scenarios where absolute differences are important. Manhattan distance and Chebyshev distance are also commonly used metrics, calculating the walking distance and the difference in maximum dimension between two points in a city block model, respectively.

[0064] In practice, one or more measurement methods can be chosen to calculate the distance between a real-time face image and each face template. For example, if the orientation information of the face image is of interest, cosine similarity can be chosen; if the absolute difference between the face images is of interest, Euclidean distance can be chosen. These distance values ​​will be organized into a distance matrix, where each row represents a real-time face image, each column represents a preset face template, and each element in the matrix represents the distance value between the corresponding face image and the face template.

[0065] The construction of the distance matrix provides the foundation for subsequent optimal matching solutions. By analyzing this matrix, it is possible to determine which facial images are most similar to specific facial templates, thereby achieving accurate identification of the broadcaster. This process not only improves the accuracy of recognition but also provides technical support for subsequent interactive operations, such as the giving of virtual gifts and the application of beauty and makeup effects.

[0066] Step S3113: Detect whether the number of rows and columns of the distance matrix are the same. If they are different, add missing rows or columns and assign zero values ​​to each element to update the distance matrix.

[0067] The number of rows in the distance matrix represents the number of face images in the real-time face set, and the number of columns represents the number of face templates in the live-stream person database. For efficient matching calculations, in this embodiment, the distance matrix is ​​normalized to a square matrix, meaning the number of rows and columns must be the same. If the number of rows and columns is inconsistent, the matrix needs to be padded to ensure that the number of rows and columns are equal.

[0068] Specifically, if the number of face images in the real-time face set exceeds the number of face templates in the live-streaming person library, or vice versa, the distance matrix needs to be adjusted. This can be achieved by adding missing rows or columns and setting each element in these new rows or columns to zero. For example, if the distance matrix originally had 5 rows and 6 columns, meaning there are only 5 face images in the real-time face set but 6 face templates in the live-streaming person library, then a row needs to be added to the matrix, and all elements in that row need to be set to zero, making the matrix 6 rows and 6 columns. Similarly, if the real-time face set has one more face image than the live-streaming person library, corresponding columns need to be added to the matrix, and all elements in these columns need to be set to zero.

[0069] This augmentation operation ensures that the distance matrix is ​​suitable for the subsequent optimal matching algorithm, which requires the distance matrix to be a square matrix. In this way, effective matching calculations can be performed even when the number of face images and face templates is inconsistent. The augmented matrix provides the basis for distance value adjustment and matching in subsequent steps, enabling the algorithm to find the optimal match and thus determine the identity of the anchor corresponding to each face image.

[0070] It should be noted that since the distance values ​​in the distance matrix, i.e., the numerical values ​​of its elements, represent the direction of similarity between the face image and the face template in different ways depending on the data distance algorithm applied. To facilitate the implementation of the matching algorithm in this embodiment, the distance values ​​in this distance matrix can be converted into a numerical form where smaller distance values ​​indicate greater similarity between the face image and the face template. Therefore, when a larger distance value indicates greater similarity between the face image and the face template, a negative sign can be added to the distance value to invert it. In this case, the optimal matching algorithm in this embodiment can be implemented using the minimum matching algorithm.

[0071] This embodiment ensures that the distance matrix has the same number of rows and columns, providing a standardized working environment for subsequent optimal matching algorithms, thus bringing significant technical advantages.

[0072] First, this method solves the problem of inconsistency between the number of face images and the number of face templates by supplementing missing rows or columns and assigning zero values, enabling the algorithm to perform matching calculations on a unified basis, thus improving the adaptability and flexibility of the matching process.

[0073] Secondly, the normalized distance matrix provides a clear framework for the algorithm, enabling it to accurately find the optimal match between each face image and the corresponding face template even in complex and ever-changing live streaming environments, thereby improving the accuracy and efficiency of recognition.

[0074] Furthermore, by converting the distance matrix into a form where smaller distance values ​​indicate higher similarity, this embodiment simplifies the implementation of the matching algorithm, allowing it to focus more on finding a solution that minimizes the total distance. This conversion not only improves the algorithm's execution efficiency but also makes it easier to implement and maintain. Ultimately, this method enables live streaming platforms to quickly respond to user commands during live broadcasts, such as sending virtual gifts or applying beauty effects, greatly enhancing the user's interactive experience and satisfaction. In this way, this embodiment not only improves the accuracy and efficiency of multi-person identity recognition in live streams but also enhances the interactivity and watchability of live streaming platforms, creating more business opportunities and user value for them.

[0075] Based on any embodiment of the method in this application, the data distance between each face image in the live stream of the live room and multiple face templates corresponding to multiple anchors in the preset live streamer database corresponding to the live room is determined to obtain a distance matrix, including:

[0076] Step S3131: Extract video frames from the live stream transmitted in real time in the live room, perform face tracking and detection on the video frames to obtain multiple face images and form a real-time face set.

[0077] The implementation of this step is the same as step S3111, and will not be repeated here.

[0078] Step S3132: Calculate the data distance between the semantic features of each face image in the real-time face set and each face template in the live broadcast character database, and represent the data distance relationship between each face image and each face template as a distance matrix;

[0079] The implementation of this step is the same as step S3112, and will not be repeated here.

[0080] Step S3133: Calculate the row and column with the larger number of elements in the distance matrix. For each row / column with the larger number of elements, calculate the mean distance value corresponding to the element values ​​of each column / row. Delete the rows / columns with the lowest similarity value represented by the mean distance value to make the number of rows and columns in the distance matrix consistent.

[0081] In live streams, the number of people in the real-time face set may differ from the number of face templates in the live stream's person database for various reasons. To address this issue, the distance matrix needs to be adjusted to ensure that the number of rows and columns are equal, thereby guaranteeing that each face image corresponds to a specific face template.

[0082] Specifically, if the number of rows in the distance matrix (representing the number of people in the real-time face set) is greater than the number of columns (representing the number of face templates in the live stream person library), then those extra rows need to be identified and deleted. Conversely, if the number of columns is greater than the number of rows, then the extra columns need to be deleted.

[0083] To determine which rows or columns should be deleted, the mean of the elements in each row and column of the distance matrix can be calculated. If there are more rows than columns, the mean of each row is calculated, and then the rows with the highest mean are selected for deletion. The row with the highest mean means that the distance between the corresponding face image and all face templates in that row is relatively large, indicating that these images have low similarity to templates in the library, and therefore these rows can be considered redundant. If there are more columns than rows, the mean of each column is calculated similarly, and the columns with the highest mean are selected for deletion.

[0084] For example, if the distance matrix has 6 columns and 7 rows, it means that there are 7 face images in the real-time face set, while the live stream person library only has 6 face templates. In this case, the mean of each row is calculated, and the row with the largest mean is found. The face images corresponding to these rows have the lowest similarity to the templates in the library, so they can be deleted from the distance matrix until the number of rows and columns are equal.

[0085] This embodiment achieves significant technical advantages in the field of multi-person identity recognition in live streams by further applying step S3133 based on the determined distance matrix, including but not limited to:

[0086] First, by adjusting the number of rows and columns in the distance matrix, the number of people in the real-time face set is ensured to match the number of face templates in the live-stream person database, thus avoiding matching errors caused by inconsistencies in the number of people. This method improves matching accuracy because it ensures that each face image can find a corresponding face template, reducing recognition errors caused by insufficient or redundant templates.

[0087] Secondly, by calculating the mean of each row and column in the distance matrix, face images with low similarity to templates in the library are intelligently identified and removed. This process not only optimizes the matching process but also improves recognition efficiency because it reduces unnecessary computation. Especially when dealing with large-scale data, this method can significantly reduce computational complexity and resource consumption.

[0088] Furthermore, this embodiment enhances adaptability and flexibility. In a live streaming environment, changes in the number of participants are common, and step S3133 provides a dynamic adjustment mechanism that can quickly respond to changes in the number of participants while maintaining high matching performance. This mechanism makes the identity recognition process more stable, ensuring the continuity and consistency of recognition results even when the number of participants fluctuates.

[0089] Furthermore, the application of step S3133 enhances the live streaming interactive experience. By ensuring that each facial image is accurately matched to the corresponding streamer, it can more accurately respond to viewer commands, such as the sending of virtual gifts and the application of beauty filters. This not only enhances viewer engagement but also creates more business opportunities for the live streaming platform, such as precise advertising and personalized content recommendations.

[0090] Based on any embodiment of the method in this application, the system responds to operation commands applied to the identity of the target anchor based on the mapping relationship data between the position information of each face image in the video frame and the identity identifier of the face template matching the face image, including:

[0091] Step S3311: In response to a touch operation command that acts on the identity of the target anchor in the display image of the video frame in the graphical user interface, determine the identity identifier of the target anchor based on the mapping relationship data between the position information of each face image in the video frame and the identity identifier of the face template that matches the face image.

[0092] In live streaming rooms, viewers often interact with video content through a graphical user interface. This application addresses this need by providing an interactive method that allows viewers to click on the target anchor figure in the video frame to trigger corresponding touch operation commands for human-computer interaction.

[0093] To respond to such touch operation commands, it is necessary to be able to identify the identity of the broadcaster being clicked. Therefore, the positional information of each facial image in the video frame can be determined in advance. When facial detection technology identifies the facial image, the corresponding model has already determined the selection box corresponding to the facial image in the video frame, and this selection box is determined in advance. These facial images have also been matched with facial templates in a pre-set live broadcast personality database, thus establishing a mapping relationship between positional information and identity identifiers.

[0094] When viewers perform touch operations on the graphical user interface, such as clicking on a presenter, the system locates the click position and retrieves the corresponding face image using a mapping table that reflects the relationship between the face image's position within the video frame and the identity identifier of the face template matching that face image. By using the previously established mapping table to query the position information, the system can quickly determine the face template corresponding to the face image, thereby identifying the target presenter.

[0095] For example, in a live streaming scenario, if a viewer clicks on a streamer in a video frame, the mapping table can be consulted based on the click's coordinates to determine the location information covering those coordinates. By mapping this location information to the identity identifier of a facial template, the identity identifier of the streamer corresponding to the clicked face image can be identified. It should be noted that this process can also be applied to various touch operations for different purposes, such as liking, commenting, or sending emojis, providing users with a way to directly interact with the live streaming content.

[0096] Therefore, it can be seen that by querying the mapping table based on the coordinates of the viewer's clicks, the target anchor can be determined effectively. This can effectively respond to the viewer's touch operation commands for the target anchor's identity in the video frame, realize the rapid matching between facial images and identity identifiers in the live stream, and enhance the interactivity and user experience of the live stream.

[0097] Step S3312: A virtual gift selection interface pops up to obtain gift configuration information, which includes the virtual gift that the user has selected to send to the target streamer.

[0098] During live stream interactions, viewers often express their support and affection for the streamer by sending virtual gifts. To implement this interactive function, a mechanism is needed that allows users to select and send virtual gifts to their chosen target streamer. When a viewer clicks on the target streamer in the graphical user interface, a virtual gift selection interface should pop up in response. This interface retrieves the user's gift-giving configuration information, including the virtual gifts the user has selected to send to the target streamer.

[0099] The virtual gift selection interface displays a range of virtual gift options, each with its own animated image and description. Users can choose one or more virtual gifts from these options based on their personal preferences and the emotions they wish to express. After selection, the user submits the gift-giving command, and the backend constructs the user's gift-giving configuration information and performs subsequent processing.

[0100] After viewers select a virtual gift through the virtual gift selection interface and trigger the backend to confirm the gift-giving configuration information, they submit it to the live streaming server. The live streaming server can then control the media server to perform subsequent processing steps based on this gift-giving configuration information.

[0101] Step S3313: Respond to the user's submitted gift-giving operation command, obtain the animated image of the virtual gift in the gift-giving configuration information, and composite the animated image into the face image of the target anchor person corresponding to the identity identifier in each video frame of the live stream.

[0102] When a viewer selects and submits a virtual gift on the graphical user interface, the live streaming server receives this gift-giving instruction and retrieves the corresponding gift-giving configuration information. This information includes details of the virtual gift selected by the user, including the animated image of the virtual gift and other related settings.

[0103] The live streaming server then sends this configuration information to the media server, instructing the media server to perform corresponding image processing operations based on the gift-giving configuration information. Upon receiving the instructions from the live streaming server, the media server determines the animated image of the virtual gift from its stored resources or real-time content. These animated images are pre-designed, correspond to the virtual gift, and are stored on the server in a computer-readable format.

[0104] Next, the media server decodes the live stream received from the broadcaster, processes the video frames of the decoded stream, and composites the animated images of virtual gifts onto the video frames of a specific broadcaster. Specifically, the media server determines the position of the target broadcaster within the video frame based on the previously established mapping between facial image location information and identity identifiers. Then, using various known computer graphics techniques, the animated images of virtual gifts are precisely overlaid onto the facial image of the target broadcaster.

[0105] In some embodiments, the compositing process may involve operations such as layer blending, transparency adjustment, and position correction to ensure seamless integration of the virtual gift animation with the live video frames. For example, if the virtual gift is a flower, the animation might show the flower rotating around the streamer or slowly drifting down onto them. If the gift is a banner, it might unfold to one side of the streamer or float above them.

[0106] After compositing, the media server distributes the processed live stream to the terminal devices of users in the live stream room. While watching the live stream, viewers will see virtual gift animations being added to the live video in real time, enhancing the interactivity and viewing experience.

[0107] This embodiment, based on the real-time identification and response of multiple identities in a live stream, further realizes the deep application of the mapping relationship data between facial images and their identity identifiers, thereby bringing significant technical advantages, including but not limited to:

[0108] First, by mapping the location information of facial images to identity markers, the system can quickly and accurately identify the specific broadcaster that a viewer clicks on within a video frame. This rapid matching capability greatly enhances the interactivity of the live stream, allowing for instant responses to viewer commands and improving the user experience.

[0109] Secondly, by responding to viewers' touch input and displaying a virtual gift selection interface, users are provided with an intuitive way to express their support and appreciation for the streamer. Users can easily select and send virtual gifts, and this interaction not only increases the fun of the live stream but also creates additional revenue streams for the platform. In this way, viewer engagement and loyalty are enhanced, while also strengthening the connection between the streamer and the audience.

[0110] Furthermore, by compositing animated images of virtual gifts into the live stream video, a completely new visual experience is provided to viewers. This compositing technology makes virtual gifts appear as if they are part of the live stream scene, enhancing the realism and immersion of gift-giving. Viewers can see their gifts appear in the host's video in real time; this instant visual feedback not only makes viewers feel more involved but also adds more entertainment elements to the live stream event.

[0111] Finally, in combination with other embodiments, this embodiment offers high adaptability and flexibility. It operates stably regardless of changes in the number of broadcasters, diverse audience interaction needs, or complex and ever-changing live streaming environments, ensuring the continuity and accuracy of identity recognition and response operations within the live stream. This stability and technological robustness provide strong technical support for the live streaming platform, enabling it to stand out in a highly competitive market.

[0112] Based on any embodiment of the method in this application, the system responds to operation commands applied to the identity of the target anchor based on the mapping relationship data between the position information of each face image in the video frame and the identity identifier of the face template matching the face image, including:

[0113] Step S3331: Respond to the beautification operation command and determine the identity identifier and beautification parameters of the target anchor person to which the command is applied;

[0114] During live streaming, broadcasters often need to enhance the visual experience by beautifying the persona in the live video. This embodiment allows broadcasters to achieve this through beauty operation commands. Specifically, broadcasters can select and set beautification parameters through the beauty interface provided in the live stream. These parameters can include beauty effects such as skin smoothing, whitening, and face slimming, as well as decorative accessories such as wreaths and crowns. Each broadcaster can apply uniform beautification parameters or personalized parameters based on their personal style or audience preferences.

[0115] When a live streamer selects specific beautification parameters on the beauty interface, these parameters are recorded and associated with the corresponding live streamer's identity. Once the live streamer submits the information they set to trigger a beauty operation command, the terminal device or media server responsible for implementing the beautification process can determine the identity and beautification parameters of the target live streamer based on these preset or real-time set parameters.

[0116] For example, if a streamer wants to apply beauty filters to a female streamer in their live stream, they can select a specific beauty level and decorative items in the beauty interface. The backend associates these parameters with the female streamer's identity and applies these beautification parameters in real time within the live stream. When the streamer issues a beauty operation command, the system recognizes the female streamer's face image in the video frames of the live stream and performs beautification processing according to the previously set beautification parameters.

[0117] In one embodiment, the broadcaster can select different beautification parameters using sliders, drop-down menus, or other controls on the beautification interface of the graphical user interface. These parameters are saved and recalled when needed to process facial images in the live video in real time. The device responsible for implementing the image beautification processing accurately applies the beautification effect to the correct broadcaster based on the location information and identity of each facial image, ensuring a natural and harmonious live broadcast.

[0118] In another embodiment, considering that each streamer may be a user of their own personal live stream room, and they typically have their own preferred beautification parameters, these parameters are stored in their personal accounts along with their identity identifiers. In multi-person live streaming events, streamer users can choose whether to enable personalized beautification parameters for each streamer. When this option is enabled, the corresponding beautification parameters can be automatically retrieved from their personal accounts after automatically identifying the streamer's identity identifier corresponding to each facial image.

[0119] For example, in a multi-person live stream event, suppose there are three streamers, each with their own personal account and preset beauty effects and accessories. Before the live stream begins, the streamers can decide whether to enable personalized beautification settings for each participant. Once enabled, after recognizing the identity of each person's face, the corresponding beautification parameters are automatically extracted from their personal account and applied to the live stream video. In this way, each streamer will display their preferred beautification effects during the live stream, enhancing the personalized experience.

[0120] Step S3332: Query the mapping relationship data according to the identity identifier, determine the position of the target anchor's face image in the video frame, and apply the beautification parameters to the face image to obtain a beautified image;

[0121] Once the livestreaming user specifies the identity identifier and beautification parameters of the target livestreamer for the beautification operation command, the mapping relationship data can be queried based on the identity identifier to determine the position of the target livestreamer's face image in the video frame.

[0122] The mapping data is constructed based on the results of face detection and identity matching. The position of each face image is determined in the video frame by a selection box and associated with a corresponding identity identifier. When a broadcaster submits a beautification command, this mapping data is queried through the identity identifier to quickly locate the exact position of the target broadcaster's face image in the video frame.

[0123] Once the location is determined, the previously selected beautification parameters will be applied to the facial image. Applying these parameters can include various image processing techniques, such as color correction, filter application, and geometric transformations. For example, if the beautification parameters include a beautification effect, skin tone adjustment, skin smoothing, or contour reshaping can be performed on the target facial image. If decorative accessories are included, virtual accessories such as wreaths or crowns can be added to the facial image.

[0124] In practice, the beautification parameters can be applied using different algorithms and techniques. For example, a deep learning-based image generation network can be used to generate high-quality beautified images, or traditional image processing techniques can be used to achieve real-time beautification effects. The choice of these techniques depends on the performance requirements of the live streaming system and the available computing resources. After applying the beautification parameters, a beautified image corresponding to the face image will be generated.

[0125] Step S3333: Replace the face image of the target anchor in the corresponding video frame with the beautified image, so that the video frame displays the corresponding beautified image for the target anchor in the graphical user interface.

[0126] After obtaining the beautified image corresponding to the target anchor's face image, the facial region of the target anchor in the video frame can be accurately identified based on the face image location information determined in the mapping relationship data. Then, the beautified image obtained after applying beautification parameters is composited with the original video frame, replacing the original face image. The specific compositing process can be achieved through computer graphics techniques such as layer blending, transparency adjustment, and position correction to ensure seamless integration of the beautified image with other parts of the video frame.

[0127] For example, if the streamer selects skin-smoothing and whitening beauty effects during a live stream, these effects are applied to the detected facial image, generating a new beautified image. Then, based on the facial image's location information, the corresponding facial region is located in the video frame, and the original image is replaced with the beautified image. If the streamer also selects virtual accessories, such as a crown or wreath, these accessories are also added to the beautified image and displayed at the corresponding position in the video frame.

[0128] In implementation, this step can employ various technical solutions. For example, deep learning-based image generation networks can be used to generate high-quality beautified images, or traditional image processing techniques such as filters and color correction can be used to achieve real-time beautification effects. The choice of these techniques depends on the performance requirements of the live streaming system and the available computing resources.

[0129] After the replacement is complete, the target anchor in the video frame will appear in the viewer's interface with an enhanced image, improving the visual appeal of the live stream and the viewing experience. This real-time enhancement not only strengthens the interactivity and watchability of the live stream content but also provides anchors with the opportunity to showcase their personalized image, thereby increasing the live stream's appeal and viewer engagement. In this way, live streaming platforms can provide a richer and more personalized viewing experience, meeting the needs and preferences of different users.

[0130] This embodiment achieves unique and significant technical advantages compared to other embodiments of this application, particularly in enhancing the interactivity and personalized experience of live streaming. By allowing broadcaster users to select and apply personalized beautification parameters for each broadcaster participating in a multi-person live stream, this embodiment not only satisfies the broadcasters' personalized needs for image presentation but also provides viewers with a more diverse and richer visual experience. This personalized beautification process allows each broadcaster to appear in the live stream in their most satisfactory image, enhancing the attractiveness of the live stream and the viewing interest of the audience.

[0131] Furthermore, this embodiment automates the retrieval and application of preset enhancement parameters from the broadcaster's personal account, greatly simplifying the enhancement process during live streaming. Broadcasters no longer need to manually adjust the enhancement effects for each individual during the live stream, allowing them to focus more on content creation and interaction, thus improving efficiency and smoothness. Simultaneously, this automation reduces errors and inconsistencies caused by manual operation, ensuring the professionalism and consistency of the live stream footage.

[0132] More importantly, this embodiment achieves seamless editing and enhancement of the live stream by replacing the original facial images with the enhanced images in real time within video frames. This real-time image processing technology not only improves the viewing experience of the live stream but also creates more business opportunities for the platform. For example, the platform can attract more users by offering personalized enhancement services or increase revenue by selling virtual items and effects. Simultaneously, this technology also provides broadcasters with more creative freedom, allowing them to customize their live stream appearance according to their own style and audience preferences, thereby enhancing the interactivity and engagement of the live stream.

[0133] In summary, this embodiment, through its unique technical solution, not only improves the visual quality and personalization of live streaming, but also creates more value and opportunities for live streaming platforms and broadcasters, thus achieving significant technical advantages in the live streaming field.

[0134] Based on any embodiment of the method in this application, the optimal match between the face image and the face template is solved based on the distance matrix, so that the sum of the data distances between the face image and the face template is the optimal solution, in order to determine the identity identifier of the anchor person to which the uniquely matched face template of each face image belongs, including:

[0135] Step S3210: Subtract the minimum value of the corresponding element in each row of the distance matrix to update the distance matrix, and then subtract the minimum value of the corresponding element in each column of the distance matrix to update the distance matrix.

[0136] Once it's determined that the distance matrix has the same number of rows and columns, it can be normalized to provide suitable input for subsequent matching algorithms. Specifically, each element in the distance matrix needs to be adjusted to reduce the range of absolute values, thereby simplifying the subsequent matching calculations.

[0137] Specifically, the first step is to process each row of the distance matrix by subtracting the minimum element value from the elements of that row. This operation does not change the relative differences between elements within a row, but it ensures that the minimum value of each row is zero. This is beneficial because it makes it easier to identify which rows contain zero values ​​when searching for lines covering zero values ​​in subsequent steps. Similarly, the same operation is performed on each column of the distance matrix, subtracting the minimum element value of that column from the elements of that column to ensure that the minimum value of each column is zero.

[0138] In this way, the distance matrix is ​​transformed into a new matrix where the minimum value of each row and each column is zero. This normalization process simplifies subsequent matching algorithms because the zero values ​​can be used to determine the optimal match. A zero value indicates that the cost between a face image and a face template is zero, meaning they are an optimal match. For example, suppose there is a 6x6 distance matrix containing distance values ​​between face templates from a real-time face set and a live-streamed person library. After performing the minimum subtraction operation in the rows and columns, each row and each column of the matrix will have at least one zero value. These zero values ​​will serve as the basis for finding matches in subsequent steps. This normalization process, for the optimal matching algorithm implemented in this embodiment, not only reduces the complexity of numerical computation but also provides clear reference points for the algorithm.

[0139] Step S3220: Find the minimum number of horizontal and vertical lines that can cover all zero-value elements in the distance matrix. When the total number of lines is the same as the total number of face images, stop searching to determine the distance matrix at this time as the optimal matching matrix. Otherwise, determine the minimum value among the values ​​of the elements not covered by lines, add the minimum value to the value of the element corresponding to the intersection of any two lines, and subtract the minimum value from the value of the element not covered by lines before iterating this step.

[0140] After ensuring that the distance matrix has the same number of rows and columns and normalizing it, the next step is to find a method to determine the optimal match between a face image and a face template. In this embodiment, the optimal match refers to the minimum match. To achieve this, we need to find the minimum number of horizontal and vertical lines in the distance matrix that cover all zero-value elements. These lines represent possible matching paths, and their goal is to minimize the total matching cost, i.e., the sum of data distances, while ensuring that each face image matches a face template.

[0141] Specifically, the normalized distance matrix is ​​first examined to find all zero-value elements, and then an attempt is made to cover these zero values ​​with the fewest possible lines. Each line, whether horizontal or vertical, represents a potential match between a face image and a face template. When the number of lines found equals the total number of face images, it means that a matching face template has been found for each face image, and the distance matrix at this point is considered the optimal matching matrix.

[0142] However, if, in attempting to cover all zero-value elements, the number of lines used has not yet reached the total number of face images, this indicates that a matching face template has not yet been found for each face image. In this case, the distance matrix needs further adjustment to facilitate more matches. Specifically, find the minimum value among the elements not covered by lines, add this minimum value to the element corresponding to the intersection of any two lines, and subtract this minimum value from the elements not covered by lines, then continue iterating this step. Such adjustments help find more matching paths in the next iteration.

[0143] In this way, the distance matrix can be iteratively adjusted until the minimum number of lines covering all zero-value elements is found, and the number of lines equals the number of face images, thus determining the optimal match between each face image and the face template. This distance matrix can then be used as the subsequent optimal matching matrix. This process not only ensures matching accuracy but also improves matching efficiency because it avoids unnecessary complex calculations and directly optimizes towards finding the optimal solution.

[0144] Step S3230: Based on the optimal matching matrix, find and use augmenting paths to continuously update the matching status between the face image and the face template until no new augmenting path can be found and the optimal matching relationship between each face image and its corresponding face template is fixed.

[0145] The optimal matching matrix is ​​merely the foundation for determining the optimal match, not the optimal matching relationship itself. Therefore, after constructing the optimal matching matrix, this matrix is ​​used to find the optimal matching relationship between the face image and the face template. This can be done by searching for augmenting paths within this matrix and iteratively updating the matching state within the optimal matching matrix until no new augmenting paths can be found, thus determining the optimal matching relationship.

[0146] An augmenting path is a path from a face image to a face template within the optimal matching matrix, where each node on the path alternates between unmatched and matched states. By finding such a path, the matching state can be updated, allowing more face images to match the corresponding face templates. Specifically, when an augmenting path is found in the optimal matching matrix, the matching state of the face images and face templates along the path can be changed, thereby increasing the number of matches.

[0147] First, examine the optimal matching matrix to find zero-valued elements that can form augmenting paths. If a zero-value is found in the matrix, and the row and column containing that zero-value have not been matched before, then an augmenting path has been found. Along this path, the face image can be matched against a face template, and the corresponding row and column are marked as matched. Then, continue searching for new augmenting paths in the updated matrix, repeating this process until no more new augmenting paths can be found.

[0148] By continuously searching for and utilizing augmenting paths, the matching state can be gradually updated until all face images in the matrix have found matching face templates, thereby fixing a stable matching relationship. That is, each face image has established a unique matching relationship with its corresponding face template, which is the optimal matching relationship pursued in this embodiment.

[0149] The unique and significant technical advantage of this embodiment lies in its ability to effectively handle situations where the number of face images in a live stream is inconsistent with the number of preset face templates, ensuring the accuracy and efficiency of the matching process. By normalizing the distance matrix, not only are subsequent matching calculations simplified, but clear reference points are also provided for the algorithm, enabling it to quickly identify the optimal match. This method is particularly suitable for live streaming environments where the number of face images may vary for various reasons, and this embodiment can dynamically adjust the distance matrix to adapt to these changes.

[0150] Furthermore, this embodiment iteratively updates the matching state by finding augmenting paths. This method progressively optimizes the matching results until a globally optimal solution is reached. This iterative optimization process not only improves matching accuracy but also enhances adaptability, enabling stable operation in complex live streaming environments. In this way, this embodiment ensures that each face image can find its best-matching face template, thereby improving the accuracy of identity recognition in live streams.

[0151] More importantly, this embodiment also reduces the consumption of computing resources. Through normalization and iterative optimization, unnecessary complex calculations are avoided, making the matching process more efficient. This is especially important for live streaming platforms that need to process large amounts of data in real time, as it can ensure high performance while providing a smooth user experience.

[0152] Therefore, this embodiment, through its innovative technical solution, not only improves the accuracy and efficiency of multi-person identity recognition in live streams, but also enhances the system's adaptability and stability, providing robust technical support for live streaming platforms. These technical advantages give this application a significant competitive edge in the live streaming field, providing a solid foundation for improving live streaming interactivity and user experience.

[0153] Based on any embodiment of the method in this application, based on the optimal matching matrix, the matching state between the face image and the face template is continuously updated using augmenting paths until no new augmenting paths can be found, thus fixing the optimal matching relationship between each face image and its corresponding face template, including:

[0154] Step S3231: Create a matching relationship set and initialize the matching relationship set to an empty set;

[0155] To record and manage the matching status of each face image and its corresponding face template, a matching relationship set can be created and initialized to empty so that the matching results derived from the optimal matching matrix can be stored later.

[0156] Step S3232: Based on the optimal matching matrix, traverse each face image and find the augmentation path corresponding to one of the face templates for each face image. After reversing the augmentation path, add it to the matching relationship set to update the matching relationship.

[0157] After determining the optimal matching matrix, we can iterate through each face image in the matrix and find a corresponding face template for each face image to form the optimal matching relationship. The goal of detailed analysis of the optimal matching matrix is ​​to find a path that establishes a unique match between a face image and a face template while ensuring the lowest total matching cost.

[0158] Specifically, for each face image in the matrix, an augmenting path is sought—a path that starts from an unmatched face image, alternates between matching and unmatched states, and ultimately reaches an unmatched face template. The existence of this path indicates that the number of matches can be increased by changing the current matching state, thereby optimizing the overall matching result.

[0159] After finding such an augmenting path, a crucial operation is performed: updating the matches. This operation involves reversing the states on the augmenting path, changing the previously matched states to unmatched states and the unmatched states to matched states. This state reversal helps discover new matching opportunities because it may reveal potential matches that were not previously considered.

[0160] After this operation is completed, the matching set is updated. By adding the results of the augmenting paths to the matching set, all identified matching relationships can be tracked and managed. This process is repeated until all face images in the optimal matching matrix have been traversed and no new augmenting paths can be found.

[0161] Step S3233: Use the set of matching relationships obtained after the traversal as the representation of the optimal matching relationship.

[0162] It's easy to understand that the process of constructing the matching relationship set is as follows: whenever an augmenting path is found and the matching state is updated, the result of this update is added to the set. As the traversal proceeds, the set gradually becomes richer until all possible augmenting paths have been explored. Ultimately, the set contains the matching relationships between all face images and their corresponding face templates, and these relationships together constitute the complete representation of the optimal match.

[0163] For example, suppose there are four face images and four face templates in a live stream. By traversing and finding augmenting paths, face image 1 might be matched with template 1 first, then face image 2 with template 3, and so on. After each match, these matching relationships are added to a set. When all augmenting paths have been explored and no new augmenting paths can be found, the update stops, and the set of matching relationships at this point represents the final optimal matching result.

[0164] This set not only records the matching results but also provides a basis for subsequent operations. It can be used as a source table representing the mapping relationship between the location information of face images and the identity identifiers of face templates. For example, during a live broadcast, when it is necessary to apply specific effects to face images or respond to specific operation commands, this set can be queried directly to quickly find the face template corresponding to each face image, thereby performing the corresponding operation. This method improves matching efficiency and ensures the real-time nature and interactivity of the live broadcast.

[0165] The algorithm described in this embodiment brings significant technical advantages to this application, especially in ensuring the efficiency of multi-person identity recognition in live streams. By creating and utilizing a set of matching relationships, this process can systematically record and manage the matching status of each face image and its corresponding face template, thereby achieving fast and accurate matching result updates. The core of this method lies in its ability to iteratively find augmenting paths, which allows for the search for possible optimal matches at each step and the continuous optimization of matching relationships through reversal operations until a globally optimal solution is reached.

[0166] This augmenting path-based update mechanism not only improves the flexibility of the matching process but also significantly enhances matching efficiency. In live streaming environments, real-time performance is crucial, and the method described in this embodiment ensures the accuracy and stability of the matching results while maintaining real-time performance. By traversing the optimal matching matrix and searching for augmenting paths, it can quickly respond to changes in the live stream, such as newly appearing faces or face templates, thereby dynamically adjusting the set of matching relationships to ensure that each face image can be correctly matched with its corresponding face template.

[0167] Furthermore, the efficiency of this process is also reflected in its optimized utilization of computing resources. By avoiding unnecessary complex calculations, it can process large amounts of data in a short time, which is especially important for live streaming platforms that need to handle high frame rate video streams. This approach reduces the computational burden, allowing more resources to be invested in improving live streaming quality and enhancing the user interaction experience.

[0168] In summary, this embodiment not only improves the accuracy and real-time performance of multi-person identity recognition in live streams but also optimizes the use of computing resources, enabling live streaming platforms to provide a smoother and more interactive live streaming experience. These advantages give this application a significant competitive edge in the field of live streaming technology and provide a solid technical foundation for the future development of live streaming platforms.

[0169] Please refer to Figure 3. A multi-person identity recognition device in a live stream, according to one aspect of this application, includes a similarity calculation module 3100, a face matching module 3200, and a mapping response module 3300. The similarity calculation module 3100 is configured to determine the data distance between each face image in the live stream and multiple face templates corresponding to multiple anchors in a preset live streamer database for that live stream, obtaining a distance matrix. The face matching module 3200 is configured to solve for the optimal match between the face image and the face template based on the distance matrix, such that the sum of the data distances between the face image and the face template is the optimal solution, thereby determining the identity identifier of the anchor to which the uniquely matched face template of each face image belongs. The mapping response module 3300 is configured to respond to operation commands acting on the target anchor's identity based on the mapping relationship data between the position information of each face image in the video frame and the identity identifier of the face template matching that face image.

[0170] Based on any embodiment of the device in this application, the similarity operation module 3100 includes: a tracking and detection module, configured to extract video frames from the live stream transmitted in real time from the live broadcast room, perform face tracking and detection on the video frames to obtain multiple face images therein, forming a real-time face set; a distance calculation module, configured to calculate the data distance between the semantic features of each face image in the real-time face set and each face template in the live broadcast person database, and represent the data distance relationship between each face image and each face template as a distance matrix; and a row and column normalization module, configured to detect whether the number of rows and columns of the distance matrix are the same, and when they are different, to add missing rows or columns and assign zero values ​​to each element therein to update the distance matrix.

[0171] Based on any embodiment of the device in this application, the similarity calculation module 3100 includes: a tracking and detection module, configured to extract video frames from the live stream transmitted in real time from the live broadcast room, perform face tracking and detection on the video frames to obtain multiple face images therein, forming a real-time face set; a distance calculation module, configured to calculate the data distance between the semantic features of each face image in the real-time face set and each face template in the live broadcast person database, and represent the data distance relationship between each face image and each face template as a distance matrix; and a row and column alignment module, configured to calculate the larger number of rows and columns in the distance matrix, calculate the average distance corresponding to the element values ​​of each column / row for each row / column with a larger number of rows / columns, and delete several rows / columns with the lowest similarity represented by the average distance, so that the number of rows and columns in the distance matrix is ​​consistent.

[0172] Based on any embodiment of the device in this application, the mapping response module 3300 includes: a gift-giving response module, configured to respond to a touch operation command acting on the target anchor's identity in the display image of the video frame in the graphical user interface, and determine the identity identifier of the target anchor's identity based on the mapping relationship data between the position information of each face image in the video frame and the identity identifier of the face template matching the face image; a gift-giving determination module, configured to pop up a virtual gift selection interface for obtaining gift-giving configuration information, which includes the virtual gift that the user determines to give to the target anchor's identity; and a gift-giving synthesis module, configured to respond to the gift-giving operation command submitted by the user, obtain the animated image of the virtual gift in the gift-giving configuration information, and synthesize the animated image into the face image of the target anchor's identity in each video frame of the live stream corresponding to the identity identifier.

[0173] Based on any embodiment of the device in this application, the mapping response module 3300 includes: a beautification response module, configured to respond to a beautification operation command, determine the identity identifier and beautification parameters of the target anchor person whose identity is affected by the command; a beautification application module, configured to query mapping relationship data according to the identity identifier, determine the position of the face image of the target anchor person in the video frame, apply the beautification parameters to the face image to obtain a beautified image; and a beautification synthesis module, configured to replace the face image of the target anchor person in the corresponding video frame with the beautified image, so that the video frame displays the corresponding beautified image for the target anchor person in the graphical user interface.

[0174] Based on any embodiment of the device in this application, the face matching module 3200 includes: an initial processing module, configured to subtract the minimum value of the corresponding element in each row of the distance matrix to update the distance matrix, and then subtract the minimum value of the corresponding element in each column of the distance matrix to update the distance matrix; a flow calculation module, configured to find the minimum number of horizontal and vertical lines that can cover all zero-value elements in the distance matrix, and stop searching when the total number of lines is the same as the total number of face images to determine the distance matrix at this time as the optimal matching matrix; otherwise, determine the minimum value among the values ​​of elements not covered by lines, add the minimum value to the value of the element corresponding to the intersection of any two lines, and subtract the minimum value from the value of the element not covered by lines and iterate this step; and a matching determination module, configured to find and continuously update the matching status of the face image and the face template based on the optimal matching matrix and using augmenting paths until no new augmenting path can be found and the optimal matching relationship between each face image and its corresponding face template is fixed.

[0175] Based on any embodiment of the apparatus in this application, the matching determination module includes: a set creation module, configured to create a set of matching relationships and initialize the set of matching relationships to an empty set; an augmentation update module, configured to traverse each face image in the optimal matching matrix, find an augmentation path corresponding to one of the face templates for each face image, reverse the augmentation path and add it to the set of matching relationships to update the matching relationships; and a relationship representation module, configured to use the set of matching relationships obtained after traversal as a representation of the optimal matching relationship.

[0176] Another embodiment of this application provides a device for multi-person identification in a live stream. Figure 4 shows a schematic diagram of the internal structure of the device. This device includes a processor, a computer-readable storage medium, a memory, and a network interface connected via a system bus. The computer-readable, non-volatile storage medium stores an operating system, a database, and computer-readable instructions. The database may store information sequences. When the computer-readable instructions are executed by the processor, the processor can implement a method for multi-person identification in a live stream.

[0177] The processor of the multi-person identification device in the live stream provides computing and control capabilities to support the operation of the device throughout the live stream. The memory of the multi-person identification device in the live stream may store computer-readable instructions, which, when executed by the processor, cause the processor to perform the multi-person identification method in the live stream of this application. The network interface of the multi-person identification device in the live stream is used for communication with the terminal.

[0178] Those skilled in the art will understand that the structure shown in Figure 4 is merely a block diagram of a portion of the structure related to the present application and does not constitute a limitation on the multi-person identification device in the live stream to which the present application is applied. The specific multi-person identification device in the live stream may include more or fewer components than those shown in the figure, or combine certain components, or have different component arrangements.

[0179] In this embodiment, the processor executes the specific functions of each module in Figure 3, and the memory stores the program code and various types of data required to execute the above modules or sub-modules. The network interface is used to realize data transmission between user terminals or servers. The non-volatile readable storage medium in this embodiment stores the program code and data required to execute all modules in the multi-person identity recognition device in the live stream of this application, and the server can call the server's program code and data to execute the functions of all modules.

[0180] This application also provides a non-volatile readable storage medium storing computer-readable instructions, which, when executed by one or more processors, cause the one or more processors to perform the steps of the multi-person identification method in a live stream according to any embodiment of this application.

[0181] This application also provides a computer program product, including a computer program / instructions that, when executed by one or more processors, implement the steps of the method described in any embodiment of this application.

[0182] Those skilled in the art will understand that all or part of the processes in the methods of the above embodiments of this application can be implemented by a computer program instructing related hardware. This computer program can be stored in a non-volatile readable storage medium, and when executed, it can include the processes of the embodiments of the above methods. The aforementioned storage medium can be a computer-readable storage medium such as a magnetic disk, optical disk, read-only memory (ROM), or random access memory (RAM).

[0183] In summary, this application accurately handles common issues in live streaming scenarios such as profile views, head tilting, occlusion, exaggerated expressions, and motion blur, ensuring precise identity recognition in multi-person live streaming environments. By constructing and optimizing the distance matrix, this application not only improves the accuracy of face recognition but also achieves globally optimal matching, avoiding the mismatch and duplication of anchor identities that may occur with traditional methods in multi-person live streaming scenarios. Furthermore, by establishing a mapping relationship between each face image and its corresponding face template, this application enables the system to respond to user commands for specific individuals, such as sending virtual gifts and applying beauty effects, thereby greatly enhancing the interactivity and fun of live streaming. These advantages not only improve the user experience but also provide live streaming content creators with more personalized and interactive tools, further enriching live streaming gameplay and promoting the innovation and development of live streaming technology.

Claims

1. A method for identifying multiple persons in a live stream, characterized in that, include: Determine the data distance between each face image in the live stream and multiple face templates corresponding to multiple anchors in the preset live streamer database for that live stream, and obtain the distance matrix; Based on the distance matrix, the optimal match between the face image and the face template is solved so that the sum of the data distances between the face image and the face template is the optimal solution, so as to determine the identity identifier of the anchor person to which the uniquely matched face template of each face image belongs; Based on the mapping relationship data between the position information of each face image in the video frame and the identity identifier of the face template that matches the face image, the system responds to operation commands that apply to the identity of the target anchor.

2. The method of claim 1, wherein, Determine the data distance between each facial image in the live stream and multiple facial templates corresponding to multiple anchors in the preset live streamer database for that live stream, and obtain a distance matrix, including: Video frames are extracted from the live stream transmitted in real time from the live broadcast room, and face tracking and detection are performed on the video frames to obtain multiple face images and form a real-time face set. Calculate the data distance between the semantic features of each face image in the real-time face set and each face template in the live-streaming character database, and represent the data distance relationship between each face image and each face template as a distance matrix; Check if the number of rows and columns of the distance matrix are the same. If they are different, add the missing rows or columns and assign zero values ​​to each element therein to update the distance matrix.

3. The method of claim 1, wherein, Determine the data distance between each facial image in the live stream and multiple facial templates corresponding to multiple anchors in the preset live streamer database for that live stream, and obtain a distance matrix, including: Video frames are extracted from the live stream transmitted in real time from the live broadcast room, and face tracking and detection are performed on the video frames to obtain multiple face images and form a real-time face set. Calculate the data distance between the semantic features of each face image in the real-time face set and each face template in the live-streaming character database, and represent the data distance relationship between each face image and each face template as a distance matrix; Calculate the row and column with the larger number of elements in the distance matrix. For each row / column with the larger number of elements, calculate the mean distance between the elements in each column / row. Delete the rows / columns with the lowest mean distance to make the number of rows and columns in the distance matrix consistent.

4. The method of claim 1, wherein, Based on the mapping relationship data between the position information of each facial image in the video frame and the identity identifier of the facial template matching that facial image, the system responds to operation commands applied to the identity of the target anchor, including: In response to a touch operation command that acts on the identity of the target anchor in the display image of the video frame in the graphical user interface, the identity identifier of the target anchor is determined based on the mapping relationship data between the position information of each face image in the video frame and the identity identifier of the face template that matches the face image. A virtual gift selection interface pops up to obtain gift configuration information, which includes the virtual gifts that the user has selected to send to the target streamer. In response to the user's gift-giving instruction, the system obtains the animated image of the virtual gift from the gift-giving configuration information and composites the animated image into the face image of the target anchor person corresponding to the identity identifier in each video frame of the live stream.

5. The method of claim 1, wherein, Based on the mapping relationship data between the position information of each facial image in the video frame and the identity identifier of the facial template matching that facial image, the system responds to operation commands applied to the identity of the target anchor, including: Responding to beautification operation commands, determine the identity identifier and beautification parameters of the target anchor whose identity is affected by the command; Based on the identity identifier query mapping relationship data, determine the position of the target anchor's face image in the video frame, and apply the beautification parameters to the face image to obtain a beautified image; The beautified image replaces the face image of the target anchor in the corresponding video frame, so that the video frame displays the corresponding beautified image for the target anchor in the graphical user interface.

6. The method of claim 1-5, wherein, Based on the distance matrix, the optimal match between the face image and the face template is calculated, so that the sum of the data distances between the face image and the face template is the optimal solution. This is used to determine the identity identifier of the anchor whose face template uniquely matches each face image, including: The distance matrix is ​​updated by subtracting the minimum value of the corresponding element from each element in each row of the distance matrix, and then the distance matrix is ​​updated by subtracting the minimum value of the corresponding element from each element in each column of the distance matrix. Find the minimum number of horizontal and vertical lines that can cover all zero-value elements in the distance matrix. When the total number of lines is the same as the total number of face images, stop searching to determine the distance matrix at this time as the optimal matching matrix. Otherwise, determine the minimum value among the values ​​of the elements not covered by lines, add the minimum value to the value of the element corresponding to the intersection of any two lines, and subtract the minimum value from the value of the elements not covered by lines, and then iterate this step. Based on the optimal matching matrix, the matching status between the face image and the face template is continuously updated by finding and utilizing augmenting paths until no new augmenting paths can be found, thus fixing the optimal matching relationship between each face image and its corresponding face template.

7. The method of claim 6, wherein, Based on the optimal matching matrix, the matching state between the face image and the face template is continuously updated using augmenting paths until no new augmenting paths can be found, thus fixing the optimal matching relationship between each face image and its corresponding face template. This includes: Create a set of matching relationships, and initialize the set of matching relationships to an empty set; Based on the optimal matching matrix, each face image is traversed, and for each face image, an augmentation path corresponding to one of the face templates is found. The augmentation path is then reversed and added to the matching relationship set to update the matching relationship. The set of matching relationships obtained after completing the traversal is used as the representation of the optimal matching relationship.

8. A device for identifying multiple individuals in a live stream, characterized in that, include: The similarity calculation module is set to determine the data distance between each face image in the live stream of the live room and multiple face templates corresponding to multiple anchors in the preset live character library of the live room, and obtain a distance matrix; The face matching module is configured to solve for the optimal match between the face image and the face template based on the distance matrix, so that the sum of the data distances between the face image and the face template is the optimal solution, so as to determine the identity identifier of the anchor person to which the uniquely matched face template of each face image belongs; The mapping response module is configured to respond to operation commands applied to the identity of the target anchor based on the mapping relationship data between the position information of each face image in the video frame and the identity identifier of the face template that matches the face image.

9. A device for identifying multiple individuals in a live stream, comprising a central processing unit and a memory, characterized in that, The central processing unit is used to invoke and run a computer program stored in the memory to perform the steps of the method as described in any one of claims 1 to 7.

10. A non-volatile readable storage medium, characterized in that, It stores, in the form of computer-readable instructions, a computer program implemented according to any one of claims 1 to 7, which, when invoked by a computer, executes the steps included in the corresponding method.

Images