Provision of an output of video framings

The method automates video framing selection using scoring algorithms and user feedback to enhance viewer engagement in multi-camera setups, reducing costs and improving interaction capture.

WO2026234003A1PCT designated stage Publication Date: 2026-11-12BARCO NV
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
PCT/EP2026/062947
Authority / Receiving Office
WO · WO
Patent Type
Applications
Current Assignee / Owner
Priority Date
2025-05-06
Filing Date
2026-05-05
Publication Date
2026-11-12

AI Technical Summary

Technical Problem

Existing multi-camera setups in applications like corporate presentations and TV broadcasting require skilled technical teams, are time-consuming, and expensive, and often fail to capture all significant interactions between persons due to inadequate automation.

Method used

A method for generating and selecting video framings based on scoring algorithms applied to person coordinate points, allowing automated selection and control of camera angles to highlight relevant interactions, with optional user feedback and reinforcement learning for improved decision-making.

Benefits of technology

Reduces the need for skilled personnel, enhances viewer engagement by capturing dynamic interactions, and provides flexible, cost-effective multi-camera setups adaptable to various environments.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure EP2026062947_12112026_PF_FP_ABST
    Figure EP2026062947_12112026_PF_FP_ABST
Patent Text Reader

Abstract

The invention provides, amongst other aspects, a method comprising: obtaining input video streams and processing the same to identify at least one person therein, wherein the at least one person is identified by respectively detecting at least one set of person coordinate points of the at least one identified person, generating a respective plurality of video framings for each of the identified at least one person, determining a respective score for each of the video framings by applying at least one first scoring algorithm to each of the video framings and to the respective detected at least one set of person coordinate points, and providing the output comprising a first video framing and a second video framing from among the respective video framings, wherein the first video framing is selected over the second video framing based on the respective scores of the first and second video framings.
Need to check novelty before this filing date? Find Prior Art

Description

Provision of an output of video framingsField of the invention

[0001] The present invention relates to the technical domain of providing an output of video framings, particularly comprising a first video framing and a second video framing from among a generated plurality of video framings, wherein the first video framing is selected over the second video framing based on respective scores thereof generated by applying at least one first scoring algorithm to the generated plurality of video framings and to a respective detected at least one set of person coordinate points for an identified at least one person in one or more input video streams.Background art

[0002] In applications such as corporate presentations, hybrid meetings, TV broadcasting, and high-end online education, person framing plays a crucial role in maintaining viewer engagement and delivering a professional visual experience. By varying camera angles and perspectives, these applications preferably use multiple cameras to capture dynamic, real-time interactions. This multicamera approach not only highlights the main speaker but also includes the reactions and interactions of others, creating a more immersive and captivating experience for the audience.

[0003] Deploying multiple cameras requires a skilled technical team, including camera operators, audio engineers, and a director, to keep the production visually appealing and engaging. However, this process is both time-consuming and expensive.

[0004] To address the high costs and extensive resources required for multi-camera setups, automation becomes essential. By implementing automated systems, the need for a large technical crew is significantly reduced, thereby cutting down on expenses. Additionally, automation makes it feasible to integrate this multi-camera approach into smaller-scale applications, such as meeting rooms, where having a full technical team can be impractical.

[0005] US11048919B1 describes tracking people across multiple segments of video data, which can correspond to different scenes in a single video file, or multiple video streams or feeds, where an instance of video data can be broken up into segments that can each be analysed to determine faces and bodies represented therein, and bodies can be analysed across frames of the segment to determine body tracklets that are consistent across the segment. Associations of faces and bodies can be determined based using relative distances and / or spatial relationships. A subsequent clustering of these associations is performed to attempt to determine consistent associations that correspond to unique individuals. Unique identifiers are determined for each person represented in one or more segments of an instance of video data. Such an approach enables individual representations to be correlated across multiple instances.

[0006] Fully automated systems suffer from several challenges, such as focusing only on the person speaking, and therefore, miss important non-verbal cues of other persons, or such as when multi-camera setups are used, they lack sufficient decision making of which camera is to be used for broadcasting. These challenges highlight the need for improved methods that can overcome these limitations, such as smarter solutions to fully capture all significant interactions between persons. The present invention aims at addressing issues, such as the issues mentioned above.Summary of the invention

[0007] According to a first aspect, the present invention provides a (computer-implemented) method for providing an output of video framings, the method comprising:obtaining one or more input video streams,processing the one or more input video streams, preferably by applying at least one detection algorithm, such as at least one neural network, to each of the one or more input video streams, to identify at least one person in the one or more input video streams and to detect at least one set of person coordinate points of the at least one identified person,generating a respective plurality of video framings for each of the identified at least one person,determining a respective (characteristic) score for each of the generated plurality of video framings by applying at least one first scoring algorithm to each of the generated plurality of video framings and to body coordinate points comprised in the detected respective at least one set of person coordinate points within the respective plurality of video framings, andproviding the output comprising a first video framing and a second video framing from among the respective plurality of video framings, wherein the first video framing is selected over the second video framing based on the respective (characteristic) scores of the first and second video framings.

[0008] Herein, the output provided may relate to "singling out" the generated video framing(s) that best matches a technical feature that is to be detected. This is particularly done based on the determined respective (characteristic) scores of the generated video framings. In advantageous embodiments, the technical feature concerns an image-related metric (e.g., any one or combination of: resolution, sharpness, contrast, color accuracy, noise, dynamic range, compression, bit depth, etc.). However, the technical feature may be anything that can be automatically inferred from the generated video framings, and may, e.g., relate to a human activity.

[0009] The method of the present invention may advantageously provide an output of the “relevant” or “representative” framings, which “illustrate best” what matches the relevant characteristics of the framings of the at least one person identified therein. Thereby, the output comprises a selection of the first video framing above or over the second video framing. The selection is equivalent to a ranking in a sequence or a short-listing of the video framings. Thereby, in embodiments, the relative ranking of video framings, e.g., according to descending (characteristic) scores, may be a further advantage in that the most “relevant” example may appear first. Thus, the method ensures not only that the output comprises “relevant” video framings, but also video framings which score highly on particular relevant characteristic(s), such as at least one of: how well a (potentially moving) person is visible in the framing, how well a person’s face is visible in the framing, how well a person’s face is aligned in the framing, etc. The relevant characteristic(s) may be any image / video-related metric, such as at least one of: resolution, sharpness, contrast, color accuracy, noise, dynamic range, compression, bit depth, etc.).

[0010] By continuously processing the one or more input video streams, the method allows to track the identified at least one person in the one or more input video streams. Therefore, the method provides the output continuously at different moments in the one or more input video streams.

[0011] In embodiments, the method further comprises:outputting first display signals for displaying the output on a control display, and preferably outputting second display signals for broadcasting the video framing having the highest (characteristic) score.

[0012] By outputting first display signals to the control display, a director can more easily detect the selection process or ranking of the generated video framings. Furthermore, the director can more easily detect the generated video framing to be displayed / broadcasted to one or more other displays. This can allow the director to direct one or more camera personnel to capture the desired shot according to the first output, i.e., according to the first video framing or the video framing having the highest score. Preferably, the method can allow the director to automate the broadcasting of a video framing by outputting the second display signals, with or without the intervention of the director.

[0013] In embodiments, said transmitting first display signals comprises cropping out the generated video framings and optionally upscaling the cropped video framing to be displayed on the control display. Additionally, or alternatively, said transmitting first display signals comprises generating control signals for controlling a camera to perform at least one of panning, tilting, and optical zooming for capturing the video framings. In embodiments, said transmitting second display signals comprises cropping out the video framing having the highest score and upscaling the cropped video framing to be broadcasted. Additionally, or alternatively, said transmitting second display signals comprises generating control signals for controlling a camera to perform at least one of panning, tilting, and optical zooming for capturing the video framing having the highest score. Thus, the method according to the invention can be used with cameras which are not controllable, e.g., laptop camera, fixed camera, fisheye camera, etc. However, the method may additionally or alternatively be used camera which are controllable, e.g., PTZ cameras. It may be that even if a controllable camera is used, the first display signals may comprise at least one cropped-out video framing of a respective at least one of the generated video framings. Additionally, or alternatively, the second display signals may comprise a cropped-out video framing of the generated video framing having the highest score. The method furthermore provides more flexibility, where a combination of fixed and controllable cameras are setup.

[0014] In embodiments, the method further comprises:providing a graphical user interface for receiving user feedback from a user with respect to a relevance of the provided output, andadapting said determining of a (characteristic) score based on the received user feedback.

[0015] Preferably, the user feedback may comprise one or more user selections with respect to one or more video framings from the plurality of video framings for the detected person to be continuously displayed on the control display and / or to be directly broadcasted.

[0016] Such an approach allows a user (e.g., director) to guide and teach at least one of the at least one first, second, third scoring algorithm, by providing feedback to the algorithm on what is a more relevant or representative framing. This allows the user to maintain control over crucial aspects while automating routine tasks. This not only enhances the algorithm’s performance through real-time feedback but also provides users with more time to focus on higher-level decisions. The present invention aims to address this balance, combining human expertise with improved decision making for optimal results. Furthermore, such a hybrid approach where users can interact with the user interface, allows to improve the decision making of which framing(s) is / are displayed on the first display and which framing is to be broadcast and at what time.

[0017] In embodiments, the method further comprises:providing a large language model (LLM) based agent,providing a user interface for receiving a user prompt from a user to communicate with the LLM-based agent,adapting said determining of a (characteristic) score and / or adapting the output based on the received user prompt.

[0018] The LLM-based agent may be configured to communicate with an application programming interface (API), e.g., to interpret API definitions, and thereby interact with one or more algorithms (e.g., at least one of the first, second, third scoring algorithms) according to the invention. The system according to the present invention may be equipped with a Representational State Transfer Application Programming Interface (REST API) which provides endpoints to get information on the detected persons, the generated video framings and the scores / rankings thereof, as well allows to take actions to select and display certain framings. By providing the LLM-based agent with access to the API and allowing the agent to reason on the meaning of each of the endpoints, routines can be directed to the agent by an operator by means of an input prompt in natural language. Said REST API may communicate with the LLM-based agent over HTTP. An advantage of providing an LLM-based agent and a user interface to interact therewith is that the method not only allows to the user to provide feedback, as described before using the graphical user interface, in the form of prompts (note that the graphical user interface for receiving user feedback may be the same as the user interface for receiving user prompts), however, the method also enriches routines with functionalities provided by devices within the system according to the invention. For example, the user may instruct the LLM-based agent to broadcast immediately a close-up of a displayed person, the LLM-based agent may adapt the score of close-up framings by increasing their score and / or may send instructions to a controllable camera to zoom in to a specified position, such that the displayed person’s face is in close-up. In another example, the user prompt may be conditional, such as “when person X appears, display a close-up shot of person X”, or may describe a script for an event, in which different people appear over time and require certain displayed shots at certain moments in time.

[0019] Furthermore, the method allows a user and / or the LLM-based agent to interact with other (artificial intelligence) agents to collaborate with other smart systems. For example, the LLM-basedagent can communicate with a building automation system, e.g., for diming one or more lights when a person of interest is detected on stage.

[0020] By allowing rules in natural language the method can be tailored to the specific needs of an operator without explicitly defining said needs in the scoring algorithm(s).

[0021] In embodiments, the method further comprises:obtaining one or more input audio streams corresponding to the one or more input video streams, respectively, andprocessing the one or more input audio streams to detect an audio event relating to the detected person,wherein the (characteristic) score for each of the generated plurality of video framings is further determined by applying a second scoring algorithm to the detected audio event.

[0022] In examples, audio information may be extracted from the audio stream to detect whether a person is speaking. Additionally, or alternatively, video information may be extracted from the video stream to detect whether a person is speaking. In examples, the video stream may be used to detect a person’s body pose and other gestures such as nodding, body language, raising hands, a person’s looking direction, etc. In examples, audio information may be extracted from the audio stream to detect an event, e.g., relating to a group of people such as laughter, applause, or general sentiment detection. Additionally, or alternatively, video information may be extracted from the video stream to detect the event.

[0023] In embodiments, the at least one first scoring algorithm comprises a plurality of first scoring algorithms and / or the at least one second scoring algorithm comprises a plurality of second scoring algorithms and / or the at least one third scoring algorithm comprises a plurality of third scoring algorithms. In embodiments, said adapting comprises applying a reinforcement learning algorithm to adapt at least one weight of the respective of at least one of the at least one first scoring algorithm and / or at least one of the at least one second scoring algorithm and / or at least one of the at least one third scoring algorithm. Additionally, or alternatively, said adapting comprises applying a reinforcement learning algorithm to adapt at least one scoring parameter of at least one of the at least one first scoring algorithm and / or at least one of the at least one second scoring algorithm and / or at least one of the at least one third scoring algorithm. For example, adapting hyperpara meters of a machine learning model, a cost matrix, initialization parameters, optimization criteria, threshold (e.g., the predetermined threshold or any other threshold relating to the Hungarian algorithm or any other similar algorithm).

[0024] The adapting of said determining of a (characteristic) score may involve adapting the scoring algorithm (e.g., at least one of the at least one first, second, third scoring algorithms). The adapting may therefore be understood as a reinforcement learning where the machine learning (ML) algorithm learns to make decisions by interacting with the (video and / or audio) stream(s), and receives feedback from the user (e.g., in the form of rewards) which causes the ML algorithm to develop one or more policies for selecting actions that would lead to the highest reward. Said policies may relate to more than one of the at least one first, second, third scoring algorithms, suchas adapting at least one weight of any of the more than one of the at least one first, second, third scoring algorithms. Thus, the ML algorithms can learn optimal behaviours for achieving its goals.

[0025] In embodiments, the at least one person comprises a plurality of persons and said detecting at least one set of person coordinate points comprises detecting a plurality of sets of person coordinate points, respectively. In embodiments, the output comprises, for the plurality of detected persons, a third video framing and a fourth video framing from among the plurality of video framings generated for the plurality of persons, wherein the third video framing is selected over the fourth video framing based on the respective (characteristic) scores of the third and fourth video framings.

[0026] In embodiments, the plurality of sets of person coordinate points may comprise a plurality of sets of body coordinate points, respectively, wherein said processing may further comprise:detecting the plurality of persons in the one or more input video streams, and detecting a plurality of person bounding boxes, a plurality of sets of face features for the detected plurality of persons, respectively,generating a plurality of triplets of person features relating to the detected plurality of persons, respectively, based on the detected plurality of person bounding boxes, plurality of sets of body coordinate points, and plurality of face features, respectively.

[0027] The plurality of persons may be identified based on the respective plurality of triplets of person features and wherein said at least one first scoring algorithm may be further applied to the respective plurality of triplets of person features. Even more preferably, said determining a respective score may be based on the generated respective plurality of triplets of person features. Thus, said determining a respective score for each of the generated plurality of video framings may be performed by applying the at least one first scoring algorithm to each of the generated respective plurality of video framings and to at least the detected respective plurality of sets of person coordinate points comprised in the respective plurality of triplets of person features, preferably applied to each of the generated respective plurality of video framings and to each of the respective plurality of triplets of person features.

[0028] The steps of detecting a plurality of person bounding boxes, a plurality of sets of face features for the detected plurality of persons, respectively, may comprise:detecting a plurality of faces, wherein said plurality of face bounding boxes is detected for the detected plurality of faces, respectively, wherein said plurality of sets of face features may comprise a plurality of sets of face coordinate points for the detected plurality of faces, respectively,

[0029] In preferred embodiments, the plurality of sets of person coordinate points may comprise a plurality of sets of body coordinate points, respectively, wherein said processing may further comprise:detecting the plurality of persons in the one or more input video streams. detecting a plurality of person bounding boxes for the detected plurality of persons, respectively,detecting a plurality of faces,detecting a plurality of face bounding boxes for the detected plurality of faces, respectively,detecting a plurality of sets of face features comprising a plurality of sets of face coordinate points for the detected plurality of faces, respectively,determining which of the plurality of detected person bounding boxes matches which of the detected plurality of sets of body coordinate points,determining which of the detected plurality of face features matches which of the detected plurality of sets of body coordinate points, andcombining the matched plurality of person bounding boxes, plurality of sets of body coordinate points, and plurality of face features to generate a plurality of triplets of person features relating to the detected plurality of persons, respectively.

[0030] Alternatively, the method comprises determining which of the detected plurality of face features matches which of the detected plurality of detected person bounding boxes, and determining which of the plurality of detected person bounding boxes matches which of the detected plurality of sets of body coordinate points.

[0031] More preferably, the plurality of persons may be identified based on the respective plurality of triplets of person features and wherein said at least one first scoring algorithm may be further applied to the respective plurality of triplets of person features. Even more preferably, said determining a respective score may be based on the generated respective plurality of triplets of person features. Thus, said determining a respective score for each of the generated plurality of video framings may be performed by applying the at least one first scoring algorithm to each of the generated respective plurality of video framings and to at least the detected respective plurality of sets of person coordinate points comprised in the respective plurality of triplets of person features, preferably applied to each of the generated respective plurality of video framings and to each of the respective plurality of triplets of person features.

[0032] By combining the result of the body-box matching (i.e., matching the detected person bounding boxes with detected plurality of sets of body coordinate points) and body-face matching (i.e., matching the detected plurality of face features matches with the detected plurality of sets of body coordinate points), one can combine each bounding box, body coordinate points and face features as belonging to a unique person in a video stream. Thus, said processing can be configured to identify a person in a video stream. The application of a neural network may be used for said identifying / detecting (i.e., person, person bounding box, sets of person coordinate points, body coordinate points, face, face bounding box, sets of face features, etc.). Furthermore, said scoring of the generated video frames can be more effectively performed based on further person features in addition to the coordinate points.

[0033] This may be particularly advantageous wherein the one or more input video streams comprise a plurality of input video streams, and wherein a plurality of persons are detected in any of the plurality of input video streams, preferably in more than one of the plurality of input video streams. For example, a first set of persons are detected in a first set of input video streams, a second set of persons are detected in a second set of input video streams, and at least one personin the first set and / or the second set of persons is detected in both the first and second set of input video streams. Thus, by identifying the plurality of persons in each input video stream, the at least one person detected in both the first and second set of input video streams would be identified as the same at least one person.

[0034] In embodiments, the method further comprises detecting a plurality of bodies and detecting a plurality of sets of body coordinate points of the respective detected bodies.

[0035] In embodiments, the method may further comprise generating at least one facial fingerprint vector for at least one of said detected at least one face, respectively, based on at least one of the detected plurality of sets of face coordinate points, wherein, for each of the one or more input video streams, at least one of the detected face features may further comprise at least one detected face bounding boxes and at least one facial fingerprint vector, respectively. In embodiments, said detecting plurality of sets of face coordinate points comprises cropping the respective detected plurality of faces based on the respective detected plurality of face bounding boxes, wherein said generating a plurality of facial fingerprint vectors comprises mapping the detected plurality of sets of face coordinate points to a respective at least one set of reference coordinate points to provide an aligned cropped face for each cropped detected face. Preferably, said detecting a plurality of sets of face coordinate points may comprise cropping the respective detected plurality of faces based on the respective detected plurality of face bounding boxes, wherein said generating at least one facial fingerprint vector may comprise mapping at least one of the detected plurality of sets of face coordinate points to a respective at least one set of reference coordinate points to obtain an aligned cropped face for each cropped detected face. In embodiments, said determining a respective score may be additionally or alternatively based on the generated respective plurality of triplets of person features. Thus, said determining a respective score for each of the generated plurality of video framings may be performed by applying the at least one first scoring algorithm to each of the generated respective plurality of video framings and to at least the detected respective plurality of sets of person coordinate points comprised in the respective plurality of triplets of person features, preferably applied to each of the generated respective plurality of video framings and to each of the respective plurality of triplets of person features.

[0036] In embodiments, the method further comprises, for each of the one or more input video streams:matching at least one element of at least one of the plurality of triplets of person features to a respective at least one stored element of at least one of a plurality of stored triplets of person features, andupdating the at least one of a plurality of stored triplets of person features to include the at least one of the plurality of triplets of person features, respectively.

[0037] If a triplet of person features is matched to a person whose features are stored (in memory), e.g., if the triplet is mapped to person already identified, the stored feature(s) is / are updated with the newly detected at least one person feature. This allows to keep the most up-to-date features of a person. Otherwise (i.e. , if not matched), the detected person features are analysed, such that if the triplet is not matched to any stored feature, it is assumed that the triplet corresponds to a personthat has not been identified yet (i.e., the corresponding person features are not yet stored and is the person is not yet tracked). Thus, a new person is created in the memory and the detected features (e.g., the triplet) are used to initialize the tracking of said person.

[0038] In embodiments, the method further comprises, for each of the one or more input video streams:matching at least one stored element of a first stored triplet of a plurality of stored triplets of person features to a respective at least one stored element of a second stored triplet of the plurality of stored triplets of person features,merging the first and second stored triplets of person features.

[0039] By combining tracking across time within each video stream and tracking across video streams, the method can uniquely identify and track one or more persons that appear in at least one, preferably more than one, of the one or more of the input video streams. By storing a person’s features (e.g., a triplet), preferably the person’s facial fingerprint, for a given time frame, the person can disappear from one or all of the one or more input video streams and reappear later and still be identified as the same person. In comparison to state-of-the-art geometric approaches that match objects in different streams by triangulation or other geometry-based approaches, the method according to the present invention, preferably using the facial features to match people across video streams, does not rely on knowledge of the exact pose of each stream's viewpoint. Therefore, no (camera) calibration is required to retrieve this information. Thus, this allows the user to change the location of any camera. Furthermore, the method allows to perform person tracking across different types of video streams. For example, it is possible to track the same person simultaneously within a network stream and a local camera stream.

[0040] In embodiments, said processing further comprises:detecting a respective mouth for each detected face based on the respective set of face coordinate points,detecting a respective mouth bounding box for the respective mouth,detecting a person speaking based on the detected respective mouth bounding box, wherein the (characteristic) score for each of the generated plurality of video framings is further determined by applying at least one third scoring algorithm to the detected respective mouth bounding box.

[0041] The detected mouth and mouth bounding box may be considered as being comprised in the video information extracted from the video stream to detect the event, such as detecting a person speaking. By detecting a person speaking, the method provides improved selection of a video framing where a framing showing the person speaking is ranked higher.

[0042] According to a second aspect, the invention provides a device comprising a processor and a memory comprising instructions which, when executed by the processor, cause the device to execute the method according to the first aspect of the present invention.

[0043] According to a third aspect, the invention provides embodiments relating to the first aspect, wherein each device comprised in a cluster of devices obtains a respective one or more input video stream,wherein said processing is performed by each device comprised in the cluster of devices, wherein said generating a respective plurality of video framings is performed by each device comprised in the cluster of devices,wherein said determining a respective (characteristic) score is performed by each device comprised in the cluster of devices,wherein said providing the output is performed by each device comprised in the cluster of devices, wherein the method comprises reaching, by the cluster of devices, a consensus on a ranking of the generated plurality of video framings to be displayed on a control display.

[0044] Preferably, the method may further comprise reaching, by the cluster of devices, a consensus on the generated plurality of triplets of person features relating to the detected plurality of persons.

[0045] Thus, the method allows a cluster of devices to at least process a plurality of input video streams, where each device at least processes one or more of the plurality of input video streams, thereby allowing fora larger number of input video streams to be at least processed simultaneously. By reaching a consensus on the ranking of the plurality of video framings, the method provides improved outputting of video framings, and thereby improved outputting on the control display and broadcasting of the video framing having the highest score. The consensus may be performed by a device comprised in the cluster of devices, which may obtain the respective output from each of the other of the devices comprised in the cluster of devices. Said device may additionally obtain at least one of: the determined scores, the generated plurality of video framings, the identified person(s), the set(s) of person coordinate points, and the plurality of triplets of person features, from each of the other devices. Thus, the consensus may be based on the outputs and / or the scores from the cluster of devices. The consensus may be alternatively performed by the cluster of devices, such as a majority vote, wherein each device comprised in the cluster of devices shares the output with the other devices comprised in the cluster of devices, wherein the consensus is based on the outputs and / or the scores from the cluster of devices.

[0046] According to a fourth aspect, the invention provides a cluster of devices comprising respective processors and respective memories comprising instructions which, when executed by the respective processors, cause the cluster of devices to execute the method according to the third aspect of the present invention.

[0047] According to a fifth aspect, the invention provides a system comprising:a device according to the second aspect of the present invention or a cluster of devices according to the fourth aspect of the present invention, anda graphical user interface rendered on a control display, preferably for receiving user input relating to feedback.

[0048] According to a sixth aspect, the invention provides computer program comprising instructions which, when the program is executed by a computer, cause the computer to carry out the method according to the first aspect of the present invention, or when the program is executedby a plurality of computers, cause the computers to carry out the method according to the third aspect of the present invention.

[0049] According to a seventh aspect, the invention provides a non-transitory computer readable medium containing a computer executable software which when executed on a device, performs the method according to the first aspect of the present invention, or when executed on a cluster of devices, performs the method according to the third aspect of the present invention.

[0050] Preferred embodiments and their advantages are provided in the description and the dependent claims.Brief description of the drawings

[0051] The present invention will be discussed in more detail below, with reference to the attached drawings.

[0052] Fig. 1 illustrates a system according to the invention.

[0053] Figs. 2a, 2b, 3a, 3b, 4a, 4b, 5 illustrate embodiments of the method according to the invention.

[0054] Fig. 6a, 6b, 6c illustrate a graphical user interface according to the invention.Description of embodiments

[0055] The following descriptions depict only example embodiments and are not considered limiting in scope. Any reference herein to the disclosure is not intended to restrict or limit the disclosure to exact features of any one or more of the exemplary embodiments disclosed in the present specification.

[0056] Furthermore, the terms first, second, third and the like in the description and in the claims are used for distinguishing between similar elements and not necessarily for describing a sequential or chronological order. The terms are interchangeable under appropriate circumstances and the embodiments of the invention can operate in other sequences than described or illustrated herein.

[0057] Furthermore, the various embodiments, although referred to as “preferred” are to be construed as exemplary manners in which the invention may be implemented rather than as limiting the scope of the invention.

[0058] The term “comprising”, used in the claims, should not be interpreted as being restricted to the elements or steps listed thereafter; it does not exclude other elements or steps. It needs to be interpreted as specifying the presence of the stated features, integers, steps or components as referred to, but does not preclude the presence or addition of one or more other features, integers, steps or components, or groups thereof. Thus, the scope of the expression “a device comprising A and B” should not be limited to devices consisting only of components A and B, rather with respect to the present invention, the only enumerated components of the device are A and B, and further the claim should be interpreted as including equivalents of those components.

[0059] In this document, the term “video framing” refers to an arrangement and / or composition of elements within a video frame or shot, i.e., how subjects and / or objects are positioned and organized within those frames or shots. A video framing may be generated by cropping out the field of view of a camera (i.e., the scope of what the camera captures within a shot). Additionally oralternatively, a video framing may be generated by performing at least one of panning, tilting, and optical zooming. Thus, a video framing may refer to a framing of a vide stream, thereby, referring to a framed video stream.

[0060] The term “video stream” refers to a (continuous) transmission of video data from a (local) camera or from another apparatus, such as a network camera or a server. The video data may be sent in a sequence of packets that are rendered in real-time.

[0061] Person coordinate points may include body coordinate points and / or face coordinate points, preferably both. Body coordinate points refer to specific locations on a person's body, such as joints (shoulders, elbows, knees, hips, etc.), within a video framing. These points may be used to track and analyze the movements and positions of different parts of the body. Face coordinate points refer to specific locations on a person's face, such as the eyes, nose, and mouth, within a video framing, but may also be any two distinct points on the face. These points may be used to track and analyze facial features and expressions.

[0062] The term “characteristic” refers to anything that can be (automatically) determined from or based on a video framing (and the person coordinate points and / or any of the person features within the video framing), such as overall quality of the video framing. Examples thereof include any one or combination of: quality of image (e.g., any one or combination of: resolution, sharpness, contrast, color accuracy, noise, dynamic range, compression, bit depth, etc.) and / or quality of video (e.g., any one or combination of: resolution, frame rates, bitrate, temporal artifacts and / or resolution, temporal consistency, objective metrics such as signal-to-noise ratio, SNR, etc.) and / or quality of audio (e.g., any one or combination of: frequency response, SNR, distortion, dynamic range, total harmonic distortion, bit depth, sample rate, frequency range, etc.), etc. In embodiments, the respective characteristic scores of the respective plurality of video framings are proportional to the extent that the respective plurality of video framings are of the relevant characteristic.

[0063] In embodiments, the processing of the one or more input video streams is performed by applying at least one detection algorithm, preferably performed by applying a neural network (NN), such as a convolutional NN (CNN), to each of the one or more input video streams. The NN may comprise at least one detection algorithm (e.g., NN), preferably a plurality of algorithms (e.g., NNs). The at least one detection algorithm or the NN may be applied to detect or determine at least one of: a person bounding box, person bounding box coordinates, position coordinates of body points, face bounding box coordinates, position coordinates of face points, a mouth bounding box, mouth bounding box coordinates, face fingerprint vector, audience sentiment state, audience applause state, person gesture state, person speaking state, etc. As described herein, the at least one detection algorithm or the NN may comprise at least one of: a person detector algorithm, a body detector algorithm, a face detector algorithm, a face point detector algorithm, a face aligner algorithm, a mount aligner algorithm, a sentiment detector algorithm, an applause detector algorithm, a gesture detector algorithm, and a speaking detector algorithm. Other detector algorithms relating to object detection may also be understood to be part of the present invention.

[0064] In embodiments, the face detector algorithm is applied to an input video stream to detect a face bounding box, preferably face bounding box coordinates of a detected face in the input videostream. In embodiments, a face point detector algorithm is applied to the face bounding box, preferably to the face bounding box coordinates, to crop the face out of the input video stream, processes said cropped out face, and provide face point coordinates (i.e ., position coordinates of face points). In embodiments, the face aligner algorithm is applied to at least two (distinct) face point coordinates and a corresponding at least two reference point coordinates to compute a best-fit similarity transformation which maps the detected face point coordinates to the corresponding reference point coordinates. Said transformation may preferably be applied to the cropped out face to transform the face pixels in a standardized way, such as according to standardized dimensions. A face encoder may be applied to the cropped out face, which may preferably be aligned according to the standardized way (i.e., mapped to reference point coordinates).

[0065] The term “control display” refers to a display screen which displays possible video framings which can be broadcast, and may be referred to as a video switcher or a vision mixer. In live video streaming, the control display may allow the operator to choose which input video stream is broadcast at any given moment. The method according to the present invention allows the operator to receive the output and which of the video framings to be broadcast, thereby enabling smooth transitions and dynamic live video streaming. The broadcasting of the video framing may relate to transmitting display signals for displaying the video framing on other one or more screens.

[0066] The term “display signals” may refer to signals sent to a display and / or a network for displaying one or more input video one or more video framings based on any one or more of the one or more input video streams. For example, the first display signals refer to signals comprising one or more of the generated plurality of video framings, preferably the output. The first display signals may comprise instructions relating to how the one or more of the generated plurality of video framings are displayed on the control display. The first display signals may be output in real-time, e.g., within at most 1 second from each other, preferably within at most 0.5 seconds, within at most 0.1 second, etc. In another example, the second display signals refer to signals comprising the video framing having the highest score. The second display signals or each signal comprised therein may comprise instructions to relay the video framing based on the respective input video steam, which has the highest score. Thereby, one signal may be sufficient to instruct the broadcasting of a video stream framed according to the video framing having the highest score. In embodiments, the second display signals comprise signals which are each output subsequently after a (predetermined) delay from one another, wherein each signal comprises the video framing having the highest score. In alternative embodiments, the second display signals are updated for broadcasting an updated video framing having the highest score within a delay of outputting the first display signals. For example, within a predetermined delay of at least 5 seconds, at least 10 seconds, at least 15 seconds, etc. This may allow the user to provide user input based on the output provided (e.g., the ranking). This may allow that the updated video framing to be broadcast is determined to have the highest score for said delay, ensuring that the updating of the second display signals would broadcast the video framing which is relevant.

[0067] In embodiments, said generating a respective plurality of video framings for each of the identified at least one person is based on the detected person features, preferably based on the atleast one set of coordinate points. The generated video framings may be selected from a set of predefined framings, e.g., close-up shot, medium shot, long shot, full shot, etc. of a person. Such framings may consist of a rectangular shape, while centering the person horizontally. Said generating a respective plurality of video framings may comprise performing vertical alignment such that there is a predefined space between the head of the identified person and the top of the framing. Said generating a respective plurality of video framings may additionally or alternatively, comprise performing bottom placement of the framing, preferably based on the type of shot. For example, for a close-up shot, the bottom is placed at the height of the person’s chest; For a medium shot, the bottom is placed at the height of the person’s hip; For a long shot: the bottom is placed at the height of the person’s knees; For a full shot, the bottom is placed just below the person’s feet. Depending on how the person is located in the video stream and which body points are visible, the framings may be generated accordingly.

[0068] In embodiments, said identification of at least one person in the (respective) one or more input video streams concerns identifying a person in each of the (respective) one or more input video streams, wherein the same person may be identified in at least two of the (respective) one or more input video streams.

[0069] In embodiments, said generated video framings may be used as reference for a motion smoother. For example, the motion smoother may internally use a PID filter to ensure that the framings selected for output have a smooth motion over time, mimicking the motion a cameraman would make while framing a person using a camera.

[0070] Based on the generated plurality of video framings in the one or more input video streams for at least one person, the method can determine which of the video framings is to be broadcast at which moment. This may be done by comparing the plurality of video framings with each other and with the current displayed video framing, particularly comparing the score thereof. For example, when a video framing has a score higher than that of the broadcast video framing, then said video framing will replace the broadcast video framing. The scoring comprises applying at least one first scoring algorithm to a video framing. Each of the at least one first scoring algorithm may a numerical score, such as on a numerical scale (e.g., 3-point, 5-point, 7-point, 10-point, etc., scales), or a rating, such as on a rating scale, e.g., one of: neutral (or a 0), upvote (or a +1), downvote (or a -1), or rejected (or not further considered) or on an ordinal scale (e.g., one of: very bad, bad, neutral, good, very good). A rating may be determined based on the determined score, e.g., based on a predetermined threshold, which can allow the scores to be normalized and comparable among more than one first scoring algorithm, and thereby allowing an overall score to be determined for a video framing.

[0071] The at least one first scoring algorithm may include any one or combination of: a framing quality scoring algorithm, a face visibility scoring algorithm, a face alignment scoring algorithm, a person speaking scoring algorithm, a framing variation scoring algorithm, person contextual information scoring algorithm, and global contextual information scoring algorithm. The framing quality scoring algorithm may comprise determining how well a (potentially moving) person is framed according to the imposed shot type, by computing, e.g., an intersection-over-union of thecurrent framing rectangle and the ideal framing rectangle, from the person body point coordinates. For example, the framing quality scoring algorithm calculates a score which can be used to determine a rating, e.g., if the score exceeds a predetermined framing quality threshold, the respective video framing is given a neutral rating, otherwise a downvote. The face visibility scoring algorithm may comprise determining how well a person’s face is visible in a video framing, by computing, e.g., the confidence of detection of the face, such as by applying neural-network for detecting the face. A moving-average of the face detection scores may be calculated. If a face is not detected in a video framing, the moving-average filter is fed with a score of 0. If the score exceeds a predetermined face visibility scoring threshold, the respective video framing is given a neutral rating, otherwise a downvote. The face alignment scoring algorithm may comprise determining how well a person is looking towards a camera, e.g., determining a percentage of alignment of the person’s face with the camera view. This may be performed by computing a best fit similarity transformation between the detected face points and a set of reference points that represent a perfect face alignment. A residual of said best fit (e.g., an average distance between each point and the transformed point) may be used to compute the metric. Said computing a best fit may be defined as (1 / exp(alpha * residual)), where a perfect alignment (i.e. , residual = 0) results in a score of +1 , while a bad alignment (i.e., residual -> inf, or residual larger than a predetermined threshold) results in a score of 0, and where alpha is for example equal to 10, however, other scores of alpha may be considered for controlling the influence of the residual on the overall fit, such a score larger than 10 for making the algorithm more sensitive to larger errors, or a score smaller than 10 for making the algorithm less sensitive to larger errors. If the score exceeds a predetermined face visibility scoring threshold, the respective video framing is given a neutral rating, otherwise a downvote. The person speaking scoring algorithm may comprise determining whether a person in a video framing is speaking. If a person is determined to be speaking, then the video framing gets an upvote, otherwise it gets a neutral score. The framing variation scoring algorithm may comprise at least one of a shot type variation scoring algorithm, a source variation scoring algorithm, and a person variation scoring algorithm. The shot type variation scoring algorithm may comprise detecting whether one or more video framings with the same shot type as the current broadcasted video framing. If detected, said one or more video framings get a downvote, if not, they get a neutral score. The source variation scoring algorithm comprises detecting whether one or more video framings belong to the same video stream as the current broadcasted video framing. If detected, said one or more video framings get a downvote, if not, they get a neutral score. The source variation scoring algorithm may further comprise detecting whether one or more framings belong to the same video stream as the current broadcasted video framing, which frame the same person, If detected, the one or more video framings get a rejected score. The person variation scoring algorithm may comprise detecting whether one or more video framings include the same person framed as the current broadcasted video framing. If detected, the one or more video framings get a downvote, if not, they get a neutral score.

[0072] In embodiments, the respective scores, preferably the ratings, of a plurality of first scoring algorithms are aggregated. Thus, a respective overall score for each generated video framing maybe determined. As a result, a ranking of the video framings can be determined. In embodiments, the respective score, preferably the rating, of the at least one first scoring algorithm may be aggregated with the respective score, preferably the rating, of the any one or combination of the at least one second scoring algorithm and the at least one third scoring algorithm to determine the overall score. Preferably, the ratings, particularly the values corresponding to the ratings, may be combined / aggregated by means of a weighted sum, where the weights express the importance of each first scoring algorithm. Said weights may each be set to a value of 1 , thereby providing an equal weight to each first scoring algorithm. At least one of said weights may be adapted, e.g., based on user feedback. For example, the user provides feedback through the GUI to favour or show preference of a particular video framing, such as by giving a “thumbs up” or “thumbs down”. This allows the reinforcement learning algorithm to incrementally adapt the respective weight of at least one of the (first, second and / or third) algorithms. In this example, if a video framing gets a ’’thumbs up” from the user, then the weight of at least one of the (first, second and / or third) algorithms that gave an upvote to the framing is increased by a predetermined value (e.g., by 25%, 50%, 75%, 100%, etc.), and the weight of at least one of the (first, second and / or third) algorithms that gave a downvote to the video framing is decreased by a predetermined value (e.g., by 25%, 50%, 75%, etc.); If a video framing gets a ’’thumbs down” from the user, then the weight of at least one of the (first, second and / or third) algorithms that gave a downvote to the video framing is increased by a given value (e.g., by 25%, 50%, 75%, 100%, etc.), and the weight of at least one of the (first, second and / or third) algorithms that gave an upvote to the video framing, is decreased by a given value (e.g., by 25%, 50%, 75%, etc.).

[0073] The method according to the present invention may automatically broadcast the video framing which it has determined to have the highest score. However, the method may provide the output as a suggestion to the user who decides when and which video framing is broadcasted. Preferably, the method may be a combination of the former and the latter, where the method automatically broadcasts while allowing the user to intervene.

[0074] In embodiments, the system may comprise the control display on which the GUI is rendered. This would allow the user to more easily provide user input, e.g., feedback, based on the displayed output, particularly the selected or ranked video framings. The user input may be a selection of a subset of the (selected or ranked) video framings. The user input may be a user rating of the (selected or ranked) video framings. The user input may be a user preference of the (selected or ranked) video framings.

[0075] In embodiments, the system may comprise one or more cameras from which the one or more input video streams are obtained by the device, respectively. Alternatively, the system may comprise a (cloud) server from which the one or more input video stream are obtained by the device. The systems may however comprise one or more cameras and a (cloud) server from which the one or more input video stream are obtained by the device.

[0076] In embodiments, the device may comprise the graphical user interface (GUI) for receiving a user input from the user, such as feedback and / or prompt. The GUI may be external to the device and part of the system comprising said device.

[0077] In embodiments, the non-transient computer readable medium may include an electrical connection having one or more wires, a portable computer diskette, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing. In the context of this document, a non-transient computer readable medium may be any tangible medium that can contain, or store a program for use by or in connection with an instruction execution system, apparatus, device or module.

[0078] Example embodiments of the invention will be described with reference to Figs. 1 , 2a, 2b, 3a, 3b, 4a, 4b, 5, 6a-6c, which are not intended to limit the scope of the invention in any way.

[0079] Example 1 : example system according to the invention

[0080] This example is described with reference to Fig. 1. In this example according to embodiments of the invention, the system comprises at least a device 110 according to the present invention and a GUI 120, such as being comprised in a control display. In this example, the GUI 120 is external to the device 110. However, in an alternative example embodiment, the device 110 comprises the GUI 120.

[0081] As may be seen in Fig. 1 , the device 110 is configured to obtain a first input video stream 111 from a first camera 101 , a second input video stream 112 from a second camera 102 and a third video stream 113 from a network 130, such as a cloud server. Furthermore, the device 110 is configured to process the first to third input video streams 111-113 to identify a first person 104, a second person 105 and a third person 106 in said first to third input video streams 111-113. As shown in Fig. 1 , the first person 104 is identified in the first and second input video streams H I-112, the second person 105 is identified in the second input video stream 112, and the third person 106 is identified in the third input video stream 113.

[0082] The control display on which the GUI 120 is rendered displays a plurality of video framings 121 generated by the device 110, which may be provided in a decreasing order of scores (e.g., decreasing from left to right and from top to bottom) determined by the device 110.

[0083] The device 110 is further configured to output display signals for broadcasting the video framing 103 being ranked highest or having the highest score. In this example, it can be seen that in the current moment, the video framing 103 being broadcast displaying the second person 105 is not ranked as the video framing with the highest score by the device 110, i.e., as shown in the GUI 120. Therefore, it is expected that the broadcasting of the video framing 103 is to be updated to display the first person, e.g., within a predetermined delay.

[0084] Note that this example embodiment of the device may be configured to perform any of the steps described herein, such as in the below examples.

[0085] Example 2: first example method according to the invention

[0086] This example is described with reference to Fig. 2b which shows the method comprising the steps of obtaining a video stream 21 which is processed by applying a person detector algorithm 201 to detect a person bounding box of a detected person, particularly person bounding box coordinates 23 of said detected person. Fig. 2b further shows that the video stream 21 is processed by applying a body detector algorithm 202 to detect position coordinates 24 of body points of saiddetected person. Fig. 2b further shows that the video stream 21 is processed by applying a face detector algorithm 203 to detect a face bounding box of said detected person, particularly face bounding box coordinates 210, 25 of said detected person. Optionally, the video stream 21 is processed by applying a sentiment detector algorithm 204 to detect a sentiment sate of the audience 233, which is an example of determining a contextual global sentiment based on contextual global information. Furthermore, the video stream is processed by an applause detector algorithm 205 to detect an applause state 234 of the audience. Further optionally, a gesture detector algorithm 213 is applied to the detected position coordinates 24 of body points to detect a gesture state 235 of said detected person.

[0087] A face point detector algorithm 211 is applied to the detected face bounding box coordinates 210 to detect position coordinate of face points 212, 26. Optionally, a face aligner algorithm 221 is applied to the detected position coordinates of face points 212 to provide an aligned cropped face 223 which is then encoded by a face encoder 231 to provide a face fingerprint vector 27. Further optionally, a mouth aligning algorithm 222 is applied to the detected position coordinate of face points 212 to provide a mouth bounding box, preferably mouth bounding box coordinates 224, to which a speaking detector algorithm 232 is applied to detect a speaking state of a person 236.

[0088] The inventors have shown that the processing of the input video stream 21 is sufficient to detect audience sentiment and / or applause states and / or a person’s gesture and / or speaking states. However, the inventors have shown that the processing of an input audio stream 22 would complement the detection of the audience applause state 234 and / or the person’s speaking state 236. Thus, the input audio stream 22 (corresponding to the input video stream 21) may be obtained and processed by the applause detector algorithm 205 to detect the applause state 234 of the audience. Thus, the applause detector algorithm 205 may process both the input video stream 21 and the corresponding input audio stream 22 to better detect the applause state 234 of the audience. Furthermore, the audio stream 22 may be processed by the speaking detector algorithm 232 to detect the speaking state 236 of said person. Thus, the speaking detector algorithm 232 may process both the input video stream 21 and the corresponding input audio stream 22 to better detect the speaking state 236 of the detected person.

[0089] Example 3: second example method according to the invention

[0090] This example is described with reference to Fig. 2a which shows the method comprising the steps of applying a body-box matching algorithm 241 to the person bounding box coordinates 23 and the position coordinates of body points 24 for determining which of the plurality of detected person bounding boxes matches which of the detected plurality of sets of body coordinate points, and applying a body-face matcher algorithm 242 to the position coordinates of body points 24 and to at least one of the following face features: the face bounding box coordinates 25, the position coordinates of face points 26 and the face fingerprint vector 27, for determining which of the detected plurality of face features matches which of the detected plurality of sets of body coordinate points.

[0091] The body-box matcher algorithm 241 may, given a set of person bounding boxes and a set of person body points, determine which box corresponds to which body points. The body-box matcher algorithm 241 may then compute a matching score for each possible pair of box and body points. The matching score may be defined as the intersecting area between the box and the bounding box of body points, divided by the area of the box. This score lies between 0 (no match) and 1 (perfect match). Here, the Hungarian algorithm is used to find matching pairs of boxes and bodies based on the matching scores. Only matches with a score above a given threshold are considered valid.

[0092] The body-face matcher algorithm 242 may, given a set of face features and a set of person body points, determines which body corresponds to which face. Face features may comprise the face bounding box 25, face points 26, and the face fingerprint vector 27. As the facial features may be extracted from the same face crop from the input video stream, they can be more easily grouped together. To match face features with body points, the face points are used which include, e.g., the location of the left eye, the right eye and the nose. Those points may also be included in the body point set. A matching cost is computed for each possible set of the face points and each possible subset of body points. Here, the matching cost is defined as the average distance of each of the corresponding points between the two sets. Here, the Hungarian algorithm is used to find matching pairs of faces and bodies based on the matching costs. Only matches with a cost below a given threshold are considered valid. Thus, the matched person features 240 are combined to generate a triplet of person features relating to the detected person.

[0093] By combining the result of the body-box matching and body-face matching, one can combine each box, body and face features as belonging to a unique person in a video stream.

[0094] It will be understood that any one or combination of the person bounding box coordinates 23, the face bounding box 25, face points 26, and the face fingerprint vector 27 may be provided as described in Example 2, but is not limited thereto, e.g., may be provided as part of metadata in the input video stream(s).

[0095] Example 4: third example method according to the invention

[0096] This example is described with reference to Fig. 2c which shows the method comprising the step of combining detected person features across the different video streams and matched to unique persons for which the respective features are stored and tracked in memory. It will be understood that the detected person features may be provided as described in Example 3, but is not limited thereto, e.g., may be provided as part of metadata in each input video stream.

[0097] The method comprises a matching algorithm which matches detected person features with stored person features of a respective person. Said matching algorithm 270 may comprise a cascade of matching components. If person features of a detected person are matched to stored person features by a matching component, at least one of the stored person features 260 which are matched are updated with the newly detected features. Otherwise, the detected person features are analysed by the subsequent matching component. If detected person features are not matched by any matching component, it is assumed that the detected person features relate to a person thatis not stored and tracked yet. A new person is then created in storage and the detected person features are used to initialize its trackers.

[0098] The matching components comprise face fingerprint matching 273, bounding box matching 271 , and body point coordinates matching 272.

[0099] The face fingerprint matching 273 comprises computing a face fingerprint matching score for each possible pair of detected persons and stored persons. Two face fingerprints are compared, e.g., by computing 0.5 * (1 + cosine distance between the two fingerprint vectors). This is a value between 0 (no face resemblance) and 1 (maximum face resemblance). The face fingerprint matching score between a detected and stored person is defined as the maximum score over all streams in which both are detected. Here, the Hungarian algorithm is used to find matching pairs between detected and stored persons using the face fingerprint matching scores. Only matches with a score above a given threshold are considered valid.

[0100] The bounding box matching 271 comprises computing a bounding box matching score, within each video stream, for each possible pair of detected persons and stored persons. Two boxes are compared, e.g., by computing the intersection-over-union (intersecting area of boxes divided by the union area of boxes). This is a value between 0 (no box overlap) and 1 (boxes overlap completely). Here, the Hungarian algorithm is used to find matching pairs between detected and stored persons using the bounding box matching scores. Only matches with a score above a given threshold are considered valid.

[0101] The body point coordinates matching 272 comprises computing a body matching score within each video stream, for each possible pair of detected persons and stored persons. Two sets of body points are compared, e.g., by computing the distance between each pair of corresponding points. The number of points for which this distance lies below a given maximum threshold, divided by the total number of points, results in a score between 0 and 1. Here, the Hungarian algorithm is used to find matching pairs between detected and stored persons using the body matching scores. Only matches with a score above a given threshold are considered valid.

[0102] When detected person features are matched with stored person features 260 of a stored person, the latter is updated with the detected person features. The detected features are processed by person feature trackers 250 which filter potential noisy detections over time. The person feature trackers comprise a box tracker 251 , a body tracker 252 and a face fingerprint tracker 253.

[0103] The box tracker 251 tracks center position coordinates and the aspect ratio and the height of the box, e.g., by applying a linear Kalman Filter. As such, a constant-velocity model may be applied.

[0104] The body tracker 252 may comprise sub-trackers for each body point; each sub-tracker implementing a state machine representing the state of a point: idle, active, inactive, terminated. The state traverses from idle to active when a point has seen enough consecutive detection updates. The state traverses from active to inactive when a point has missed a detection update. The state traverses from inactive to active when a point gets again a detection update. The state traverses from inactive to terminated when a point missed detection updates for a certain amountof time. The point is then removed from memory. A point is updated when it is detected and its relative position to a reference point has not changed too much with respect to the previous detection. As relative position, the nose point is used as it is placed in the center of a person's face for the application of person framing.

[0105] The face fingerprint tracker 253 keeps track of the first detected fingerprint vector of a face as well as a moving-average of the fingerprint vectors over time. In the face fingerprint matching 273 component, both are compared to the detected fingerprint vector. The best matching score is taken into account.

[0106] Fig. 2c shows the method further comprising the step of merging 280, which comprises matching at least one stored element of a first stored triplet of a plurality of stored triplets of person features to a respective at least one stored element of a second stored triplet of the plurality of stored triplets of person features. The matching by face may be similar to the face fingerprint matching 273 component, in which pairs of stored persons are evaluated to see if they match by face fingerprint vector. Additionally, or alternatively, the matching by bounding box, which is similar to the bounding box matching 271 component, evaluates pairs of stored persons to see if they match by bounding box in certain video streams. When pairs of stored persons match, they are merged.

[0107] By combining tracking across time within each video stream and tracking across video streams, the system can uniquely identify and track people that appear in one or more of the video streams. By storing a person’s facial fingerprints for a given time frame, the person can disappear from one or all video streams and reappear later and still be identified as the same person.

[0108] When a person is weakly detected, e.g., because of lighting conditions or possible occlusions, it can happen that the feature matching fails. The result is that a new person is added to storage which in reality resembles or is similar to an already stored person. However, when time evolves, detections can improve again which allows the method to detect from the tracked person features that two persons resemble too much and should be merged.Example 5: fourth example method according to the invention

[0109] This example is described with reference to Fig. 3a and 3b, which show the method comprising generating a plurality of video framings by a framing generator 310, and determining a respective score 330 therefor and providing the output by a framing ranker 320.

[0110] Fig. 3a shows the tracked person features 255 in each video stream are used to generate a set of possible shot framings. More specifically, the body point coordinates are used to compute a set of predefined framings (close-up shot, medium shot, long shot, full shot) of a person. Such video framings may comprise a rectangular shape, centering the person horizontally. The framing may be vertically aligned such that there is a predefined space between the head of the identified / detected person and the top of the framing. The bottom placement of the framing depends on the type of shot. For example, in a close-up shot, by a close-up generator 311 , the bottom is placed at the height of the person’s chest; in a medium shot, by a medium shot generator 312, the bottom is placed at the height of the person’s hip; in a long shot, by a long shot generator 313, the bottom is placed at the height of the person’s knees; in a full shot, by a full shot generator 314, thebottom is placed just below the person’s feet. Thus, depending on how the person is located in the video stream and which body points are visible, one or more of those framings can be generated. Said generated video framings are used as reference for a (respective) motion smoother 315.

[0111] It will be understood that the tracked person features 255 may be provided as described in Example 4, but is not limited thereto, e.g., they may be provided as part of metadata in the input video stream(s).

[0112] Fig. 3b shows a plurality of scoring algorithms which score the generated video framings, e.g., as described with reference to Fig. 3a. As shown in Fig. 3b, the plurality of scoring algorithms comprise a framing quality rater 321 , a person visibility rater 322, a gazing direction rater 323, a person speaking rater 324, and framing variation rater 325, etc., each of which determining a score for the generated respective plurality of video framings. The scores of each rater are aggregated to determine an overall score of each respective video framing, e.g., by means of a weighted sum, where a predetermined weight is attributed to a respective rater.

[0113] In this example, the output is provided comprising a ranking of the generated video framings based on the respective overall score thereof.

[0114] Example 6: fifth example method according to the invention

[0115] This example is described with reference to Fig. 4a and 4b, each of which shows the method comprising the step of applying a video framing.

[0116] Fig. 4a shows broadcasting a desired video framing (e.g., having the highest score), such as applying or bringing to action, by cropping out a framing rectangle by a cropper 411 from its respective input video stream. The crop is upscaled by an upscaler 412 to the desired resolution. This results in a video framing of the respective input video stream that is displayed / broadcast to the desired destination.

[0117] Alternatively, Fig. 4b shows the input video stream being obtained from a camera which has mechanical pan / tilt / zoom control capabilities, e.g., a PTZ camera. The method can send control signals to the camera controller 422, such that the framing is set up by moving the camera (e.g. by panning and tilting it) and by optical zooming. Therefore, a feedback PID controller 421 compares said desired video framing with the current video stream that should represent the framing. Possible errors may be measured and the PID controller generates speed commands for pan, tilt and zoom that reduce those errors. Those commands are applied to the mechanical PTZ camera.

[0118] It will be understood that said desired video framing may be provided as the video framing having the highest score in the ranking provided as described in Example 5 but is not limited thereto.Example 7: sixth example method according to the invention

[0119] This example is described with reference to Fig. 5 which shows the method used as a smart and teachable assistant.

[0120] The operator can provide user input, where the user input is in the form of a user intervention during a fully automated mode of the method or in the form of a manual user selection 501 of a video framing based on the provided ranking (as a recommendation).

[0121] The operator can provide user input in the form of user feedback to the output, e.g., to favour a certain video framing, e.g. by giving a “thumbs up” or “thumbs down” (see ref. 125 in Fig.1). This allows the method to incrementally shape the weights of each rater and therefore the importance of each frame rater criterion. Any of the weights of the raters may be adapted by a reinforced weights learner 502.

[0122] The method further comprises providing an LLM-based agent 503 which receives a user input prompt from the operator. The user input prompt may be an instruction to the LLM to select a video framing (i.e., corresponding to the manual user selection) or an instruction for generating a video framing according to user-defined criteria. The LLM agent 503 may perform the video framing selection 501 and / or may communicate with another LLM-based agent 513. Said other LLM-based agent 513 may, e.g., generate a new video framing according to the user prompt or to collaborate with other smart systems, such as a building automation system where the lighting is controlled by the other LLM-based agent 513.

[0123] It will be understood that the video framing ranking may be provided as described in Example 6 but is not limited thereto.Example 8: example GUI according to the invention

[0124] This example is described with reference to Fig. 6a-6c which show an example GUI 620 rendered on the control display.

[0125] Fig. 6a shows the GUI 620 at a first time point, where the output comprising a ranking of a first video framing 611 , a second video framing 612, a third video framing 613, and a fourth video framing 614, wherein in the ranked video framings four persons are identified 631. In the first video framings 611 a first person is framed, in the second video framings a second person is framed, in the third video framing 613 the second person and a third person are framed, and in the fourth video framing 614 the first, second and third persons are framed.

[0126] The GUI 620 comprises a first display portion 640 which shows a video framing being broadcast and a second display portion 641 which shows a video framing to be subsequently broadcast. The first display portion 640 may be labelled with a first label 649, shown as PGM or program, and the second display portion 641 may be labelled with a second label 647, shown as PVW or preview.

[0127] In this example, the GUI allows the operator to provide user input, in the form of a manual user selection, such as dragging and dropping at least one of the ranked video framings, in a third display portion in the GUI. This user selection may be seen as selecting a subset of the output of the method, i.e., a subset of the ranking. In Fig. 6a, the user input relates to a user selection of another video framing which is now displayed in a first location 621 in the third display portion.

[0128] Fig. 6b shows the GUI 620 at a second time point, where the user selection included selecting the first video framing 611 shown in Fig. 6a and displaying it in a second location 622 in the third portion, selecting the first video framing 611 shown in Fig. 6b in the ranking and displaying it in a third location 623 in the third portion, selecting the second video framing 612 shown in Fig.6b and displaying it in a fourth location 624 in the third portion, selecting the second video framing 612 shown in Fig. 6a in the ranking and displaying it in a fifth location 625 in the third portion, selecting the third video framing 613 shown in Fig. 6a in the ranking and displaying it in a sixth location in the third portion, etc.

[0129] The user input may also be an indication 618 of a user preference for a particular video framing. In this example, the indication 618 of user preference is provided on the second video framing as a star, as shown in Fig. 6b. This indication 618 of user preference may also be indicated 628 in the respective video framing in the third portion of the GUI.

[0130] Fig. 6c shows the GUI 620 at a third time point, where the second video framing 612, shown in Fig. 6a being displayed in the ranking and shown in Fig. 6b and 6c being displayed in the fifth portion 625, is displayed in the first portion 640 of the GUI 620, i.e., said video framing is being broadcast.

[0131] The GUI 620 further provides a label 629 in the third portion which indicates the video framing being broadcast, wherein said label 629 is equivalent to the first label 649, and another label 627 in the third portion which indicates the video framing to be subsequently broadcast, wherein said other label 627 is equivalent to the second label 647.

[0132] (End of example 8)

[0133] In the above, several examples are given wherein the device executes the method according to the present invention. However, the examples may involve a cluster of devices each of which executes the method according to the present invention, preferably, wherein the cluster of devices reach a consensus on at least one of the generated triplets of person features and of the ranking of the plurality of video framings.

Claims

Claims1. A computer-implemented method for providing an output of video framings, the method comprising:obtaining one or more input video streams (21),processing the one or more input video streams, preferably by applying a neural network to each of the one or more input video streams, to identify at least one person in the one or more input video streams and to detect at least one set of person coordinate points (24; 25) of the at least one identified person,generating a respective plurality of video framings (311-314) for each of the identified at least one person,determining a respective score (330) for each of the generated respective plurality of video framings by applying at least one first scoring algorithm (321-325) to each of the generated respective plurality of video framings and to body coordinate points comprised in the detected respective at least one set of person coordinate points within the respective plurality of video framings, andproviding the output comprising a first video framing and a second video framing from among the respective plurality of video framings, wherein the first video framing is selected over the second video framing based on the respective scores of the first and second video framings,wherein the at least one person comprises a plurality of persons and said detecting at least one set of person coordinate points (24:25) comprises detecting a plurality of sets of person coordinate points, respectively,wherein the plurality of sets of person coordinate points may comprise a plurality of sets of body coordinate points, respectively,wherein said processing further comprises:detecting the plurality of persons (201) in the one or more input video streams, and detecting a plurality of person bounding boxes (23), a plurality of sets of face features for the detected plurality of persons (201), respectively, andgenerating a plurality of triplets of person features relating to the detected plurality of persons, respectively, based on the detected plurality of person bounding boxes, the plurality of sets of body coordinate points, and the plurality of face features, respectively.

2. The method according to claim 1 , further comprising:outputting first display signals for displaying the output on a control display, and outputting second display signals for broadcasting the video framing having the highest score.

3. The method according to claim 2, wherein said transmitting second display signals comprises cropping out (411) the video framing having the highest score and upscaling (412) the cropped video framing to be broadcasted.

4. The method according to claim 2 or claim 3, wherein said transmitting second display signals comprises generating control signals for controlling (421 ; 422) a camera to perform at least one of panning, tilting, and optical zooming for capturing the video framing having the highest score.

5. The method according to any of claims 1-4, further comprising:providing a graphical user interface for receiving user feedback from a user with respect to a relevance of the provided output, andadapting said determining of a score based on the received user feedback.

6. The method according to claim 5, wherein the user feedback comprises one or more user selections with respect to one or more video framings from the plurality of video framings for the detected person to be continuously displayed on the control display and / or to be directly broadcasted.

7. The method according to any of claims 1-6, further comprising:providing a large language model-based agent (503),providing a graphical user interface (110) for receiving a user prompt from a user to communicate with the large language model-based agent,adapting said determining of a score and / or adapting the output based on the received user prompt.

8. The method according to any of claims 1-7, further comprising:obtaining one or more input audio streams corresponding to the one or more input video streams, respectively, andprocessing the one or more input audio streams to detect an audio event relating to the detected person,wherein the score for each of the generated plurality of video framings is further determined by applying a second scoring algorithm to the detected audio event.

9. The method according to any of claims 5-8, wherein the at least one first scoring algorithm comprises a plurality of first scoring algorithms and preferably the at least one second scoring algorithm,wherein said adapting comprises applying a reinforcement learning algorithm (502) to adapt at least one weight of at least one of the plurality of first scoring algorithms and / or preferably at least one of the at least one second scoring algorithm, respectively.

10. The method according to any of claims 1-9,wherein the output comprises, for the plurality of identified persons, a third video framing and a fourth video framing from among the plurality of video framings generated for the plurality of persons,wherein the third video framing is selected over the fourth video framing based on the scores of the third and fourth video framings.

11. The method according to any of claims 1-10, wherein said processing further comprises:detecting a plurality of faces (203), detecting a plurality of face bounding boxes (25) for the detected plurality of faces, respectively, wherein the plurality of sets of face features comprises a plurality of sets of face coordinate points for the detected plurality of faces (26), respectively,determining which of the plurality of detected person bounding boxes matches which of the detected plurality of sets of body coordinate points,determining which of the detected plurality of face features matches which of the detected plurality of sets of body coordinate points, andcombining the matched plurality of person bounding boxes, plurality of sets of body coordinate points, and plurality of face features to generate the plurality of triplets of person features relating to the detected plurality of persons, respectively, preferably wherein the plurality of persons is identified based on the respective plurality of triplets of person features and wherein said at least one first scoring algorithm is further applied to the respective plurality of triplets of person features.

12. The method according to claim 11 , further comprising generating a plurality of facial fingerprint vectors for said detected plurality of faces, respectively, based on the detected plurality of sets of face coordinate points,wherein, for each of the one or more input video streams, the detected plurality of face features further comprises a detected plurality of face bounding boxes and a plurality of facial fingerprint vectors, respectively.

13. The method according to claim 12, wherein said detecting plurality of sets of face coordinate points comprises cropping the respective detected plurality of faces based on the respective detected plurality of face bounding boxes,wherein said generating a plurality of facial fingerprint vectors (27) comprises mapping the detected plurality of sets of face coordinate points to a respective at least one set of reference coordinate points to obtain an aligned cropped face for each cropped detected face.

14. The method according to any of claims 11-13, further comprising, for each of the one or more input video streams:matching (271-273) at least one element of at least one of the plurality of triplets of person features to a respective at least one stored element of at least one of a plurality of stored triplets of person features, andupdating the at least one of a plurality of stored triplets of person features to include the at least one of the plurality of triplets of person features, respectively.

15. The method according to any of claims 11-14, further comprising, for each of the one or more input video streams:matching (271-273) at least one stored element of a first stored triplet of a plurality of stored triplets of person features to a respective at least one stored element of a second stored triplet of the plurality of stored triplets of person features, merging (280) the first and second stored triplets of person features.

16. The method according to any of claims 11-15, wherein said processing further comprises:detecting a respective mouth (222) for each detected face based on the respective set of face coordinate points (212),detecting a respective mouth bounding box (224) for the respective mouth, detecting a person speaking (236) based on the detected respective mouth bounding box,wherein the score for each of the generated plurality of video framings is further determined by applying at least one third scoring algorithm to the detected respective mouth bounding box.

17. A device (110) comprising a processor, and a memory comprising instructions which, when executed by the processor, cause the device to execute the method according to any of claims 1-16.

18. The method according to any of claims 1-16, wherein each device comprised in a cluster of devices obtains a respective one or more input video stream,wherein said processing is performed by each device comprised in the cluster of devices, wherein said generating a respective plurality of video framings is performed by each device comprised in the cluster of devices,wherein said determining a respective score is performed by each device comprised in the cluster of devices,wherein said providing the output is performed by each device comprised in the cluster of devices,wherein the method comprises reaching, by the cluster of devices, a consensus on a ranking of the generated plurality of video framings to be displayed on a control display, based on the output and / or the scores from each of cluster of devices.

19. The method according to claim 18, further comprising reaching, by the cluster of devices, a consensus on the generated plurality of triplets of person features relating to the detected plurality of persons.

20. A cluster of devices each of which comprising a respective processor and a respective memory comprising instructions which, when executed by the respective processor, cause the respective device to execute the method according to claim 18 or 19.

21. A system comprising:a device (110) according to claim 17 or a cluster of devices according to claim 20, and a graphical user interface (120) rendered on a control display, preferably for receiving user input relating to feedback.

22. A computer program comprising instructions which, when the program is executed by a computer, cause the computer to carry out the method according to any of claims 1-16 or when the program is executed by a plurality of computers, cause the computers to carry out the method according to claim 18 or 19.

23. A non-transitory computer readable medium containing a computer executable software which when executed on a device, performs the method according to any of claims 1-16 or when executed on a cluster of devices, performs the method according to claim 18 or 19.