Video scene retrieval with importance ratio

Abstract

A method for video scene retrieval based on relative importance ratio is disclosed. The method includes receiving a set of input video frames and an input textual query. A total budget associated with retrieval of a set of video scenes is received. The method further includes splitting the set of input video frames into a set of semantic chunks. A relative importance ratio associated with the set of semantic chunks is determined, based on correspondence between the input textual query and the set of semantic chunks. A first budget associated with each chunk is determined, based on the total budget and the relative importance ratio. The set of input video frames is condensed to determine the set of video scenes based on the first budget. A large multi-modal (LMM) model is applied on the set of video scenes and the input textual query to determine an output, which is rendered.

Description

[0001] VIDEO SCENE RETRIEVAL WITH IMPORTANCE RATIO

[0002] FIELD

[0003] The embodiments discussed in the present disclosure are related to retrieval of important image frames from a video scene based on relative importance ratio.

[0004] BACKGROUND

[0005] With advancements in internet and multimedia technology, there has been a proliferation of number and size of videos. In recent years, there has been a considerable surge in a pervasive issue of video scene retrieval for various application areas, such as, video indexing and archiving. Typically, the entire video may be condensed independent of a user query to generate condensed frames. The condensed frames and the user query may be then fed into a machine learning (ML) model to generate an answer associated with the content of the video. However, this technique may lead to degradation in the performance and accuracy of the ML model due to removal of important video frames related to the user query, thereby further leading to generation of incorrect answers associated with the content of the video. Other traditional techniques may entail a generation of visual embeddings and feeding of the visual embeddings into the ML model to generate the answer for the user query. But, a major drawback of this technique is that unimportant parts of the video may be overestimated, or important parts of the video may be lost.

[0006] The subject matter claimed in the present disclosure is not limited to embodiments that solve any disadvantages or that operate only in environments such as those described above. Rather, this background is only provided to illustrate one example technology area where some embodiments described in the present disclosure may be practiced.SUMMARY

[0007] According to an aspect of an embodiment, a method may include a set of operations which may include receiving a set of input video image frames and an input textual query. The set of operations may further include receiving a total budget associated with retrieval of set of video scenes. The set of operations may further include splitting the set of input video frames into a plurality of semantic chunks. The set of operations may further include determining a relative importance ratio associated with each chunk of the set of semantic chunks, based on correspondence between the input textual query and the set of semantic chunks. The set of operations may further include determining a first budget associated with each chunk of the set of semantic chunks, based on the total budget and the relative importance ratio. The set of operations may further include condensing the set of input video frames to determine the set of video scenes based on the first budget. The set of operations may further include applying a large multi-modal (LMM) model on the set of video scenes and the input textual query. The set of operations may further include determining an output associated with the input textual query based on the application of the LMM model. The set of operations may further include controlling rendering of the output associated with the input textual query.

[0008] The objects and advantages of the embodiments will be realized and achieved at least by the elements, features, and combinations particularly pointed out in the claims.

[0009] Both the foregoing general description and the following detailed description are given as examples and are explanatory and are not restrictive of the invention, as claimed.BRIEF DESCRIPTION OF THE DRAWINGS

[0010] Example embodiments will be described and explained with additional specificity and detail through the use of the accompanying drawings in which:

[0011] FIG. 1 is a diagram representing an example network environment related to video scene retrieval based on a relative importance ratio;

[0012] FIG. 2 is a block diagram that illustrates an exemplary electronic device of FIG.

[0013] 1 for video scene retrieval based on a relative importance ratio;

[0014] FIG. 3 is a diagram that illustrates an exemplary execution pipeline for video scene retrieval based on a relative importance ratio;

[0015] FIG. 4 is a diagram that illustrates an exemplary execution pipeline for video scene retrieval based on a relative importance ratio;

[0016] FIG. 5 is a diagram that illustrates an exemplary execution pipeline for determination of a set of video scenes based on a relative importance ratio and a first budget;

[0017] FIG. 6 is a diagram that illustrates an exemplary execution pipeline for generation of condensed video frames based on a relative importance ratio and a first budget;

[0018] FIG. 7A is a diagram that illustrates an exemplary execution pipeline for generation of a set of semantic chunks from a set of input video frames based on a queryagnostic semantic chunking technique;

[0019] FIG. 7B is a diagram that illustrates an exemplary execution pipeline for generation of a set of captions associated with a set of input video frames based on a query-aware semantic chunking technique;FIG. 7C is a diagram that illustrates an exemplary execution pipeline for determination of a relative importance ratio based on a frame-based importance ratio determination technique; and

[0020] FIG.8 is a diagram that illustrates a flowchart of an exemplary method for video scene retrieval based on a relative importance ratio,

[0021] all according to at least one embodiment described in the present disclosure. DESCRIPTION OF EMBODIMENTS

[0022] Some embodiments described in the present disclosure may relate to methods and electronic devices for video scene retrieval based on a relative importance ratio. In the present disclosure, a set of input video frames and an input textual query may be received. A total budget associated with a retrieval of a set of video scenes may be received. The set of input video frames may be split into a set of semantic chunks. The relative importance ratio may be determined based on correspondence between the input textual query and the set of semantic chunks. The relative importance ratio may be associated with each chunk of the set of semantic chunks. A first budget may be determined based on the total budget and the relative importance ratio. The first budget may be associated with each chunk of the set of semantic chunks. The set of input video frames may be condensed to determine the set of video scenes based on the first budget. A large multi-modal (LLM) model may be applied on the set of video scenes and the input textual query. An output may be determined based on the application of the LLM model. The output may be associated with the input textual query. Rendering of the output associated with the input textual query, may be controlled.

[0023] The technological field of video scene retrieval maybe improved by configuring an electronic device to retrieve video scenes based on semantic chunking technique usingrelative importance ratios. The electronic device may receive a set of input video frames and an input textual query. The electronic device may receive a total budget associated with the retrieval of a set of video scenes from a set of input video frames. The electronic device may split the set of input video frames into a set of semantic chunks. The electronic device may determine a relative importance ratio based on correspondence between the input textual query and the set of semantic chunks. The relative importance ratio may be associated with each chunk of the set of semantic chunks. The electronic device may determine a first budget based on the total budget and the relative importance ratio. The first budget may be associated with each chunk of the set of semantic chunks. The electronic device may condense the set of input video frames to determine a set of video scenes based on the first budget. Thereafter, a large multi-modal (LMM) model may be applied on the set of video scenes and the input textual query. The electronic device may determine an output associated with the input textual query, based on the application of the LLM on the prompt. The electronic device may control rendering of the output associated with the input textual query.

[0024] The disclosed approach may offer several advantages over traditional techniques used for video scene retrieval. The traditional techniques may involve retrieval of the video scene in which the entire video is condensed independent of a user query to generate condensed frames. The condensed frames and the user query may be then fed into a machine learning (ML) model to generate an answer associated with the content of the video. However, this technique may lead to degradation in the performance and accuracy of the ML model due to removal of important frames related to the user query, thereby further leading to generation of incorrect answers associated with the content of the video. The proposed method of the present disclosure solves this drawback by efficient retrieval of important video frames. This may be achieved using techniqueslike semantic chunking and video condensation. Another typical technique may involve generation of visual embeddings and feeding of the visual embeddings into the ML model to generate the answer for the user query. But a major drawback of this technique is that unimportant parts of the video may be overestimated, or important parts of the video may be lost.

[0025] The proposed method includes splitting of the set of input video frames into the set of semantic chunks, based on the relative importance ratio of each chunk with respect to the input textual query. Thus, those video frames whose textual caption (determined based on a captioning model) is similar to the input textual query may be selected and grouped together in a semantic chunk. This may lead to efficient detection of the important image frames that may be required to be retrieved for further processing, such as, visual question answering. Based on the effective retrieval of the important video scenes of interest to the user, the performance and cost associated with a video question answering system (such as, the LMM model) may be optimized.

[0026] The proposed technique may involve effective assignment of the budget for each of the set of semantic chunks based on a given total budget. Hence, the number of frames in each chunk and the total number of frames in the entire retrieved video may be controlled based on the relative importance ratio. The relative importance ratio may be in-turn governed by the semantic relevance or similarity of each frame with respect to the input textual query. Therefore, the retrieved video may have high accuracy and relevance with respect to the input textual query. Also, the use of chunk-level frame budgets (i.e., the first budget) and the total budget may optimize a retrieval of essential video frames, remove unwanted video frames, and avoid overestimation of the important video frames. The video scene retrieval method of the disclosure may be useful across diverse applications such as education or surveillance related applications.Embodiments of the present disclosure are explained with reference to the accompanying drawings.

[0027] FIG. 1 is a diagram representing an example network environment related to video scene retrieval based on a relative importance ratio, arranged in accordance with at least one embodiment described in the present disclosure. With reference to FIG. 1, there is shown an environment 100. The environment 100 may include an electronic device 102, a server 104, a database 106, a large multi-modal (LMM) model 120, and a communication network 122. Further, the electronic device 102 maybe communicatively coupled to the server 104, via the communication network 122. The database 106 may store a set of input video frames 110, a set of set of semantic chunks 116, and a set of video scenes 118. The electronic device 102 may include the LMM model 120 and a user input 108. The user input 108 may include an input textual query 112 and a total budget 114.

[0028] The electronic device 102 may include suitable logic, circuitry, interfaces and / or code that may be configured to receive the set of input video frames 110 and the input textual query 112. The electronic device 102 may receive the total budget 114 associated with a retrieval of the set of video scenes 118. The total budget 114 may correspond to a total count of video frames of the set of input video frames 110 to be included in the set of video scenes 118, or a size of main memory or graphics processing unit (GPU) on the electronic device 102. The electronic device 102 may further split the set of input video frames 110 into the set of semantic chunks 116. The electronic device 102 may determine the relative importance ratio associated with each of the set of semantic chunks 116, based on the correspondence between the input textual query 112 and the set of semantic chunks 116. Also, the electronic device 102 may determine a first budget associated with each chunk of the set of semantic chunks 116, based on the totalbudget 114 and the relative importance ratio. The first budget associated with each chunk of the set of semantic chunks 116 may correspond to a count of video frames of the set of input video frames 110 to be included in the corresponding chunk. The electronic device 102 may condense the set of input video frames 110 to determine the set of video scenes 118 based on the first budget. The electronic device 102 may further apply the LMM model 120 on the set of video scenes 118. The electronic device 102 may further determine an output associated with the input textual query 112 based on the application of the LMM model 120. The electronic device 102 may control rendering of the output associated with the input textual query 112.

[0029] The electronic device 102 may further determine a set of visual embeddings associated with the set of input video frames 110. The splitting of the set of input video frames 110 into the set of semantic chunks 116 may be based on the set of visual embeddings. The set of input video frames 110 may be split into the set of semantic chunks 116 using a semantic chunking technique, based on a contextual meaning of each video frame. For example, the electronic device 102 may identify a semantic coherence across each of the set of input video frames 110 to ensure that each of the set of semantic chunks 116 is a meaningful unit. To identify the semantic coherence, the electronic device 102 may determine the set of visual embeddings of each of the set of input video frames 110. Further, the electronic device 102 may determine similarity scores between each of the set of input video frames 110, to ensure that similar consecutive video frames are grouped as a single semantic chunk while dissimilar video frames are in different semantic chunks. The electronic device 102 may determine a set of textual embeddings associated with the input textual query 112. The determination of the relative importance ratio may be based on the set of semantic chunks 116 or the chunk embeddings and the set of textual embeddings. The relative importance ratio may be associated with eachchunk of the set of semantic chunks 116 and determined based on the correspondence between the input textual query 112 and the set of semantic chunks 116.

[0030] In an embodiment, the electronic device 102 may control a display device (e.g., a display device 206A of FIG. 2). The display device 206A may be communicatively coupled to the electronic device 102 or may be a standalone device configured to render the output associated with the input textual query 112. The output associated with the input textual query 112 may be determined based on the application of the LMM model 120. Examples of the electronic device 102 may include, but may not be limited to, a computing device, a smartphone, a mainframe machine, a server, a consumer electronic (CE) device, a computer workstation, and / or a device with a graph-processing capability (such as, a device with a set of graphic processor units (GPU)).

[0031] In one or more embodiments, the electronic device 102 may retrieve the set of input video frames 110, the input textual query 112, and the total budget 114 from the database 106, based on the user input 108. The total budget 114 may be associated with the retrieval of the set of video scenes 118.

[0032] Each input video frame of the set of input video frames 110 may correspond to a single still image which if played in sequence with other input video frames of the video may create motion on a playback surface. For example, each input video frame may be a complete image or may be a transformation of other frames in a sequence of videos or a clever removal of duplicated image data. The set of input video frames 110 may further correspond to analog frames or digital frames encoded using a software that implements a video codec. Further, the video codec may involve use of different categories of each input video frame to express the video. The different categories of the set of input video frames 110 may correspond to an intra-coded picture, a predictive coded picture, or a bi-predictive coded picture. In context of video image processing, these categories of the setof input video frames 110 are well known to one ordinary skilled in the art. Therefore, a description of each category of the set of input video frames 110 is omitted from the disclosure for the sake of brevity.

[0033] In one or more embodiments, each input video frame maybe a single image in a sequence of pictures. For example, one second of the input video frame may be comprised of 24 or 30 frames per second (FPS). Each of the set of input video frames 110 may be presented for a specific time duration on the display device. Further, each of the set of input video frames 110 may be split into the set of semantic chunks 116. The set of semantic chunks 116 may be an image context-based chunking unit associated with the each of the set of input video frames 110. Each chunk may be an independently playable smaller segment associated with each of the set of input video frames 110.

[0034] The electronic device 102 and the server 104 maybe communicatively coupled with each other via the communication network 122. In some embodiments, the electronic device 102 may include the LMM model 120 and the display device 206A. In some other embodiments, the display device 206A may be separate from the electronic device 102 and / or communicative with the electronic device 102. The server 104 maybe communicatively coupled with the database 106.

[0035] The server 104 may be implemented as a cloud server that may be configured to store the set of input video frames 110 on the database 106. In some embodiments, the server 104 may also store the user input 108 including the input textual query 112 and the total budget 114 associated with the retrieval of the set of video scenes 118. Further, the server 104 may also store the LMM model 120 on the database 106. The server 104 may be configured to retrieve data (for example, the set of input video frames 110, the input textual query 112, and the total budget 114) from the database 106 and transmit the retrieved data to the electronic device 102. Additionally, or alternatively, the server104 may be implemented using on-premises hosting (local servers), colocation hosting (third-party data centers), bare metal servers (dedicated servers), edge computing (local data processing), fog computing (decentralized data processing), mesh computing (distributed computing), hybrid cloud (combination of on-premises and cloud), or multicloud (multiple cloud providers).

[0036] The server 104 may be implemented as a cloud server and may execute operations through web applications, cloud applications, hypertext transport protocol (HTTP) requests, repository operations, file transfer, and the like. In at least one embodiment, the server 104 may be implemented as a plurality of distributed cloudbased resources by use of several technologies that are well known to those ordinary skilled in the art. In certain embodiments, the functionalities of the server 104 may be incorporated in its entirety or at least partially in the electronic device 102, without a departure from the scope of the disclosure.

[0037] The database 106 may include suitable logic, circuitry, interface, and / or code that may be configured to store the set of input video frames 110 and the input textual query 112. As used herein, the term “input textual query” may be natural language text describing a task to be performed by the LMM model 120. The natural language text may be in the form of a structured instruction or a user-input prompt that may be interpreted by the LMM model 120. For example, a typical textual query may be a description of a desired output. The database 106 may further store the total budget 114 associated with the retrieval of the set of video scenes 118. In an embodiment, the input textual query 112 may include the total budget 114. In another embodiment, the total budget 114 may be separate from the input textual query 112 and may be included as a standalone instruction in the user input 108. The database 106 may be derived from data off a relational or non-relational database, or a set of comma-separated values (csv) files in aconventional storage or a big-data storage. The database 106 may be stored or cached on a device, such as, the server 104 or the electronic device 102. The device storing the database 106 may be configured to receive a query for the set of input video frames 110. In response to the query, the device storing the database 106 may be configured to retrieve and transmit the set of input video frames 110 to the electronic device 102.

[0038] In accordance with an embodiment, the database 106 may be hosted on a plurality of servers stored at same or different locations. The operations of the database 106 may be executed using hardware including a processor, a microprocessor (for example, to perform or control performance of one or more operations), a field-programmable gate array (FPGA), or an application-specific integrated circuit (ASIC). In some other instances, the database 106 may be implemented using software.

[0039] A person with ordinary skill in the art will understand that the scope of the disclosure may not be limited to the implementation of the server 104 (or the electronic device 102) and the database 106 as two separate entities. In certain embodiments, the functionalities of the database can be incorporated in its entirety or at least partially in the server 104, without a departure from the scope of the disclosure.

[0040] The LMM model 120 may be a language model and include suitable logic, circuitry, interfaces, and / or code that may be applied on the input textual query 112 and the set of video scenes 118. Based on the application of the LMM model 120 on the input textual query 112 and the set of video scenes 118, the output associated with the input textual query 112 may be determined. For example, the input textual query 112 may include a statement including an instruction to determine the output for the statement. The LMM model 120 may determine the output associated with the input textual query 112, based on the statement including the instruction. In one instance, a trained LMM model may be able to determine a set of retrieved video scenes based on the relativeimportance ratio associated with each chunk of the set of semantic chunks 116. For example, the set of retrieved video scenes may refer to a set of video images retrieved from the set of video scenes 118. The LMM model 120 may analyze information represented as the output associated with the input textual query 112. The rendering of the output associated with the input textual query 112 may be controlled. In one instance, the LMM model 120 may determine a set of visual embeddings associated with the set of input video frames 110. The splitting of the set of input video frames 110 into the set of semantic chunks 116 may be based on the visual embeddings. In another instance, the LMM model 120 may further determine a set of textual embeddings associated with the input textual query 112. The determination of the relative importance ratio may be based on the set of semantic chunks 116 or the chunk embeddings and the set of textual embeddings.

[0041] The LMM model 120 may be a multi-modal Artificial Intelligence (Al) / Machine Learning (ML) model that may be trained on vast amounts of diverse data, enabling the LMM model 120 to perform a wide range of natural language processing tasks, such as, language translation, summarization, question answering, video generation, image generation, and text generation. The LMM model 120 may be an Al model that can process multiple type of data modalities. The data modalities may include text, images, audio, video, and other data types like sensory data. Further, the LMM model 120 may integrate different types of data inputs and may learn to correlate text with images or understand sequences in videos. The LMM model 120, for example, may use transformer architectures, which allow them to process and generate text efficiently. During training, the LMM model 120 may learn a statistical relationship between words and phrases by analyzing large datasets. This training may enable the LMM model 120 to learn how to determine a context, syntax, and semantics associated with any natural language text,making them capable of generating coherent and contextually relevant responses. Fine-tuning of the LMM model 120 during training may not only involve specialized datasets for each modality, but also datasets that help the LMM model 120 learn cross-modal relationships.

[0042] The LMM model 120 may be evaluated on a wider range of metrics due to their involvement in a diverse set of applications. The metrics may include, but is not limited to, an image recognition accuracy, an audio processing quality, and an ability to integrate the information across different modalities. The large multi-modal model (such as, the LMM model 120) may include, for example, but is not limited to, a Generative Pre-trained Transformer (GPT) model, a Bi-directional Encoder Representations from Transformers (BERT) model, a Text-To-Text Transfer Transformer (T5) model, and the like. The LMM model 120 may be implemented using hardware including a processor, a microprocessor (e.g., to perform or control performance of one or more operations), a field-programmable gate array (FPGA), or an application-specific integrated circuit (ASIC). In some other instances, the LMM model 120 may be a code, a program, or set of software instructions. The LMM model 120 maybe implemented using a combination of hardware and software.

[0043] The communication network 122 may include various communication media through which the electronic device 102 may communicate with the server 104, or devices storing the database 106. Examples of the communication network 122 may include, but are not limited to, the Internet, a cloud network, a Wireless Fidelity (Wi-Fi) network, a Personal Area Network (PAN), a Local Area Network (LAN), a cellular network (such as, a Long-term evolution (or 4G) cellular network or a 5G cellular network), a satellite network (such as, a network of low earth orbit satellites), and / or a Metropolitan Area Network (MAN)). Various devices in the example environment 100 may beconfigured to connect to the communication network 122, in accordance with various wired and wireless communication protocols. Examples of such wired and wireless communication protocols may include, but are not limited to, at least one of a Transmission Control Protocol and Internet Protocol (TCP / IP), User Datagram Protocol (UDP), Hypertext Transfer Protocol (HTTP), File Transfer Protocol (FTP), ZigBee, EDGE, IEEE 802.11, light fidelity(Li-Fi), 802.16, IEEE 802.11s, IEEE 802.11g, multi-hop communication, wireless access point (AP), device to device communication, cellular communication protocols, and / or Bluetooth (BT) communication protocols, or a combination thereof.

[0044] In operation, the electronic device 102 may be configured to receive the set of input video frames 110 and the inputtextual query 112. The inputtextual query 112 may include a statement including an instruction to determine the output for the statement. The reception of the set of input video frames and the input textual query is described further, for example, with reference to FIG. 3, FIG.4, and FIG. 5.

[0045] The electronic device 102 may be configured to receive the total budget 114 associated with the retrieval of the set of video scenes 118. The total budget 114 may correspond to the total count of input video frames, to be included in the set of video scenes 118, or the size of the main memory or the graphics processing unit (GPU) on the electronic device 102. The reception of the total budget is described further, for example, with reference to FIG. 3, FIG. 4, FIG. 5, and FIG. 6.

[0046] The electronic device 102 may be configured to split the set of input video frames 110 into the set of semantic chunks 116. The electronic device 102 may determine the set of visual embeddings associated with the set of input video frames 110. The set of input video frames 110 maybe split into the set of semantic chunks 116 based on the set of visual embeddings. The electronic device 102 may generate, by use of a captioningmodel, each caption of a first set of captions associated with each corresponding video frame of the set of input video frames 110. The electronic device 102 may further compare a first caption of a first video frame with a second caption of a second video frame. The first set of captions may include the first caption and the second caption. The set of input video frames 110 may include the first video frame and the second video frame neighboring the first video frame in the set of input video frames 110. Thereafter, the electronic device 102 may determine a first similarity score between the first caption and the second caption, based on the comparison of the first caption with the second caption. Further, the set of input video frames 110 may be split into the set of semantic chunks 116 based on the first similarity score. In an example, the electronic device 102 may determine whether the first similarity score between the first caption and the second caption is greater than a threshold. The electronic device 102 may identify a chunk of the set of semantic chunks 116 based on the determination that the first similarity score between the first caption and the second caption is greater than the threshold. Further, the set of input video frames 110 maybe split into the set of semantic chunks 116 based on the identification of the chunk. The splitting of the set of input video frames into the set of semantic chunks is described further, for example, in FIG.4, FIG.6, FIG. 7A, and FIG.

[0047] 7B.

[0048] The electronic device 102 may be configured to determine the relative importance ratio associated with each chunk of the set of semantic chunks 116, based on the correspondence between the input textual query 112 and the set of semantic chunks 116. The electronic device 102 may determine the set of textual embeddings associated with the input textual query 112. The determination of the relative importance ratio may be based on the set of semantic chunks 116 or the chunk embeddings and the set of textual embeddings. Further, the electronic device 102 may determine a second similarityscore between each video frame of the set of input video frames 110 and the input textual query 112. The electronic device 102 may determine an average similarity value corresponding to each chunk of the set of semantic chunks 116, based on the second similarity score. The electronic device 102 may normalize the average similarity value corresponding to each chunk of the set of semantic chunks 116 to determine a first normalized score associated with each chunk of the set of semantic chunks 116. The determination of the relative importance ratio associated with each chunk of the set of semantic chunks 116 may further be based on the first normalized score.

[0049] In an example, the electronic device 102 may further generate, by use of the captioning model, each caption of a second set of captions associated with each corresponding chunk of the set of semantic chunks 116. The electronic device 102 may determine a third similarity score between each caption of the second set of captions and the inputtextual query 112. Thereafter, the electronic device 102 may normalize the third similarity score corresponding to each chunk of the set of semantic chunks 116 to determine a second normalized score associated with each chunk of the set of semantic chunks 116. The determination of the relative importance ratio associated with each chunk of the set of semantic chunks may be further based on the second normalized score. The determination of the relative importance ratio associated with each chunk based on the correspondence between the input textual query and the set of semantic chunks is described further, for example, in FIG. 4, FIG. 5, and FIG. 6.

[0050] The electronic device 102 may be configured to determine the first budget associated with each chunk of the set of semantic chunks 116, based on the total budget 114 and the relative importance ratio. The first budget may correspond to the total count of video frames of the set of input video frames 110 to be included in the set of videoscenes 118. The determination of the first budget associated with each chunk of the set of semantic chunks is described further, for example, in FIG. 3, FIG.4, FIG. 5, and FIG. 6.

[0051] The electronic device 102 may be configured to condense the set of input video frames 110 to determine the set of video scenes 118 based on the first budget. The condensation of the set of input video frames to determine the set of video scenes is described further, for example, in FIG. 3, FIG. 4, FIG. 5, and FIG. 6.

[0052] The electronic device 102 may be configured to apply the large multi-modal (LMM) model 120 on the set of video scenes 118 and the input textual query 112. The application of the LMM model on the set of video scenes and the input textual query is described further, for example, in FIG.4.

[0053] The electronic device 102 may be configured to determine the output associated with the input textual query 112 based on the application of the LMM model 120. The determination of the output associated with the input textual query based on the application of the LMM model is described further, for example, in FIG. 4.

[0054] The electronic device 102 may be configured to control a rendering of the output associated with the input textual query 112. The control of the rendering of the output associated with the input textual query is described further, for example, in FIG.

[0055] 5.

[0056] Modifications, additions, or omissions may be made to FIG. 1 without departing from the scope of the present disclosure. For example, the environment 100 may include more or fewer elements than those illustrated and described in the present disclosure. For instance, in some embodiments, the environment 100 may include the electronic device 102 but not the database 106. In addition, in some embodiments, the functionality of the database 106 may be incorporated into the electronic device 102, without a deviation from the scope of the disclosure.FIG. 2 is a block diagram that illustrates an exemplary electronic device of FIG.

[0057] 1 for video scene retrieval based on a relative importance ratio, arranged in accordance with at least one embodiment described in the present disclosure. FIG. 2 is explained in conjunction with elements from FIG. 1. With reference to FIG. 2, there is shown a block diagram 200 of the electronic device 102. The electronic device 102 may include a processor 202, a memory 204, the LMM model 120, the database 106, an input / output (I / O ) device 206, and a network interface 208. The I / O device 206 may include a display device 206A. The memory 204 may include a data (for e.g. the set of input video frames 110 and the user input 108 including the input textual query 112, and the total budget 114).

[0058] The processor 202 may include suitable logic, circuitry, interfaces, and / or code that may be configured to execute program instructions associated with different operations to be executed by the electronic device 102. The operations may include, but are not limited to, video frame reception, total budget reception, input video frames splitting, relative importance ratio determination, first budget determination, input video frames condensation, LMM application, output determination, and output rendering control. The processor 202 may include any suitable special-purpose or general-purpose computer, computing entity, or processing device, including various computer hardware or software modules, and may be configured to execute instructions stored on any applicable computer-readable storage media. For example, the processor 202 may include a microprocessor, a microcontroller, a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a Field-Programmable Gate Array (FPGA), or any other digital or analog circuitry configured to interpret and / or to execute program instructions and / or to process data.Although illustrated as a single processor in FIG. 2, the processor 202 may include any number of processors configured to, individually or collectively, perform or direct performance ofany number of operations of the electronic device 102, as described in the present disclosure. Additionally, one or more of the processors may be present on one or more different electronic devices 102, such as different servers.

[0059] In some embodiments, the processor 202 may be configured to interpret and / or execute program instructions and / or process data stored in the memory 204. In some embodiments, the processor 202 may fetch program instructions from the LMM model 120 and load the program instructions in the memory 204. After the program instructions are loaded into memory 204, the processor 202 may execute the program instructions. Some of the examples of the processor 202 may be a Graphical Processing Unit (GPU), a Central Processing Unit (CPU), a Reduced Instruction Set Computer (RISC) processor, an Application-Specific Integrated Circuit (ASIC) processor, a Complex Instruction Set Computer (CISC) processor, a co-processor, and / or a combination thereof.

[0060] The memory 204 may include suitable logic, circuitry, interfaces, and / or code that may be configured to store program instructions executable by the processor 202. In certain embodiments, the memory 204 may be configured to store information, such as, but not limited to, the set of input video frames 110, the input textual query 112, the total budget 114, and the LMM model 120. The memory 204 may further store the relative importance ratio associated with each chunk of the set of semantic chunks 116. The memory 204 may further store the first budget associated with each chunk of the set of semantic chunks 116. The memory 204 may further store the set of video scenes 118. The memory 204 may further store the output associated with the input textual query 112 determined based on the application of the LMM model 120.The memory 204 may include computer-readable storage media for carrying or having computer-executable instructions or data structures stored thereon. Such computer-readable storage media may include any available media that may be accessed by a general-purpose or special-purpose computer, such as the processor 202. By way of example, and not limitation, such computer-readable storage media may include tangible or non-transitory computer-readable storage media, including but not limited to, a CPU cache, a Hard Disk Drive (HDD), a Solid-State Drive (SSD), Random Access Memory (RAM), Read-Only Memory (ROM), Electrically Erasable Programmable Read-Only Memory (EEPROM), Compact Disc Read-Only Memory (CD-ROM), a Secure Digital (SD) card, or other optical disk storage, magnetic disk storage or other magnetic storage devices, or flash memory devices (e.g., solid state memory devices). The computer-readable storage may also include any other storage medium which may be used to carry or store particular program code in the form of computer-executable instructions or data structures, and which may be accessed by a general-purpose or special-purpose computer. Combinations of the above may also be included within the scope of computer-readable storage media. Computer-executable instructions may include, for example, instructions and data configured to cause the processor 202 to perform a certain operation or group of operations associated with the electronic device 102.

[0061] The I / O device 206 may include suitable logic, circuitry, interfaces, and / or code that may be configured to receive data such as the user input 108. For example, the user input 108 may indicate a selection of the set of input video frames 110, the input textual query 112, and the total budget 114. The I / O device 206 may be further configured to provide an output in response to the user input. For example, the output may correspond to the retrieved set of video scenes 118. The I / O device 206 may include various input and output devices, which may be configured to communicate with theprocessor 202 and other components, such as the network interface 210. Examples of the input devices may include, but are not limited to, a touch screen, a keyboard, a mouse, a joystick, and / or a microphone. Examples of the output devices may include, but are not limited to, the display device 206A and a speaker. The I / O device 206 may be within the electronic device 102 or may be communicatively coupled and separate from the electronic device 102.

[0062] The display device 206A may include logic, circuitry, and interfaces configured to display the set of input video frames 110, the input textual query 112, the total budget 114, and the output. The display device 206A may be a touch screen which may enable a user to provide user-inputs via the display device 206A. The touch screen may be at least one of a resistive touch screen, a capacitive touch screen, or a thermal touch screen. The display device 206A may be realized through several known technologies such as, but not limited to, a Liquid Crystal Display (LCD) display, a Light Emitting Diode (LED) display, a plasma display, or an Organic LED (OLED) display technology, or other display devices. In accordance with an embodiment, the display device 206A may refer to a display screen of a head mounted device (HMD), a smart-glass device, a see-through display, a projection-based display, an electro-chromic display, or a transparent display.

[0063] The network interface 208 may include suitable logic, circuitry, and interfaces that may be configured to facilitate communication between the processor 202 (i.e., the electronic device 102) and the server 104, via the communication network 122. The network interface 208 may be implemented by use of various known technologies to support wired or wireless communication of the electronic device 102 with the communication network 122. The network interface 208 may include, but is not limited to, antenna, a radio frequency (RF) transceiver, one or more amplifiers, a tuner, one ormore oscillators, a digital signal processor, a coder-decoder (CODEC) chipset, a subscriber identity module (SIM) card, or a local buffer circuitry.

[0064] The network interface 208 may be configured to communicate via wireless communication with networks, such as the Internet, an Intranet, or a wireless network, such as a cellular telephone network, a wireless local area network (LAN), and a metropolitan area network (MAN). The wireless communication may be configured to use one or more of a plurality of communication standards, protocols and technologies, such as Enhanced Data GSM Environment (EDGE), wideband code division multiple access (W-CDMA), Long Term Evolution (LTE), 5thGeneration (5G) New Radio (NR), Global System for Mobile Communications (GSM), code division multiple access (CDMA), time division multiple access (TDMA), Bluetooth, Wireless Fidelity (Wi-Fi) (such as IEEE 802.11a, IEEE 802.11b, IEEE 802.11g or IEEE 802. lln), voice over Internet Protocol (VoIP), light fidelity (Li-Fi), Worldwide Interoperability for Microwave Access (Wi-MAX), a protocol for email, instant messaging, and a Short Message Service (SMS).

[0065] In certain embodiments, the electronic device 102 maybe divided into a frontend subsystem and a backend subsystem. The front-end subsystem may be solely configured to receive requests / instructions from a user device, one or more of third-party servers, web servers, client machine, and the backend subsystem. These requests may be communicated back to the backend subsystem, which may be configured to act upon these requests. For example, in case the electronic device 102 is in communication with multiple servers, few of the servers may be front-end servers configured to relay the requests / instructions to remaining servers associated with the backend subsystem.

[0066] Modifications, additions, or omissions may be made to the example electronic device 102 without departing from the scope of the present disclosure. For example, insome embodiments, the example electronic device 102 may include any number of other components that may not be explicitly illustrated or described for the sake of brevity.

[0067] FIG. 3 is a diagram that illustrates an exemplary execution pipeline for video scene retrieval based on a relative importance ratio, in accordance with an embodiment of the disclosure. FIG. 3 is described in conjunction with elements from FIG. 1 and FIG. 2. With reference to FIG. 3, there is shown an exemplary execution pipeline 300. The execution pipeline 300 may include a sequence of operations that may be executed by the processor 202 of the electronic device 102 of FIG. 1 for the retrieval of the video scene based on the relative importance ratio.

[0068] The execution pipeline 300 includes an operation for reception of set of input video frames 302, an operation for encoding of the input video frames by a frame encoder unit 304, an operation for generation of the set of semantic chunks 116 by a semantic chunking unit 306, and an operation for reception of the input textual query 112 by a text encoder unit 308. The execution pipeline 300 further includes an operation for determination of the relative importance ratio by a relative importance ratio computing unit 310, an operation for determination of the first budget 416 by a budget computing unit 312, an operation for condensation of the input video frames by a condensation unit 314, and an operation for generation of retrieved video image frames 316 as the output. In an embodiment, the processor 202 may include instructions to execute the various operations of the execution pipeline 300. The instructions may implement, for example, but not limited to, the frame encoder unit 304, the semantic chunking unit 306, the text encoder unit 308, the relative importance ratio computing unit 310, the budget computing unit 312, and the condensation unit 314.

[0069] In an embodiment, the operation for the reception of the set of input video frames 302 may be executed. The processor 202 of the electronic device 102 may beconfigured to receive the set of input video frames 302. In one or more embodiments, the processor 202 may be configured to receive the set of input video frames 302 associated with an input video from the database 106. The reception of the set of input video frames is described further, for example, in FIG.4, FIG. 5, FIG. 6, FIG. 7A, FIG. 7B, and FIG. 70.

[0070] In an embodiment, the operation for the encoding of the input video frames, by the frame encoder unit 304, may be executed. The processor 202 may be configured to encode the set of input video frames 110 into vector data. In one or more embodiments, the processor 202 may be configured to encode, using the frame encoder unit 304, the set of input video frames 110 into the vector data. The encoding of the set of input video frames is described further, for example, in FIG.4.

[0071] In an embodiment, the operation for generation of the set of semantic chunks 116, by the semantic chunking unit 306, may be executed. In an embodiment, the processor 202 may be configured to generate the set of semantic chunks 116 associated with the set of input video frames 110. The processor 202 may split, using the semantic chunking unit 306, the set of input video frames 110 into the set of semantic chunks 116. In an instance, the processor 202 maybe further configured to determine the set of visual embeddings associated with the set of input video frames 110. The splitting of the set of input video frames 110 into the set of semantic chunks 116 may be based on the set of visual embeddings.

[0072] In some instances, the processor 202 may be further configured to generate each caption of the first set of captions associated with each corresponding video frame of the set of input video frames 110. Thereafter, the first caption of the first video frame may be compared with the second caption of the second video frame. Based on the comparison of the first caption with the second caption, the first similarity score between the first caption and the second caption may be determined. The splitting of the set ofinput video frames 110 into the set of semantic chunks 116 may be based on the first similarity score. The splitting of the set of input video frames into the set of semantic chunks is described further, for example, in FIG.4, FIG. 5, FIG. 6, and FIG. 7A.

[0073] In an embodiment, the operation for reception of the input textual query 112, by the text encoder unit 308, may be executed. In an embodiment, the processor 202 may be configured to receive the input textual query 112 for retrieval of the set of video scenes 118. The input textual query 112 may be natural language text describing a task to be performed by the LMM model 120. In an example, the typical textual query may be a description of the desired output. The reception of the input textual query is described further, for example, in FIG. 5, FIG. 6, FIG. 7B and FIG. 70.

[0074] In an embodiment, the operation for determination of the relative importance ratio, by the relative importance ratio computing unit 310, may be executed. The processor 202 may be configured to determine, using the relative importance ratio computing unit 310, the relative importance ratio associated with each chunk of the set of semantic chunks 116. The processor 202 may be further configured to determine the relative importance ratio associated with each chunk of the set of semantic chunks 116, based on the correspondence between the input textual query 112 and the set of semantic chunks 116. The processor 202 may be further configured to determine the set of textual embeddings associated with the input textual query 112. The determination of the relative importance ratio may be based on the set of semantic chunks 116 or the chunk embeddings and the set of textual embeddings.

[0075] In one instance, the processor 202 may be further configured to determine the second similarity score between each input video frame of the set of input video frames 110 and the input textual query 112. Thereafter, the average similarity value corresponding to each chunk of the set of semantic chunks 116 maybe determined, basedon the second similarity score. The average similarity value corresponding to each chunk of the set of semantic chunks 116 may be then normalized to determine the first normalized score associated with each chunk of the set of semantic chunks 116. Based on the first normalized score, the relative importance ratio associated with each chunk of the set of semantic chunks 116 may be determined. Further, the determination of the relative importance ratio based on the first normalized score may correspond to a framebased relative importance ratio determination technique.

[0076] In another instance, the processor 202 may be further configured to generate, by use of the captioning model, each caption of a second set of captions associated with each corresponding semantic chunk of the set of semantic chunks 116. Thereafter, a third similarity score may be determined between each caption of the second set of captions and the input textual query 112. The third similarity score corresponding to each chunk of the set of semantic chunks 116 may be then normalized to determine a second normalized score associated with each chunk of the set of semantic chunks 116. Based on the second normalized score, the relative importance ratio associated with each chunk of the set of semantic chunks 116 may be determined. Further, the determination of the relative importance ratio based on the second normalized score may correspond to a caption-based importance ratio determination technique. The determination of the relative importance ratio is described further, for example, in FIG. 4, FIG. 5, FIG. 6, and FIG. 7C.

[0077] In an embodiment, the operation for determination of the first budget by the budget computing unit 312 may be executed. The processor 202 may be configured to determine, using the budget computing unit 312, the first budget associated with each chunk of the set of semantic chunks 116 based on the total budget 114 and the relative importance ratio. The first budget associated with each chunk of the set of semanticchunks 116 may correspond to the count of input video frames of the set of input video frames 110 to be included in the corresponding semantic chunk. The determination of the first budget associated with each chunk of the set of semantic chunks is described further, for example, in FIG.4, FIG. 5, and FIG. 6.

[0078] In an embodiment, the operation for the condensation of the input video frames, by the condensation unit 314, may be executed. The processor 202 may be configured to condense the set of input video frames 110 to determine the set of video scenes 118. In an embodiment, the processor 202 may be configured to condense, using the condensation unit 314, the set of input video frames 110 to determine the set of video scenes 118 based on the first budget associated with each chunk of the set of semantic chunks 116. Further, the LMM model 120 may be applied on the input textual query 112 and the set of video scenes 118. The condensation of the set of input video frames is described further, for example, in FIG. 5 and FIG. 6.

[0079] In an embodiment, the operation for generation of the retrieved video image frames 316 as the output may be executed. The processor 202 may be configured to determine the output associated with the input textual query 112 based on the application of the LMM model 120. The output may correspond to the retrieved video image frames 316. The retrieved video image frames 316 may be input video frames of the set of input video frames 110 retrieved based on the relative importance ratio associated with each chunk of the set of semantic chunks 116. For example, the retrieved video image frames 316 may include the set of video scenes 118. The determination of the output associated with the input textual query is described further, for example, in FIG. 4 and FIG. 5.

[0080] In an embodiment, based on the determined output, an operation for controlling a rendering of the output associated with the input textual query 112 may beexecuted. The processor 202 may be further configured to control the rendering of the output associated with the input textual query 112. For example, the processor 202 may control a display of the output associated with the input textual query 112 on the display device 206A.

[0081] The disclosed approach may offer several advantages over traditional techniques used for video scene retrieval. The traditional techniques may involve retrieval of the video scene in which the entire video is condensed independent of a user query to generate condensed frames. The condensed frames and the user query may be then fed into a machine learning (ML) model to generate an answer associated with the content of the video. However, this technique may lead to degradation in the performance and accuracy of the ML model due to removal of important frames related to the user query, thereby further leading to generation of incorrect answers associated with the content of the video. The proposed method of the present disclosure solves this drawback by efficient retrieval of important video frames from the set of input video frames 110. This may be achieved using techniques like semantic chunking and video condensation. Another typical technique may involve generation of visual embeddings and feeding of the visual embeddings into the ML model to generate the answer for the user query. But a major drawback of this technique is that unimportant parts of the video may be overestimated, or important parts of the video may be lost.

[0082] The proposed method includes splitting of the set of input video frames 110 into the set of semantic chunks 116, based on the relative importance ratio of each chunk with respect to the input textual query 112. Thus, those video frames whose textual caption (determined based on a captioning model) is similar to the input textual query 112 may be selected and grouped together in a semantic chunk. This may lead to efficient detection of the important image frames that may be required to be retrieved for furtherprocessing, such as, visual question answering. Based on the effective retrieval of the important video scenes of interest to the user, the performance and cost associated with a video question answering system (such as, the LMM model 120) may be optimized.

[0083] The proposed technique may involve effective assignment of the budget for each of the set of semantic chunks based on a given total budget (e.g., the total budget 114). Hence, the number of frames in each chunk and the total number of frames in the entire retrieved video (i.e., the set of video scenes 118) may be controlled based on the relative importance ratio. The relative importance ratio may be in-turn governed by the semantic relevance or similarity of each frame with respect to the input textual query 112. Therefore, the retrieved video may have high accuracy and relevance with respect to the input textual query 112. Also, the use of chunk-level frame budgets (i.e., the first budget) and the total budget 114 may optimize a retrieval of essential video frames, remove unwanted video frames, and avoid overestimation of the important video frames from the set of input video frames. The video scene retrieval method of the disclosure may be useful across diverse applications such as education or surveillance related applications.

[0084] FIG. 4 is a diagram that illustrates an exemplary execution pipeline for video scene retrieval based on a relative importance ratio, in accordance with an embodiment of the disclosure. FIG.4 is described in conjunction with elements from FIG. 1, FIG. 2, and FIG. 3. With reference to FIG. 4, there is shown an exemplary execution pipeline 400 for the video scene retrieval. The execution pipeline 400 may include the set of input video frames 302, the input textual query 112, the LMM model 120, an operation for semantic chunking 402, a set of semantic chunks 404, an encoder 406, a set of visual embeddings 408, a text encoder 410, a set of textual embeddings 412, a relative importance ratio 414, a first budget 416, and a retrieved video image frames 418.The set of input video frames 302 and the input textual query 112 may be received. Further, the total budget 114 associated with the retrieval of the set of video scenes 118 may be received. In one or more embodiments, the processor 202 may be configured to receive the set of input video frames 302 and the input textual query 112. The input textual query 112 may be a natural language text describing a task to be performed by the LMM model 120. The natural language text may be in form of a structured instruction that may be interpreted by the LMM model 120. The processor 202 may be further configured to receive the total budget 114 associated with the retrieval of the set of video scenes 118. The total budget 114 may correspond to the total count of video frames, of the set of input video frames 302, to be included in the set of video scenes 118, or the size of the main memory or the GPU on the electronic device 102.

[0085] At 402, the operation for semantic chunking may be executed. The processor 202 may be configured to split the set of input video frames 110 into the set of semantic chunks 404. The set of semantic chunks 404 maybe further encoded to determine the set of visual embeddings 408. In an embodiment, the processor 202 may be configured to determine, using the encoder 406, the set of visual embeddings 408 associated with the set of input video frames 302. The splitting of the set of input video frames 302 into the set of semantic chunks 404 may be based on the set of visual embeddings 408.

[0086] Based on the correspondence between the input textual query 112 and the set of semantic chunks 116, the relative importance ratio 414 associated with each chunk of the set of semantic chunks 404 may be determined. In one or more embodiment, the processor 202 may be configured to determine the relative importance ratio 414 associated with each chunk of the set of semantic chunks 404, based on the correspondence between the input textual query 112 and the set of semantic chunks 116. In one instance, the processor 202 may be further configured to determine, using the textencoder 410, the set of textual embeddings 412 associated with the input textual query 112. Based on the set of textual embeddings 412, the relative importance ratio 414 associated with each chunk of the set of semantic chunks 404 may be determined.

[0087] Based on the total budget 114 and the relative importance ratio 414, the first budget 416 associated with each chunk of the set of semantic chunks 404 may be determined. In one embodiment, the processor 202 may be configured to determine the first budget 416 associated with each chunk of the set of semantic chunks 404. The first budget 416 associated with each chunk of the set of semantic chunks 404 may correspond to a count of video frames, of the set of input video frames 302, to be included in the corresponding semantic chunk.

[0088] Based on the first budget 416, the set of input video frames 302 may be condensed to determine the set of video scenes 118. The set of input video frames 302 may be condensed to generate condensed video image frames (i.e., the retrieved video image frames 418). In one embodiment, the processor 202 may be configured to condense the set of input video frames 302 to determine the setofvideo scenes 118 (such as, the condensed video image frames, i.e., the retrieved video image frames 418).

[0089] Further, the LMM model 120 maybe applied on the setofvideo scenes 118 and the input textual query 112. Based on the application of the LMM model 120, the output associated with the input textual query 112 may be determined. The processor 202 may be configured to determine the output associated with the input textual query 112, based on the application of the LMM model 120 on the setofvideo scenes 118. For example, the input textual query 112 may include a statement including an instruction to determine the output for the statement. The LMM model 120 may determine the output associated with the input textual query 112, based on the statement including the instruction. The rendering of the determined output associated with the input textual query 112 may becontrolled. In one or more embodiments, the processor 202 maybe configured to control the rendering of the output associated with the input textual query 112.

[0090] FIG. 5 is a diagram that illustrates an exemplary scenario for determination of a set of video scenes based on a relative importance ratio and a first budget, in accordance with an embodiment of the disclosure. FIG. 5 is described in conjunction with elements from FIG. 1, FIG.2, FIG. 3, and FIG.4. With reference to FIG. 5, there is shown an exemplary scenario 500 for the determination of the set of video scenes 118 based on the relative importance ratio and the first budget. The scenario 500 may include a set of input video frames 502, the operation for semantic chunking 402, the relative importance ratio 414, the first budget 416, the condensation unit 314, an input textual query 504, a set of semantic chunks 506, a total budget 508, and a set of video scenes 510. The set of semantic chunks 506 may include a first semantic chunk 506A and a second semantic chunk 506B.

[0091] The set of input video frames 502 and the input textual query 504 may be received. Further, the total budget 508 associated with the retrieval of the set of video scenes 510 may be received. In one or more embodiments, the processor 202 may be configured to receive the set of input video frames 502 and the input textual query 504. The processor 202 may further receive the total budget 508.

[0092] In an instance, the input textual query 504 may include a statement including an instruction to determine the output for the statement. The relative importance ratio 414 associated with each chunk of the set of semantic chunks 506 may be determined based on the statement including the instruction. In one or more embodiments, the processor 202 may be configured to determine the relative importance ratio 414 associated with each chunk of the set of semantic chunks 506, based on the correspondence between the input textual query 504 and the set of semantic chunks 506.In an example, the input textual query 504 may include the statement prepared to include the instruction as: “Is there any accident?”. Based on this prepared statement including the instruction, the relative importance ratio 414 associated with each chunk of the set of semantic chunks 506 maybe determined. For example, the relative importance ratio 414 associated with the first semantic chunk 506A and the second semantic chunk 506B may be determined as “0.2” and “0.8”, based on the prepared statement including the instruction.

[0093] The processor 202 may be configured to determine the first budget 416 associated with each chunk of the set of semantic chunks 506, based on the total budget 508 and the relative importance ratio 414. For example, the total budget 508 of “5” associated with the retrieval of the set of video scenes 510 maybe received. Based on the total budget 508 of “5” and the relative importance ratio values of “0.2” and “0.8” (associated with the first semantic chunk 506A and the second semantic chunk 506B), the first budget 416 associated with each chunk of the set of semantic chunks 506 may be determined. In an example, the first budget 416 associated with the first semantic chunk 506A may be determined as “1” (i.e., 5*0.2=1) and the first budget 416 associated with the second semantic chunk 506B may be determined as “4” (i.e., 5*0.8=4).

[0094] The processor 202 may be configured to condense the set of input video frames 502 to determine the set of video scenes 510 based on the first budget 416. For example, the processor 202 may be configured to condense, using the condensation unit 314, the set of input video frames 502 to determine the set of video scenes 510 based on the first budget 416. The set of video scenes 510 may be the condensed video image frames retrieved from the set of input video frames 502. For example, in case, the first budget 416 associated with the first semantic chunk 506A and the second semantic chunk 506B are determined as “1” and “4”, the set of video scenes 510 may include “1” video framecorresponding to the first semantic chunk 506A and “4” video frames corresponding to the second semantic chunk 506B.

[0095] The LMM model 120 maybe applied on the set ofvideo scenes 510 to determine an output associated with the input textual query 504. The processor 202 may be configured to determine the output associated with the input textual query 504, based on the application of the LMM model 120 on the set of video scenes 510. For example, the output may be the set of video scenes 510 representing the images of any accident event that occurred around a traffic or sparse area.

[0096] It should be noted that the scenario 500 of FIG. 5 is for exemplary purposes and should not be construed to limit the scope of the disclosure.

[0097] FIG. 6 is a diagram that illustrates an exemplary scenario for generation of condensed video frames based on a relative importance ratio and a first budget, in accordance with an embodiment of the disclosure. FIG.6 is described in conjunction with elements from FIG. 1, FIG. 2, FIG. 3, FIG. 4, and FIG. 5. With reference to FIG. 6, there is shown an exemplary scenario 600 for the generation of the condensed video frames (for example, the retrieved video image frames 418) based on the relative importance ratio 414 and the first budget 416. The scenario 600 may include a set of input video frames 602, the operation for semantic chunking 402, the relative importance ratio 414, the first budget 416, the condensation unit 314, a set of semantic chunks 604, an input textual query 606, a total budget 608, and a set of video scenes 610. The set of semantic chunks 604 may include a first semantic chunk 604A, a second semantic chunk 604B, a third semantic chunk 6040, and a fourth semantic chunk 604D.

[0098] The set of input video frames 602 and the input textual query 606 may be received. Further, the total budget 608 associated with the retrieval of the set of videoscenes 118 may be received. In one or more embodiments, the processor 202 may be configured to receive the set of input video frames 602 and the input textual query 504.

[0099] In an instance, the input textual query 606 may include a statement including an instruction to determine the output for the statement. The relative importance ratio 414 associated with each chunk of the set of semantic chunks 604 may be determined based on the statement including the instruction. In one or more embodiments, the processor 202 may be configured to determine the relative importance ratio 414 associated with each chunk of the set of semantic chunks 604, based on the correspondence between the input textual query 606 and the set of semantic chunks 604. In an example, the input textual query 606 may include the statement prepared to include the instruction as: “What happened in the video?”. Based on this prepared statement including the instruction, the relative importance ratio 414 associated with each chunk of the set of semantic chunks 604 may be determined. Based on this prepared statement including the instruction, the relative importance ratio 414 associated with each chunk of the set of semantic chunks 604 may be determined. For example, values of the relative importance ratio 414 associated with the first semantic chunk 604A, the second semantic chunk 604B, the third semantic chunk 604C, and the fourth semantic chunk 604D may be determined as “0.24”, “0.26”, “0.26”, and “0.24” respectively, based on the prepared statement including the instruction.

[0100] The processor 202 may be configured to determine the first budget 416 associated with each chunk of the set of semantic chunks 604, based on the total budget 608 (e.g., “10”) and the relative importance ratio 414. For example, the first budget 416 for the first semantic chunk 604A may be determined as “2” (i.e., 0.24*10). The processor 202 may be further configured to condense the set of input video frames 602 to determine the set of video scenes 610 (i.e., the condensed video image frames) based onthe first budget 416. In an example, in case the total budget 608 is “10”, a count of the total number of video frames in the set of video scenes 610 may be “10”.

[0101] The LMM model 120 maybe applied on the set ofvideo scenes 510 to determine the output associated with the input textual query 112. The processor 202 may be configured to determine the output associated with the input textual query 606, based on the application of the LMM model 120 on the set of video scenes 610. For example, the output may be the set of video scenes 610 representing the images of any event that occurred around a specific area at a particular time instance.

[0102] It should be noted that the scenario 600 of FIG.6 is for exemplary purposes and should not be construed to limit the scope of the disclosure.

[0103] FIG. 7A is a diagram that illustrates an exemplary execution pipeline for generation of a set of semantic chunks from a set of input video frames based on a queryagnostic semantic chunking technique, in accordance with an embodiment of the disclosure. FIG. 7A is described in conjunction with elements from FIG. 1, FIG. 2, FIG. 3, FIG.4, FIG. 5, and FIG.6. With reference to FIG.7A, there is shown an exemplary execution pipeline 700A for the generation of the set of semantic chunks 116 from the set of input video frames 110, based on a query-agnostic semantic chunking technique. The execution pipeline 700A may include a set of input video frames 702A, first video frames 704A, an operation for caption generation 706A, a captioning model 708A, and a first set of captions 710A. The execution pipeline 700A may further include an operation for similarity scores computation 712A, similarity scores 714A, a similarity score 716A, a similarity score 718A, an operation for comparison of a similarity score 720A, and an operation for input video frames splitting 722A.

[0104] The set of input video frames 702A and the input textual query 606 may be received. Further, the total budget 508 associated with the retrieval of the set of videoscenes 118 may be received. The processor 202 may be configured to receive the set of input video frames 702A and the input textual query 606. In an example, a semantic chunk of the set of semantic chunks 116 may include video frames, such as, the first video frames 704A.

[0105] At 706A, the operation for caption generation may be executed. The processor 202 may be configured to generate, by use of the captioning model 708A, each caption of the first set of captions 710A associated with each corresponding video frame of the set of input video frames 702A. For example, the first set of captions 710A may be represented as “Caption 1” associated with a first video frame, “Caption 2” associated with a second video frame, and “Caption 3” associated with a third video frame. In an example, the first video frames 704A may include the first video frame, the second video frame, and the third video frame. The generation of the first set of captions 710A associated with the set of input video frames 702A may correspond to the query-agnostic semantic chunking technique.

[0106] At 712A, the operation for computation of similarity scores may be executed. The processor 202 may compute the similarity scores 714A. The similarity scores 714A may include the similarity score 716A and the similarity score 718A. The similarity score 716A (for example, a first similarity score) may be a similarity metric between a first caption (for example, the “Caption 1”) and a second caption (for example, the “Caption 2”) of the first set of captions 710A. The similarity score 718A (for example, a second similarity score) may be a similarity metric between the second caption (for example, the “Caption 2”) and a third caption (for example, the “Caption 3”) of the first set of captions 710A. In an example, the similarity score 716A may be “4” and the similarity score 718A may be “1”.At 720A, the operation for comparison of a similarity score may be executed. The processor 202 may be configured to compare a similarity score (e.g., the similarity score 716A and the similarity score 718A) with a threshold. The comparison of the similarity score may correspond to comparison of a similarity between the semantic content of adjacent frames. In case, it is determined that the similarity score is greater than the threshold, the processor 202 may split the set of input video frames 702A, such that the frames between which the similarity score is calculated may be assigned the same chunk. For example, in case, the similarity score 716A between the first caption (for the first frame) and the second caption (for the second frame) is “4” and the threshold is “3”, the processor 202 may split the set of input video frames 702A, such that the first frame and the second frame may be assigned the same chunk. However, in case, the threshold is “3” and the similarity score 718A between the second caption (for the second frame) and the third caption (for the third frame) is “1”, the processor 202 may assign the second frame and the third frame different chunks.

[0107] At 722A, the operation for splitting of input video frames may be executed. The processor 202 may determine the similarity score 714A (e.g., the first similarity score) between the first caption and the second caption, based on the comparison of the first caption with the second caption. Further, the set of input video frames 702A may be split into the set of semantic chunks 116 based on the first similarity score. In an example, the processor 202 may be configured to determine whether the similarity score 716A between the first caption and the second caption is greater than a threshold. Based on the determination that the first similarity score is greater than the threshold, the set of input video frames 702A may be split into the set of semantic chunks 604. In one or more embodiments, the processor 202 may be configured to split the set of input video frames702A into the set of semantic chunks 604, based on the determination that the first similarity score (e.g., the similarity score 716A] is greater than the threshold.

[0108] In some embodiments, the processor 202 may be configured to identify a chunk of the set of semantic chunks 604 based on the determination that the first similarity score between the first caption and the second caption is greater than the threshold. The processor 202 may be further configured to split the set of input video frames 702A into the set of semantic chunks 604, based on the identification of the semantic chunk of the set of semantic chunks 604. For example, the processor 202 may identify a combination of the first frame and the second frame as a first chunk and the third frame as a second chunk.

[0109] FIG. 7B is a diagram that illustrates an exemplary execution pipeline for generation of a set of captions associated with a set of input video frames based on a query-aware semantic chunking technique. FIG. 7B is described in conjunction with elements from FIG. 1, FIG. 2, FIG. 3, FIG. 4, FIG. 5, FIG. 6, and FIG. 7A. With reference to FIG. 7B, there is shown an exemplary execution pipeline 700B for the generation of the set of captions associated with the set of input video frames 110. The execution pipeline 700B may include the input textual query 112, a set of input video frames 702B, an operation for a set of entities 704B, a target entity 706B, an operation for first set of caption generation 708B, and a first set of captions 710B.

[0110] The set of input video frames 702B and the input textual query 112 may be received. Further, the total budget 508 associated with the retrieval of the set of video scenes 118 may be received. In one or more embodiments, the processor 202 may be configured to receive the set of input video frames 702B, the input textual query 112, and the total budget 508.At 704B, the operation for the set of entities extraction may be executed. The input textual query 112 may include a statement including an instruction to determine the output for the statement. The relative importance ratio 414 associated with each chunk of the set of semantic chunks 604 may be determined based on the statement including the instruction. In one or more embodiments, the processor 202 may be configured to extract the set of entities 704B from the input textual query 112. In an example, the input textual query 112 may include the statement prepared to include the instruction as : “What was a dog playing with?”. The set of entities 704B may be extracted from this prepared statement including the instruction. In an example, the set of entities 704B may correspond to a set of subjects and / or a set of objects in the input textual query 112. The processor 202 may determine the set of entities 704B based on an application of natural language processing techniques (such as, parts-of-speech identification) on the input textual query 112. For example, the identified entity may be a “dog” in above case. Further, the processor 202 may be configured to determine the target entity 706B from the extracted set of entities 704B. In an example, the target entity 706B may be a subject, which may be identified from the set of entities 704B based on an application of natural language techniques (such as, parts-of-speech identification) on the input textual query 112.

[0111] At 708B, the operation for the first set of captions generation may be executed. Based on the determined target entity 706B, the first set of captions 710B associated with the set of input video frames 702B may be generated. In one or more embodiments, the processor 202 may be configured to generate the first set of captions 710B associated with the set of input video frames 702B, based on the target entity 706B. For example, based on the target entity 706B, “N” number of first set of captions 710B (which may be represented as “Caption 1”, “Caption 2”, “Caption 3”, ...and "Caption N”) maybe generated.The generation of the first set of captions 710B associated with the set of input video frames 702B based on the target entity 706B may correspond to the query-aware semantic chunking technique.

[0112] FIG. 7C is a diagram that illustrates an exemplary execution pipeline for determination of relative importance ratio based on a frame-based importance ratio determination technique. FIG. 7C is described in conjunction with elements from FIG. 1, FIG. 2, FIG. 3, FIG. 4, FIG. 5, FIG. 6, FIG. 7A, and FIG. 7B. With reference to FIG. 7C, there is shown an exemplary execution pipeline 700C for the determination of relative importance ratio. The execution pipeline 700C may include the input textual query 112, a set of input video frames 702C, an operation for similarity scores computation 704C, similarity scores 706C, an operation for average similarity value computation 708C, average similarity values 710C, and normalized values 712C.

[0113] The set of input video frames 702C and the input textual query 112 may be received. Further, the total budget 508 associated with the retrieval of the set of video scenes 118 may be received. The processor 202 may be configured to receive the set of input video frames 7020 and the input textual query 112. In an example, a semantic chunk of the set of semantic chunks 116 may include video frames, such as, the first video frames 704A.

[0114] At 7040, the operation for the similarity scores computation may be executed. The processor 202 may be configured to generate, by use of the captioning model 708A, each caption of the first set of captions 710A associated with each corresponding video frame of the set of input video frames 7020. The processor 202 may compute the similarity scores 7060 between each video frame of the set of input video frames 7020 and the input textual query 112. For example, the similarity scores 7060 may correspond to a second similarity score computed between each video frame of the set of input videoframes 702 and the input textual query 112. In one or more embodiments, the processor 202 may be configured to determine the second similarity score between each video frame of the set of input video frames 702 and the input textual query 112. For example, the set of input video frames 702C maybe split into three chunks. Values associated with the similarity scores corresponding to the set of input video frames 702C in the three chunks may be determined as “(3, 2)”, “(3, 4, 5, 4, 6, 3)”, and “(4, 2, 3)”, respectively.

[0115] At 708C, the operation for the average similarity value computation may be executed. The average similarity value maybe computed corresponding to each chunk of the set of semantic chunks 604. In an example, the processor 202 may be configured to compute the average similarity value 710C corresponding to each chunk of the set of semantic chunks 604, based on the second similarity score. For example, the average similarity value 710C corresponding to each chunk may be determined as: “2.5”, “4.2”, and “3.0” for the three chunks (into which the set of input video frames 702C may be split).

[0116] The determined average similarity value 710C may be then normalized to determine a first normalized score associated with each chunk of the set of semantic chunks 604. In one embodiment, the processor 202 may be configured to normalize the average similarity value corresponding to each chunk of the set of semantic chunks 604 using a normalization function, to determine the first normalized score associated with each chunk of the set of semantic chunks 604. For example, the determined average similarity values 710C maybe normalized using various normalization techniques, such as, but not limited to, a linear scaling technique, a Z-score scaling technique, a logarithmic scaling technique, a clipping technique, and a min-max normalization technique. In an example, the first normalized score that may be determined for the three chunks based on a total similarity value. The total similarity value may be calculated as a total sum ofthe average similarity values corresponding to the three chunks (i.e., 2.5+4.2+3.0=9.7). Based on the calculated total similarity value, the first normalized score for the three chunks may be determined as “0.26”, “0.43”, and “0.31” (i.e., 2.5 / 9.7=0.26, 4.2 / 9.7=0.43, and 3.0 / 9.7=0.31). The determination of the relative importance ratio 414 associated with each chunk of the set of semantic chunks 604 may be further based on the first normalized score. The determination of the relative importance ratio 414 based on the first normalized score may correspond to the frame-based importance ratio technique.

[0117] In some embodiments, the processor 202 may be configured to generate, by use of the captioning model 708A, each caption of a second set of captions associated with each corresponding semantic chunk of the set of semantic chunks 604. The processor 202 may be further configured to determine a third similarity score between each caption of the second set of captions and the input textual query 112. The processor 202 may further normalize the third similarity score corresponding to each chunk of the set of semantic chunks 604 to determine a second normalized score associated with each chunk of the set of semantic chunks 604. The determination of the relative importance ratio 414 associated with each chunk of the set of semantic chunks 604 may be further based on the second normalized score. The determination of the relative importance ratio 414 based on the second normalized score may corresponds to a caption-based importance ratio technique.

[0118] FIG. 8 is a diagram that illustrates a flowchart of an exemplary method for video scene retrieval based on a relative importance ratio, in accordance with an embodiment of the disclosure. FIG. 8 is described in conjunction with elements from FIG.

[0119] 1, FIG. 2, FIG. 3, FIG. 4, FIG. 5, FIG. 6, FIG. 7A, FIG. 7B, and FIG. 7C. With reference to FIG.

[0120] 8, there is shown an exemplary flowchart 800 of a method for video scene retrieval based on relative importance ratio. The flowchart 800 may include operations 802 to 820,which may be executed by the processor 202 (of FIG. 2) of the electronic device 102 (of FIG. 1). The flowchart 800 may start at 802 and proceed to 804.

[0121] At 804, a set of input video frames and an input textual query may be received. The processor 202 may be configured to receive the set of input video frames 110 and the input textual query 112. The input textual query 112 may be natural language text describing a task to be performed by the LMM model 120. The natural language text may be in the form of a structured instruction that may be interpreted and understood by the LMM model 120. The reception of the set of input video frames and the input textual query is described further, for example, in FIG. 3, FIG.4, FIG. 5, FIG. 6, FIG. 7A, FIG. 7B, and FIG. 70.

[0122] At 806, a total budget associated with a retrieval of a set of video scenes may be received. The processor 202 may be configured to receive the total budget 114 associated with the retrieval of the set of video scenes 118 or the video length. The total budget 114 may corresponds to the total count of video frames of the set of input video frames 110 to be included in the set of video scenes 118, or the size of the main memory or the GPU on the electronic device 102. The reception of the total budget associated with the retrieval of a set of video scenes or the video length is described further, for example, in FIG. 3, FIG. 4, FIG. 5, and FIG. 6.

[0123] At 808, the set of input video frames may be split into a set of semantic chunks. The processor 202 may be configured to split the set of input video frames 110 into the set of semantic chunks 116. The splitting of the set of input video frames 110 into the set of semantic chunks 116 is described further, for example, in FIG. 3, FIG. 4, FIG. 5, FIG. 6, and FIG. 7A.

[0124] At 810, a relative importance ratio associated with each chunk of the set of semantic chunks 116 may be determined, based on the correspondence between theinput textual query 112 and the set of semantic chunks 116. The processor 202 may be configured to determine the relative importance ratio 414 associated with each chunk of the set of semantic chunks 116, based on the correspondence between the input textual query 112 and the set of semantic chunks 116. The determination of the relative importance ratio 414 is described further, for example, in FIG. 3, FIG.4, FIG. 5, FIG. 6, and FIG. 70.

[0125] At 812, a first budget associated with each chunk of the set of semantic chunks may be determined, based on the total budget and the relative importance ratio. The processor 202 maybe configured to determine the first budget 416 associated with each chunk of the set of semantic chunks 116, based on the total budget 114 and the relative importance ratio 414. The first budget 416 may corresponds to the count of video frames of the set of input video frames 110 to be included in the corresponding semantic chunk. The determination of the first budget is described further, for example, in FIG. 3, FIG. 4, FIG. 5, and FIG. 6.

[0126] At 814, the set of video scenes may be determined based on the first budget. The processor 202 may be configured to condense the set of input video frames 110 to determine the set of video scenes 118 based on the first budget 416. The determination of the set of video scenes based on the first budget is described further, for example, in FIG. 3, FIG. 5, and FIG. 6.

[0127] At 816, the LMM model may be applied on the set of video scenes and the input textual query. The processor 202 may be configured to apply the LMM model 120 on the set of video scenes 118 and the input textual query 112. The application of the LMM model 120 on the set of video scenes 118 and the input textual query 112 is described further, for example, in FIG. 3 and FIG.4.At 818, an output associated with the input textual query may be determined based on the application of the LMM model. The processor 202 may be configured to determine the output associated with the input textual query 112 based on the application of the LMM model 120. The output may correspond to the retrieved video image frames 316. The retrieved video image frames 316 may be the input video frames of the set of input video frames 110 retrieved based on the relative importance ratio 414 associated with each chunk of the set of semantic chunks 116. The detection of the set of biases is described further, for example, in FIG. 3, FIG. 4, FIG. 5, and FIG. 6.

[0128] At 820, rendering of the output associated with the input textual query may be controlled. The processor 202 may be configured to control the rendering of the output associated with the input textual query 112. The control of the rendering of the output associated with the inputtextual query is described further, for example, in FIG.3. Control may pass to end.

[0129] Although the flowchart 800 is illustrated as discrete operations, such as 804, 806, 808, 810, 812, 814, 816, 818, and 820, the disclosure is not so limited. However, in certain embodiments, such discrete operations may be further divided into additional operations, combined into fewer operations, or eliminated, depending on the particular implementation without detracting from the essence of the disclosed embodiments.

[0130] Various embodiments of the disclosure may provide one or more non-transitory computer-readable storage medium configured to store instructions that, in response to being executed, cause a system (such as, the example electronic device 102) to perform a set of operations. The set of operations may include receiving a set of input video frames and an input textual query. The set of operations may further include receiving a total budget associated with the retrieval of a set of video scenes. The set of operations may further include splitting the set of input video frames into a set ofsemantic chunks. The set of operations may further include determining a relative importance ratio associated with each chunk of the set of semantic chunks, based on the correspondence between the input textual query and the set of semantic chunks. The set of operations may further include determining a first budget associated with each chunk of the set of semantic chunks, based on the total budget and the relative importance ratio. The set of operations may further include condensing the set of input video frames to determine the set of video scenes based on the first budget. The set of operations may further include applying a large multi-modal (LMM) model on the set of video scenes and the input textual query. The set of operations may further include determining an output associated with the input textual query based on the application of the LMM model. The set of operations may further include controlling rendering of the output associated with the input textual query.

[0131] As used in the present disclosure, the terms “module” or “component” may refer to specific hardware implementations configured to perform the actions of the module or component and / or software objects or software routines that may be stored on and / or executed by general purpose hardware (e.g., computer-readable media, processing devices, etc.) of the computing system. In some embodiments, the different components, modules, engines, and services described in the present disclosure may be implemented as objects or processes that execute on the computing system (e.g., as separate threads). While some of the system and methods described in the present disclosure are generally described as being implemented in software (stored on and / or executed by general purpose hardware), specific hardware implementations or a combination of software and specific hardware implementations are also possible and contemplated. In this description, a “computing entity” may be any computing system aspreviously defined in the present disclosure, or any module or combination of modulates running on a computing system.

[0132] Terms used in the present disclosure and especially in the appended claims (e.g., bodies of the appended claims) are generally intended as “open” terms (e.g., the term “including” should be interpreted as “including, but not limited to,” the term “having” should be interpreted as “having at least,” the term “includes” should be interpreted as “includes, but is not limited to,” etc.).

[0133] Additionally, if a specific number of an introduced claim recitation is intended, such an intent will be explicitly recited in the claim, and in the absence of such recitation no such intent is present. For example, as an aid to understanding, the following appended claims may contain usage of the introductory phrases "at least one" and "one or more" to introduce claim recitations. However, the use of such phrases should not be construed to imply that the introduction of a claim recitation by the indefinite articles "a" or "an" limits any particular claim containing such introduced claim recitation to embodiments containing only one such recitation, even when the same claim includes the introductory phrases "one or more" or "at least one" and indefinite articles such as "a" or "an" (e.g., “a” and / or “an” should be interpreted to mean “at least one” or “one or more”); the same holds true for the use of definite articles used to introduce claim recitations.

[0134] In addition, even if a specific number of an introduced claim recitation is explicitly recited, one of ordinary skill in the art will recognize that such recitations should be interpreted to mean at least the recited number (e.g., the bare recitation of "two recitations," without other modifiers, means at least two recitations, or two or more recitations). Furthermore, in those instances where a convention analogous to “at least one of A, B, and C, etc.” or “one or more of A, B, and C, etc.” is used, in general such aconstruction is intended to include A alone, B alone, C alone, A and B together, A and C together, B and C together, or A, B, and C together, etc.

[0135] Further, any disjunctive word or phrase presenting two or more alternative terms, whether in the description, claims, or drawings, should be understood to contemplate the possibilities of including one of the terms, either of the terms, or both terms. For example, the phrase “A or B” should be understood to include the possibilities of “A” or “B” or “A and B.”

[0136] All examples and conditional language recited in the present disclosure are intended for pedagogical objects to aid the reader in understanding the present disclosure and the concepts contributed by the inventor to furthering the art and are to be construed as being without limitation to such specifically recited examples and conditions. Although embodiments of the present disclosure have been described in detail, various changes, substitutions, and alterations could be made hereto without departing from the spirit and scope of the present disclosure.

Claims

We claim:

1. A method, executed by a system, comprising:receiving a set of input video frames and an input textual query; receiving a total budget associated with a retrieval of a set of video scenes; splitting the set of input video frames into a set of semantic chunks; determining a relative importance ratio associated with each chunk of the set of semantic chunks, based on the correspondence between the input textual query and the set of semantic chunks;determining a first budget associated with each chunk of the set of semantic chunks, based on the total budget and the relative importance ratio;condensing the set of input video frames to determine the set of video scenes based on the first budget;applying a large multi-modal (LMM) model on the set of video scenes and the input textual query;determining an output associated with the input textual query based on the application of the LMM model; andcontrolling a rendering of the output associated with the input textual query.

2. The method according to claim 1, further comprising:determining a set of visual embeddings associated with the set of input video frames, whereinthe splitting of the set of input video frames into the set of semantic chunks is based on the set of visual embeddings.

3. The method according to claim 1, further comprising:determining a set of textual embeddings associated with the input textual query, whereinthe determination of the relative importance ratio is based on the set of semantic chunks and the set of textual embeddings.

4. The method according to claim 1, further comprising:generating, by use of a captioning model, each caption of a first set of captions associated with each corresponding video frame of the set of input video frames; comparing a first caption of a first video frame with a second caption of a second video frame, whereinthe first set of captions includes the first caption and the second caption, andthe set of input video frames includes the first video frame and the second video frame neighboring the first video frame in the set of input video frames; anddetermining a first similarity score between the first caption and the second caption, based on the comparison of the first caption with the second caption, wherein the splitting of the set of input video frames into the set of semantic chunks is based on the first similarity score.

5. The method according to claim 4, further comprising:determining whether the first similarity score between the first caption and the second caption is greater than a threshold; andidentifying a chunk of the set of semantic chunks based on the determination that the first similarity score between the first caption and the second caption is greater than the threshold, whereinthe splitting of the set of input video frames into the set of semantic chunks is further based on the identification of the chunk.

6. The method according to claim 4, wherein the generation of the first set of captions associated with the set of input video frames corresponds to a query-agnostic semantic chunking technique.

7. The method according to claim 4, further comprising:extracting a set of entities from the input textual query; anddetermining a target entity from the set of entities, whereinthe generation of the first set of captions associated with the set of input video frames is based on the target entity.

8. The method according to claim 7, wherein the generation of the first set of captions associated with the set of input video frames based on the target entity corresponds to a query-aware semantic chunking technique.

9. The method according to claim 1, further comprising:determining a second similarity score between each video frame of the set of input video frames and the input textual query;determining an average similarity value corresponding to each chunk of the set of semantic chunks, based on the second similarity score; andnormalizing the average similarity value corresponding to each chunk of the set of semantic chunks to determine a first normalized score associated with each chunk of the set of semantic chunks, whereinthe determination of the relative importance ratio associated with each chunk of the set of semantic chunks is further based on the first normalized score.

10. The method according to claim 9, wherein the determination of the relative importance ratio based on the first normalized score corresponds to a frame-based importance ratio determination technique.

11. The method according to claim 1, further comprising:generating, by use of a captioning model, each caption of a second set of captions associated with each corresponding chunk of the set of semantic chunks; determining a third similarity score between each caption of the second set of captions and the input textual query; andnormalizing third similarity score corresponding to each chunk of the set of semantic chunks to determine a second normalized score associated with each chunk of the set of semantic chunks, whereinthe determination of the relative importance ratio associated with each chunk of the set of semantic chunks is further based on the second normalized score.

12. The method according to claim 11, wherein the determination of the relative importance ratio based on the second normalized score corresponds to a captionbased importance ratio determination technique.

13. The method according to claim 1, wherein the first budget associated with each chunk of the set of semantic chunks corresponds to a count of video frames, of the set of video frames, to be included in the corresponding chunk.

14. The method according to claim 1, wherein the total budget corresponds to a total count of video frames, of the set of video frames, to be included in the set of video scenes, or a size of main memory or a graphical-processor unit (GPU) on the electronic device.

15. One or more non-transitory computer-readable storage mediums storing instructions that, in response to being executed, causes an electronic device to perform operations, the operations comprising:receiving a set of input video frames and an input textual query; receiving a total budget associated with a retrieval of a set of video scenes; splitting the set of input video frames into a set of semantic chunks; determining a relative importance ratio associated with each chunk of the set of semantic chunks, based on the correspondence between the input textual query and the set of semantic chunks;determining a first budget associated with each chunk of the set of semantic chunks, based on the total budget and the relative importance ratio;condensing the set of input video frames to determine the set of video scenes based on the first budget;applying a large multi-modal (LMM) model on the set of video scenes and the input textual query;determining an output associated with the input textual query based on the application of the LMM model; andcontrolling a rendering of the output associated with the input textual query.

16. The one or more non-transitory computer-readable storage mediums according to claim 15, wherein the operations further comprising:generating, by use of a captioning model, each caption of a first set of captions associated with each corresponding video frame of the set of input video frames; comparing a first caption of a first video frame with a second caption of a second video frame, whereinthe first set of captions includes the first caption and the second caption, andthe set of input video frames includes the first video frame and the second video frame neighboring the first video frame in the set of input video frames; anddetermining a first similarity score between the first caption and the second caption, based on the comparison of the first caption with the second caption, wherein the splitting of the set of input video frames into the set of semantic chunks is based on the first similarity score.

17. The one or more non-transitory computer-readable storage mediums according to claim 16, wherein the operations further comprising:extracting a set of entities from the input textual query; anddetermining a target entity from the set of entities, whereinthe generation of the first set of captions associated with the set of input video frames is based on the target entity.

18. The one or more non-transitory computer-readable storage mediums according to claim 15, wherein the operations further comprising:determining a second similarity score between each video frame of the set of input video frames and the input textual query;determining an average similarity value corresponding to each chunk of the set of semantic chunks, based on the second similarity score; andnormalizing the average similarity value corresponding to each chunk of the set of semantic chunks to determine a first normalized score associated with each chunk of the set of semantic chunks, whereinthe determination of the relative importance ratio associated with each chunk of the set of semantic chunks is further based on the first normalized score.

19. The one or more non-transitory computer-readable storage mediums according to claim 15, wherein the operations further comprising:generating, by use of a captioning model, each caption of a second set of captions associated with each corresponding chunk of the set of semantic chunks;determining a third similarity score between each caption of the second set of captions and the input textual query; andnormalizing third similarity score corresponding to each chunk of the set of semantic chunks to determine a second normalized score associated with each chunk of the set of semantic chunks, whereinthe determination of the relative importance ratio associated with each chunk of the set of semantic chunks is further based on the second normalized score.

20. A system, comprising:one or more memory storages storing instructions, andone or more processors, coupled to the one or more memory storages, executing the instructions to perform a process comprising:receiving a set of input video frames and an input textual query; receiving a total budget associated with a retrieval of a set of video scenes;splitting the set of input video frames into a set of semantic chunks; determining a relative importance ratio associated with each chunk of the set of semantic chunks, based on the correspondence between the input textual query and the set of semantic chunks;determining a first budget associated with each chunk of the set of semantic chunks, based on the total budget and the relative importance ratio;condensing the set of input video frames to determine the set of video scenes based on the first budget;applying a large multi-modal (LMM) model on the set of video scenes and the input textual query;determining an output associated with the input textual query based on the application of the LMM model; andcontrolling a rendering of the output associated with the input textual query.

Images