Systems and techniques for processing long-form video content
A multimodal framework integrating vision and audio language models addresses challenges in processing long-form video content by enhancing understanding and retrieval through improved frame analysis and timestamping, particularly for longer videos.
Patent Information
- Application Number
- PCT/EP2026/063799
- Authority / Receiving Office
- WO · WO
- Patent Type
- Applications
- Current Assignee / Owner
- Priority Date
- 2025-05-12
- Filing Date
- 2026-05-12
- Publication Date
- 2026-11-19
AI Technical Summary
Existing large language models (LLMs) face challenges in processing and understanding long-form video content, particularly in terms of reducing hallucinations, improving video content reasoning, and addressing computational bottlenecks, especially when dealing with proprietary data.
A novel multimodal architecture framework that integrates vision and audio language models to process multi-format video data, including metadata extraction, caption generation, and vectorization, with enhanced search capabilities and a knowledge graph for evaluation.
Enhances the understanding and retrieval of long-form video content by integrating visual and auditory data, improving accuracy in frame analysis and timestamping, and enabling more faithful summaries and key frame identification, particularly for longer videos.
Smart Images

Figure EP2026063799_19112026_PF_FP_ABST
Abstract
Description
[0001] SYSTEMS AND TECHNIQUES FOR PROCESSING LONG-FORM VIDEO CONTENT
[0002] Field of the Disclosure
[0003] The present disclosure relates to methods and systems for processing digital content including video data. In particular, although not exclusively, the present disclosure relates to methods for processing, summarizing, and searching of long-form video content.
[0004] Background
[0005] Large Language Models (LLMs) such as GPT-4 (Ouyang et al., 2022) and LaMDA (Thoppilan et al., 2022) have substantially enhanced public access to complex information, particularly in sectors such as healthcare and public services. These models are celebrated for their capability to demystify intricate information, assisting in tasks ranging from routine inquiries to aiding clinical decision-making (Brown et al., 2020a). GenAI solutions (Vaswani et al., 2017; Brown et al., 2020b; Ramesh et al., 2021 ; Rombach et al., 2022) enable more efficient content creation and streamlined operations. This technology is revolutionizing content production by increasing reuse, reducing costs, and boosting profitability. It reduces the production time from weeks to days, simplifies the creation and approval of targeted content, and ensures brand consistency and quality across markets and channels through built-in quality controls, specific guidelines, and regulatory standards (Gozalo-Brizuela and Garrido-Merchan, 2023). Practical use cases including chat agents (OpenAI, 2024), briefing agencies (OpenAI, 2023), document searching (Lewis et al., 2021), code debug (Donvir et al., 2024), text / video summarization (Li et al., 2025), personalized message assistance (Matz et al., 2024), image adjustments (Li et al., 2024) and document quality checks (Yang et al., 2024) deliver transformative impacts.
[0006] Research has primarily focused on evaluating the capabilities of LLMs (Singhal et al., 2022) or leveraging the stochasticity in sampled responses to assess the factuality of generated outputs (Goodrich et al., 2019). However, challenges remain in producing reliable Vision Language Models (VLMs) with growing volume of video data, reducing hallucinations and memory cost, improving the quality of video content reasoning, and addressing computational bottlenecks in long video understanding (Qu et al., 2025). Industrializing VLMs for large-scale data, especially using closed-source models on proprietary data, is becoming increasingly urgent.
[0007] The present invention has been devised in light of the above considerations.
[0008] Summary of the Disclosure
[0009] To address these challenges, the present disclosure draws on lessons learned from a real-world industrial multimodal GenAI platform and introduces a novel, comprehensive framework developed in the industry. Novel contributions forming part of the present disclosure include, but are not limited to: (i) a practical multimodal architecture framework for large-scale processing of multi-format data, which was tested on 200,000 PDFs, 25,326 videos across 8 various formats (e.g., MP4, M4V, QuickTime, WMV, WebM, MSVideo, MPG, and 3GPP), and multilingual audio files covering more than 20 languages; (ii) an improved search engine with dual search capabilities; (iii) key technical findings and actionable insights in the VLMdomain (specifically: multimodality, attention mechanism, temporal reasoning, and the existence of a tradeoff of long video split and cut); and (iv) a new task domain (summarisation and key frame identification) on the state-of-the-art Video MME benchmark (Fu et al., 2024a) along with an evaluation schema based on knowledge graphs.
[0010] An aspect of the present disclosure relates to a method and system for processing digital content, which significantly enhances the extraction, translation, tagging, and vectorization of various data types, including at least videos, and optionally additional types of documents such as audio recordings, and / or documents. Thus, according to a first aspect, there is provided a computer-implemented method for processing video content, the method comprising: generating one or more captions for the video content using a vision language model; extracting metadata associated with the video content; generating, using an audio language model, a transcript from audio content extracted from or associated with the video content; and generating a vector representation of the video content based at least in part on the metadata, the transcript, and the one or more captions using a large language model; and storing, in a database, the vector representation in association with one or more tags generated based on the captions and the metadata. The present inventors have shown that the combination of visual and auditory data enhances understanding of the video content. Specifically, the methods describe enable the generation of a representation associated with video content that enables improved querying and reusing of the content by enabling reasoning on the content of the database with improved performance than was previously possible. The audio content may comprise native audio associated with the video, or a voiceover description of the video.
[0011] The combination of audio and video models, herein referred to as multimodality, significantly improves the content that is generated from the video content, particularly when this content is designed to enable reasoning tasks and querying on the content of the video, such as frame analysis and timestamping. By integrating both visual and auditory data, the system can provide a more comprehensive understanding of the video content. For instance, using Large Language Models (LLMs) and Audio Language Models (ALMs) together allows the system to generate more faithful summaries and key frame information that enhance the overall understanding of the video content. This multimodal approach is particularly beneficial for longer videos, where the integration of metadata and audio (e.g. voiceovers) can provide richer context and more accurate content representation. The tags are created as part of content enrichment and are stored as part of vector DB to create overall communities like structure (inspired from Graph representation) to facilitate similar type of document preference if they are part of same segments. When a search query is performed on the content, it can be first transformed into know tags as explained further below and to apply auto filters based on those tags to better localize content and then apply hybrid ranks based on keyword and semantic based query outcome.
[0012] The method may further comprise: processing one or more digital document files comprising text, image content and / or tabular content, said processing comprising extracting, from said digital document files, reference data comprising one or more of: text, image content, tabular content; generating a vector representation of the reference data; and storing the vector representation in the database. The method may further comprise extracting metadata associated with the digital document files, the metadata comprising one or more tags based on the reference data.The one or more captions may comprise one or both of: a summary of the video content, and information identifying one or more key frames of the video content. The captions may comprise descriptive captions that enhance the understanding of the visual content. For example, for video content, the VLM may generate captions that include a summary and key frame information (i.e. information identifying one or more key frames of the video content), providing a comprehensive semantically accurate overview of the video.
[0013] The information identifying one or more key frames of the video content may comprise, for each of one or more key frames: (i) information uniquely identifying the key frame location in the video content, optionally comprising a timestamp, and (ii) information describing the content of the keyframe.
[0014] The method may further comprise processing the video content with a video converter (e.g., FFmpeg) to extract audio content from the video content.
[0015] In embodiments, generating a vector representation of the video content based at least in part on the metadata, the transcript, and the one or more captions using a large language model comprises: generating, with the large language model, revised captions, optionally including one or both of: a revised summary and revised information identifying one or more key frames based on the metadata, the transcript, and the one or more captions. In embodiments, the method may further comprise translating the one or more captions and / or the transcript from a first language to a second language using a machine translation model. Additionally, the method may optionally include translating the captions and / or the transcript from a first language to a second language using a multilingual model for machine translation. This may be any machine translation model, including generative models.
[0016] The metadata may comprise a plurality of attributes of the video content selected from: a length of the video, a resolution of the video, a creator of the video, an orientation of the video, a rotation of the video, a frame rate of the video, a frame size of the video, a graphics mode of the video, a format of the video, a compression of the video, a creation date of the video. In embodiments, extracting the metadata comprises generating one or more tags based on the one or more captions by processing, with a large language model (LLM), the one or more captions.
[0017] In embodiments, each of the one or more tags is one of: an extractive tag, which is a tag generated using a keyword matching process and a list of keywords and optionally synonyms thereof, optionally wherein extractive tags includes one or both of a product tag and an indication tag, wherein a product tag is a tag that identifies a specific therapeutic product in a list of products, and an indication tag is a tag that identifies a clinical indication in a list of clinical indications; or an abstractive tag, which is a tag generated using contextual semantic analysis and a large language model, optionally wherein abstractive tags includes one or more of: an audience tag, a content purpose tag, a content complexity tag, a non-clinical topic tag, and a clinical topic tag, wherein an audience tag is a tag that identifies an intended audience, a content purpose tag is a tag that identified a purpose of the document, a content complexity tag is a tag that indicates a level of technical detail, a non-clinical topics tag is a tag that identifies a specific non-clinical category of content, and a clinical topics tag is a tag that identifies a specific category of medical or scientific content.
[0018] The method may also optionally include generating a knowledge graph for the video content based on the captions. This knowledge graph is a visual representation of the content, providing an objective,reproducible way to appraise the output of and compare the performance of competing visual language models used to process the video content. Thus, in embodiments, the method further comprises generating a knowledge graph representing the one or more revised captions, wherein each element of the revised caption is associated with a node, and nodes associated with elements of the revised caption are connected with a root node indicating the type of the element. In embodiments, the elements comprise key frames and the knowledge graph is displayed to a user through a user interface using a layout that places nodes associated with key frames located closer to each other in the video content closer together than nodes associated with key frames located further from each other in the video content, and / or wherein the method comprises generating revised captions using captions generated with two or more different vision language models thereby obtaining corresponding two or more sets of revised captions, and generating a knowledge graph for each set of revised captions.
[0019] The use of attention mechanisms is often essential for processing video data, especially fortasks like frame analysis and timestamping. As part of the present disclosure, Scaled Dot-Product Attention (SDPA) has been shown to unexpectedly outperform Flash Attention in videos longer than two minutes. This insight allows for a choice of attention mechanism that enhances the accuracy of long video analysis, as well as improving computational efficiency.
[0020] In embodiments, the video content is a video of at least 30 minutes. In embodiments, the method does not comprise cutting or compressing the video content. In embodiments, the method comprises reducing the number of frames per second as the only data reduction step applied on the video content prior to generating the one or more captions for the video content using the vision language model.
[0021] In embodiments, the vision language model is an attention-based model that uses scaled dot product attention.
[0022] The present inventors have also shown that the combination of visual and auditory data enhances understanding of the video content. Specifically, the methods describe enable the generation of a representation associated with video content that enables improved querying and reusing of the content by enabling reasoning on the content of the database with improved performance than was previously possible. The audio content may comprise native audio associated with the video, or a voiceover description of the video. The combination of audio and video models, herein referred to as multimodality, significantly improves the content that is generated from the video content, particularly when this content is designed to enable reasoning tasks and querying on the content of the video, such as frame analysis and timestamping. By integrating both visual and auditory data, the system can provide a more comprehensive understanding of the video content. For instance, using Large Language Models (LLMs) and Audio Language Models (ALMs) together allows the system to generate more faithful summaries and key frame information that enhance the overall understanding of the video content. This multimodal approach is particularly beneficial for longer videos, where the integration of metadata and audio (e.g. voiceovers) can provide richer context and more accurate content representation. The tags are created as part of content enrichment and are stored as part of vector DB to create overall communities like structure (inspired from Graph representation) to facilitate similar type of document preference if they are part of same segments. When a search query is performed on the content, it can be first transformed into know tags as explained further below and toapply auto filters based on those tags to better localize content and then apply hybrid ranks based on keyword and semantic based query outcome.
[0023] In embodiments, the method further comprises: receiving a first search query from a user; generating one or more enhanced search queries based on the first search query; generating a vector representation for each of the one or more enhanced search queries; performing, with each vector representation, a vector search over the database; and obtaining a first set of search result from the vector search.
[0024] According to a second aspect, there is provided a computer-implemented method for searching and retrieving content from a database that has been obtained using the methods of any embodiment of the first aspect.
[0025] The method involves receiving a search query from a user and generating enhanced search queries based on the initial query. These enhanced queries are processed to generate vector representations, which are used to perform a vector search over the database. The search results are then provided to the user. The method may also include extracting keywords from the enhanced search queries and performing a keyword search over the database. The results from the vector search and keyword search may then be integrated according to a ranking algorithm, ensuring that the most relevant results are presented to the user.
[0026] In embodiments, the captions (e.g., summary and keyframes) are stored in a separate database from the documents themselves. The contents of the two databases can be linked. The contents of the two databases can be searched in parallel.
[0027] In embodiments of the first or second aspects, the method comprises extracting one or more keywords from the one or more enhanced search queries; performing, with the one or more keywords, a keyword search over the database; obtaining a second set of search results from the keyword search; integrating the first set of search results with the second set of search results according to a ranking algorithm; and providing the integrated search results to a user through a user interface. In embodiments, generating one or more enhanced search queries based on the first search query comprises: processing, with a large language model, the first search query and generating, with the large language model, one or more related search queries based at least in part on the processing. In embodiments, generating one or more enhanced search queries based on the first search query further comprises: generating one or more search filters based on the first search query, optionally wherein said generating comprises identifying one or more filters from a predetermined set of filters that apply to the search query and / or matching one or more keywords and synonyms associated with respective filters present in the first search query. In embodiments, performing a keyword search over the database further comprises: matching the one or more keywords against one or more tags stored in the database; and / or matching the one or more keywords against the one or more captions or revised captions for the video content.
[0028] In embodiments, performing the keyword search over the database comprises applying a BM42 sparse retrieval algorithm to one or more document-identifying fields, such that documents having lexical matches in the one or more document-identifying fields are ranked higher than documents lacking lexical matches in the one or more document-identifying fields. In embodiments, performing the vector search over the database comprises retrieving documents based at least in part on semantic similarity to indexed content,the indexed content comprising one or more of: document body text, document summaries, image captions, and / or table captions. In embodiments, integrating the first set of search results with the second set of search results comprises combining ranked results from the vector search and ranked results from the keyword search using reciprocal rank fusion.
[0029] In embodiments, generating one or more enhanced search queries based on the first search query comprises: providing a raw user query and a structured auto-filter object to a large language model, the structured auto-filter object comprising predefined filter categories and corresponding canonical values and / or synonyms; and generating, by the large language model, an optimized search query and a structured set of filter values, wherein one or more predetermined semantic concepts, optionally a detected product value and / or indication value, are retained in both the optimized search query and the structured set of filter values. In embodiments, the structured set of filter values is a set of filter values validated against a predefined filter configuration such that unsupported filter values are excluded, and wherein one or more excluded filter categories are prevented from being populated by the large language model. In embodiments, the structured set of filter values comprises a language filter and a country filter, and a language filter value and a country filter value are populated independently unless the first search query explicitly links the language value and the country value.
[0030] In embodiments, the method further comprises deriving, from metadata associated with documents retrieved in the integrated search results, one or more contextual filters for presentation to a user for iterative refinement of the search results; and re-ranking at least a subset of candidate documents based on a similarity between terms of the first search query and terms of a document name and / or title. In embodiments, the similarity comprises a Jaccard similarity. In embodiments, a sparse-search-derived ranking score is determined from a weighted combination of a BM42 score and a fuzzy matching score. The method may further comprise providing to the user results of the re-ranking.
[0031] According to a third aspect, there is provided a system comprising: one or more processors and computer readable memory storing instructions that cause the processor to perform the method of any embodiment of any preceding aspect, including in particular any embodiment of the first and / or second aspect. In some embodiments, the system may comprise one or more computers, servers, or cloud-based devices, for example.
[0032] According to a fourth aspect, there is provided a non-transitory computer readable storage medium containing machine executable instructions which, when executed on a processor, cause the processor to perform the method of any embodiment of the first or second aspect, including any one, or any combination insofar as they are compatible, of the optional features set out with reference thereto.
[0033] According to a fifth aspect, there is provided a computer program comprising executable code which, when run on a computer, causes the computer to perform the method of any embodiment of the first or second aspect, including any one, or any combination insofar as they are compatible, of the optional features set out with reference thereto.
[0034] The disclosure includes the combination of the aspects and preferred features described except where such a combination is clearly impermissible or expressly avoided.Summary of the Figures
[0035] Embodiments and experiments illustrating the principles of the invention will now be discussed with reference to the accompanying figures in which:
[0036] Figure 1 illustrates an overview of an example computing system for processing and storing content, as well as for searching and retrieving content, according to one embodiment of the disclosure.
[0037] Figure 2 depicts a diagram of the method according to one embodiment of the disclosure.
[0038] Figure 3 depicts a diagram of an example computing environment configured to execute the method according to an embodiment of the disclosure.
[0039] Figure 4 depicts a diagram of the multimodal approach to processing video content according to an embodiment of the disclosure.
[0040] Figure 5 depicts a schematic representation of findings in relation to the choice of attention mechanism.
[0041] Figure 6 depicts an example knowledge graph according to an embodiment of the disclosure.
[0042] Figure 7 depicts a diagram of the method for searching the database according to an embodiment of the disclosure.
[0043] Figure 8 depicts a diagram of a dual-search method for searching the database according to an embodiment of the disclosure.
[0044] Figure 9 depicts a diagram of an example implementation of a dual-search method for searching the database according to an embodiment of the disclosure.
[0045] Figure 10 shows data demonstrating the effect of multimodality on video content understanding. Fig. 10a shows per model improvements with subtitles. Fig.10b shows accuracy by task type and condition with and without subtitles. Fig. 10c shows performance per video type per model with and without subtitles. Fig. 10d shows accuracy improvement by task type with and without subtitles. Fig. 10e shows per model improvements with vs without ALM subtitles. Performance is measured in terms of % of questions answered correctly on the Video-MME benchmark set of 900 videos annotated with 2,700 questions-answers pairs.
[0046] Figure 11 shows data illustrating time localisation challenges for VLMs. Fig. 11a shows the number of videos with valid summaries obtained using two different models. Fig. 11b shows summary accuracy and key frame accuracy for the two models. Fig. 11c shows speed and cost comparisons.
[0047] Figure 12 schematically illustrates an example search scoring process used in a system according to an embodiment of the disclosure.
[0048] Detailed Description
[0049] Aspects and embodiments of the present invention will now be discussed with reference to the accompanying figures. Further aspects and embodiments will be apparent to those skilled in the art. All documents mentioned in this text are incorporated herein by reference.
[0050] The methods described herein are computer implemented unless context indicates otherwise. Indeed, video and audio data analysis using deep learning models is of a complexity that places the methodsdescribed herein far beyond the capability of mental investigation. Thus, any method described herein may be implemented in a computer system (in particular in computer hardware or in computer software) in addition to the structural components and user interactions described.
[0051] The term “computer system” includes the hardware, software and data storage devices for embodying a system or carrying out a method according to the above-described embodiments. For example, a computer system may comprise one or more processing units such as central processing units (CPU) and / or graphics processing units (GPU), input means, output means and data storage. Preferably the computer system has a monitor to provide a visual output display. The data storage may comprise RAM, disk drives or other computer readable media. The computer system may include a plurality of computing devices connected by a network and able to communicate with each other over that network. It is explicitly envisaged that the computer system may consist of or comprise a cloud computer.
[0052] The methods described herein may be provided as computer programs or as computer program products or computer readable media carrying a computer program which is arranged, when run on a computer, to perform the method(s) described herein. The term “computer readable media” includes, without limitation, any non-transitory medium or media which can be read and accessed directly by a computer or computer system. The media can include, but are not limited to, magnetic storage media such as floppy discs, hard disc storage media and magnetic tape; optical storage media such as optical discs or CD-ROMs; electrical storage media such as memory, including RAM, ROM and flash memory; and hybrids and combinations of the above such as magnetic / optical storage media.
[0053] The terms “analysing” and “processing” may be used interchangeably in the context of applying the models described herein to content.
[0054] A “visual language model” (VLM) as used herein refers to a machine learning model trained to analyse video content. A VLM is typically a deep neural network. A VLM may be a generative model that has been trained to take as input video content and a natural language prompt, and to generate as output a natural language response to the natural language prompt. Any VLM known in the art may be used in the context of the present disclosure. The methods of the present disclosure have been tested on each of the VLM models in Table 1 , and any of these may be used in the context of the present disclosure. In embodiments, the VLM is a Qwen-2 model.
[0055] An “audio language model” (ALM) as used herein refers to a machine learning model trained to take as input audio content and produce as output a transcription of the audio content. An ALM is typically a generative deep neural network. Any ALM known in the art may be used in the context of the present disclosure. The methods of the present disclosure have been tested on each of the ALM models in Table 1 , and any of these may be used in the context of the present disclosure. In embodiments, an ALM model is selected from Whisper Turbo and Whisper Large v2.
[0056] A “large language model” (LLM) as used herein refers to a machine learning model trained to take as input a natural language prompt and produce as output a natural language response to the prompt. Any LLM known in the art may be used in the context of the present disclosure, including e.g. any models in the GPT family, the Gemini family, the Claude family of models, etc. In embodiments, the Claude Sonnet v3 model is used.A translation model is a machine learning model that has been trained to take as input natural language text in a first language (also referred to as source language) and provide as output a translation of the input into a second language (also referred to as target language). A translation model may be a specific type of LLM trained specifically for translation purposes. A translation model may be a multilingual translation model, configured to take as input natural language text and an indication of a target (and also optionally source) language.
[0057] The methods of the present disclosure are particularly useful in the context of processing of long videos, also referred to herein as “long form video content”. A long video is a video that is at least 30 minutes long. Although a definition of “long video” referring to videos that are at least 30 minutes long has been used for the purpose of evaluating the performance of methods and systems described herein in processing long form video content, the skilled person understands that the benefits of the disclosure are not expected to be strictly limited to such videos. For example, a long video can also be defined as a video that is at least 15, 20, 25 or 30 minutes long. Datasets that may be processed in the context of the present disclosure may include long videos and one or more short videos and / or medium length videos. As used herein, “medium length videos” refers to videos that are longer than short videos and shorter than long videos. For example, medium length videos may be at least 2 minutes (120 seconds), and less than 30 minutes. As used herein, “short videos” refers to videos that are shorter than medium length videos. For example, short videos may be videos that are less than 2 minutes (120 seconds).
[0058] Aspects of the present disclosure relate to multi-modal data processing and searching methods and systems for digital content of various data types, including at least video content.
[0059] An example computing system for processing and storing content, as well as for searching and retrieving content, according to one embodiment of the disclosure is shown in Figure 1. The system may include a platform 110 upon which the methods are performed. The platform 110 may include one or more processors 112 and one or more memories 114. Any of the methods described herein may be stored in memory 114 as a computer program or set of instructions 116. The platform 110 may be communicatively coupled with a database 120. For example, the platform 110 and the database 120 may operate in a same Cloud environment 101. Additionally, the platform 110 may be in communication with a user device 130 operated by a user. The user device 130 may be a remote device located outside of the Cloud environment 101 , and may include one or more processors 132 and memory 134. In some implementations, the user device 130 may display or otherwise provide a user interface that allows the user to submit a search query and to receive search results. The user interface may be a graphical and / or auditory interface.
[0060] Figure 2 depicts a flowchart for a method of processing digital content (e.g. video content) according to one aspect of the present disclosure. The method involves extracting, captioning, tagging, and vectorizing digital content of various data types, including documents and video content.
[0061] At Step 220, the method involves extracting metadata from the digital content. Metadata may include information such as the title, author, creation date, and other relevant attributes of the digital content. An example list of attributes extracted from a digital content is provided in the examples at the end of the description. In embodiments, the method also comprises extracting reference data from one or more digital document files. The digital document files can comprise text, image content and / or tabular content. The extracted reference data can encompass any data that is referenced or shown within the digital content,such as: text, image content, tabular content, etc. In an example architecture, shown in Figure 3, metadata and, if performed, reference content extraction is carried out by an Asset Extraction Pipeline that utilizes text extraction tools, document processing libraries, and metadata generation utilities to extract various elements from the uploaded files, including text, images, tables, and metadata. For instance, a PDF document containing a financial report would have its text, embedded images, and tabular data extracted and stored for further analysis.
[0062] Step 210 involves generating captions for the video content. In embodiments, captions are also generated for any reference content that may have been produced. Captions may include concise descriptions of images, video summaries, video key frame information, etc. Captions provide context and enhance the semantic understanding of the digital content. In some embodiments, this involves processing the reference data with a vision language model (VLM) to generate descriptive captions. In the example of Figure 3, a Captioning Pipeline generates captions for the extracted images and tables using a vision model. These captions provide descriptive text that enhances the understanding of the visual content. Additionally, if the digital content includes video, the VLM may process the video to generate captions that include a summary and key frame information. This is shown in more detail in Figure 4.
[0063] Step 230 involves generating, using an audio language model, a transcript from audio content extracted from or associated with the video content. In these embodiments, the video content may be processed by a video converter or other suitable tool to extract audio content from the video content. The extracted audio content may then be processed with an audio language model (ALM) to generate a video transcription. This is shown, for example, in Figure 4.
[0064] Step 210 can comprise generating one or more tags based on the captions. Tags are keywords or phrases that categorize and describe the content, making it easier to search and retrieve. The tags are derived from the captions to ensure they accurately represent the content’s themes and topics. This tagging process aids in organizing the digital content within the database and improves searchability. In the example of Figure 3, the Tagging Pipeline generates two types of tags: extractive and abstractive. Extractive tags are directly derived from the content, while abstractive tags provide a more generalized summary. These tags are generated using a tagging model and are stored in the storage system.
[0065] Extractive tags are tags generated through exact keyword or synonym matching within the text, e.g. the captioning text. An example of an extractive tag is a product tag (e.g., a tag relating to a specific drug, treatment, or product explicitly mentioned in the digital content). For example, if a document or its captioning mentions "Ocrevus" or its synonyms, a product tag for "Ocrevus" may be assigned. Another extractive tag may be an indication tag (e.g., a tag representing a medical condition or disease associated with the mentioned product or treatment). For instance, if the digital content or captioning also mentions "cardiovascular", an indication tag forthat condition may be added.
[0066] Abstractive tags are tags that are assigned based on the contextual meaning of the document and the definition of the tag categories. The following are example types of abstractive tags that may be added: audience (e.g., a tag identifying the intended viewership, such as consumers, patients, healthcare / regulatory authorities, nurses, etc.); content purpose (e.g., a tag describing the objective of the document, such as disease awareness, product information, or clinical trials); content complexity (e.g., a tag indicating the level of technical detail, ranging from introductory to expert); non-clinical topics (e.g., atag that captures themes not directly related to clinical or medical information, such as lifestyle, e-learning, or legal matters); clinical topics (e.g., overs key medical or scientific subject matter, such as mechanisms of action, disease stages, or diagnostics). For example, a document outlining the legal implications of disclosing patient disease stage information under new data protection regulations might be assigned the following tags:
[0067] Audience: Administrator
[0068] Healthcare / Regulatory Authorities
[0069] Content Purpose: Market Access
[0070] Content Complexity: Advanced
[0071] Non-Clinical Topic: Legal
[0072] Clinical Topic: Disease Stage.
[0073] Abstractive tags can be obtained by defining tags and transforming each tag to a corresponding prompt for the LLM including an example and expected outcome with reasoning. The present inventors benchmarked such an approach and identified that Zero shot learning was already at 90+ percent for most of the tags.
[0074] This hybrid tagging approach ensures comprehensive coverage of both direct mentions (extractive) and deeper semantic themes (abstractive), resulting in a more meaningful and searchable categorization of digital content.
[0075] At Step 240, a vector representation of the digital content is generated. Vector representations are mathematical models that capture the semantic meaning of the content in a format suitable for computational analysis. This involves transforming the textual content into vectors that can be used for various machine learning and data processing tasks. The vector representation encapsulates the content’s features and relationships, enabling advanced analysis, search and retrieval. In Figure 3, the Asset Vectorization & Thumbnail Generator processes the transcript, the metadata, and the generated captions, to generate a vector representation. This involves embedding the digital content as a vector. In some examples, this may involve splitting the digital content into smaller chunks and embedding these chunks as vectors, for example. Thumbnails may also be selected or generated for each embedded digital content. Vectors, along with their associated tags and thumbnails, may be stored in a vector database. In embodiment, a thumbnail is generated for each file. For example, in the case of a slide deck a thumbnail may be a first or representative slide. In the case of a video a thumbnail may be a selected frame, such as e.g. a key frame identified using a method as described herein. In embodiments, a summary is stored for each file. For example, a summary may be generated using a VLM, ALM, and / or LLM as described herein, and the summary may be stored together with the vector representation of the digital content. In embodiments, both a thumbnail and a summary are stored. This process enables efficient semantic search and retrieval of the data. For example, a user searching for information on "cancer treatment" would receive results that are semantically related to the query, even if the exact keywords are not present.
[0076] Step 250 involves storing the generated vector representation and tags in a database. This storage ensures that the processed digital content is readily accessible for future use. The database serves as a repository where the vector representations and tags are indexed and organized, facilitating efficient search and retrieval operations. By storing these elements, the method ensures that the digital content can beeffectively managed and utilized in various applications. In the example of Figure 3, a vector database service is used for storing the vector representations and tags.
[0077] As mentioned, Figure 3 shows an example implementation. In this example, the method begins with the uploading of documents and / or video content to a designated storage location. Upon uploading, a function is triggered, which initiates jobs in a batch processing system. In some implementations, the system is composed of several interconnected pipelines, each responsible for specific tasks such as asset extraction, captioning, translation, tagging, and vectorization. The processed data is directed from each pipeline to a target bucket or other temporary storage location for further use, until the vectorized data and tags are permanently stored in the database.
[0078] In Figure 3, the process commences with the Asset Extraction Pipeline. This pipeline utilizes a combination of text extraction tools, document processing libraries, and metadata generation utilities to extract various elements from the uploaded files, including text, images, tables, and metadata. The extracted results are stored in both a storage system and a database for subsequent processing. For example, a PDF document containing a financial report would have its text, embedded images, and tabular data extracted and stored for further analysis.
[0079] Following the extraction process, the Captioning Pipeline generates captions for the extracted images and tables using a vision model. These captions provide descriptive text that enhances the understanding of the visual content. The system also handles video and audio data, differentiating between videos with and without transcription and including a Video Length identifier. Services such as video conversion tools and audio language models are used for transcription. A video model generates captions including summaries and / or key frame information for video content. In embodiments, a refinement model is used to enhance the quality of the summary output. The refinement model may be a large language model that generates an improved summary based on video transcript data, metadata, and summary and keyframe information. Thus, methods described herein may comprise using a refinement model, such as, e.g., an LLM, to generate an improved summary based on video transcript data, metadata, and summary and keyframe information. For example, this information may be provided as prompt to the LLM, together with a natural language prompt instructing the LLM to generate the refined summary. The generated captions are stored in the storage system and the jobs may then be submitted to a Translation Pipeline.
[0080] The Translation Pipeline, not shown in Figure 3, is responsible for translating the extracted and captioned content into a common language, e.g. English. This can be achieved using language models capable of handling multiple languages and ensuring accurate translations. Once the translation is complete, the job is submitted to the Tagging Pipeline.
[0081] In the Tagging Pipeline, two types of tags are created: extractive and abstractive. Extractive tags are directly derived from the content, while abstractive tags provide a more generalized summary. These tags are generated using a tagging model and are stored in the storage system and the database.
[0082] The Vectorization Pipeline processes the tagged data by splitting it into smaller chunks and embedding these chunks as vectors. These vectors, along with their associated tags, are stored in a vector database. This process enables efficient semantic search and retrieval of the data. For example, a user searching forinformation on "cancer treatment" would receive results that are semantically related to the query, even if the exact keywords are not present.
[0083] Figure 4 shows a partial view of the example architecture with a focus on the multimodal approach to processing digital content that includes video content. In Figure 4, only the extraction and captioning steps are shown (i.e. steps 210 and 220 of Figure 2). The most advanced vision models currently available perform very poorly when it comes to video frame analysis. In particular, the inventors found that both top open-source and commercial models struggle with key frame detection, showing low accuracy (5-35%) and incorrect timestamps. This multimodal approach to video analysis leads to substantial improvements. For instance, the inventors found that combining metadata / voiceovers with LLM / ALMs enhances Vision Language Model (VLM) video understanding. Further, voiceover transcription boosts accuracy from 58.4% to 62.3% (+3.9%), and is especially effective for longer videos (+7.9%). It improves tasks like action recognition, object reasoning, and OCR, but may negatively impact temporal reasoning tasks.
[0084] The architecture described in this aspect of the present disclosure leverages advanced language and vision models to significantly improve tasks such as action recognition, object reasoning, and Optical Character Recognition (OCR). By integrating metadata and voiceovers with Large Language Models (LLMs) and Audio Language Models (ALMs), the system enhances Vision Language Model (VLM) video understanding, particularly for longer videos.
[0085] A technical advantage of this architecture is its ability to process, store, and create derived content for long videos in a manner that focuses on understanding the content rather than merely detecting objects in video frames. This approach ensures that the content of video data is stored in a more usable manner, accurately reflecting both the visual and audio elements. The integration of vision models and audio language models as described herein allows the system to perform object reasoning, which involves understanding the relationships and interactions between objects within a scene. This goes beyond simple object detection, providing a deeper understanding of the context and dynamics of the visual content. For example, in a video of a crowded street, the system can reason about the interactions between pedestrians, vehicles, and other objects, offering valuable insights for traffic management and urban planning. The system's ability to extract text from images and videos using OCR is enhanced by the combination of vision models and language models. This ensures accurate text extraction from various sources, including documents, signs, and subtitles within videos. The extracted text can then be processed and analyzed for further use, such as translation, tagging, and vectorization. By combining metadata and voiceovers with LLMs and ALMs, the system significantly improves video understanding. Metadata provides additional context about the video content, while voiceovers offer a narrative that complements the visual elements. This combination allows the Vision Language Model (VLM) to generate more accurate and comprehensive summaries of the video content, especially for longer videos. The system can identify key frames, actions, and interactions, and generate detailed summaries that reflect the overall content of the video.
[0086] The architecture ensures that the processed video data is stored in a structured and usable manner for reasoning-based tasks. By creating embeddings derived from captions generated from visual as well as audio content as vectors and storing them in a vector database, the system enables efficient semantic search and retrieval of content in videos. Users can search for specific information within videos using natural language queries, and the system will return results that are semantically related to the query, evenif the exact keywords are not present. This enhances the usability of the stored video data, making it easier to access and analyze.
[0087] The system's approach to processing and storing video data ensures that the content is represented in a manner that reflects both the visual and audio elements. This comprehensive representation allows for a deeper understanding of the video content, facilitating tasks such as content analysis, summarization, and indexing. By focusing on the overall understanding of the content, the system provides a more valuable and insightful representation of the video data. By combining metadata and voiceovers with LLMs and ALMs, the methods and systems enhance VLM video understanding, particularly for longer videos. A technical effect is an improved way of processing, storing, and creating derived content for videos, enabling improved focuses on understanding the content rather than just object detection. This ensures that the video data is stored in a more usable manner (e.g. a manner that is searchable and usable for generative tasks with a focus on reasoning rather than simple object detection), accurately reflecting the content of both the visual and audio elements.
[0088] A further aspect of the present disclosure relates to an innovative method and system for visually comparing the quality of summaries and key frames obtained from Al models (e.g. vision language models or audio language models) using knowledge graphs. Knowledge graphs are employed to represent the correlations between textual elements, offering a structured and visual means to analyze the data, providing an objective framework for assessing and comparing the performance of different models. This approach enhances the understanding of Al-generated summaries and key frames, facilitating detailed analysis and comparison. The knowledge graph is designed to illustrate the relationships between key frames and their corresponding summaries. Specifically, the graph includes a root node for each of one or more models to be compared, each connected to a key frame root note and a summary root node. The summary root node is connected to individual summary nodes, each associated with a summarised concept provided by one of the models. The key frame root node is connected to individual key frame nodes each associated with a key frame identified by one of the models. In embodiments, the root nodes forthe models to be compared are omitted. The summary and key frame nodes can be represented using visual indicators that allows to identify the model associated with each node. Visual indicators can be e.g. colors or shapes of the nodes, colours or font of the node names, or a code in the node names. For example, summary and key frame nodes associated with a first model can be in a first colour, summary and key frame nodes associated with a second model can be in a second colour, etc. As another example, summary and key frame nodes associated with a first model can represented with a first shape, summary and key frame nodes associated with a second model can be represented with a second shape, etc. By examining the graph, both the number of nodes and their spatial distribution are easily observable. Additionally, the distance between nodes reflects the concentration of content, where closely clustered nodessuggest a focus on fewer, closely related frames. In some implementations, the positioning of the nodes is determined automatically using a spring layout algorithm (also referred to as a force-directed graph drawing method). In one example, spring_layout(G, seed=42) may be used. This type of layout algorithm positions nodes based on the connections (i.e. edges) between them, simulating forces like springs on edges pulling connected nodes together and repulsive forces pushing all nodes apart, where the final positions represent an equilibrium state of these simulated forces. Several insights can be derived from analyzing a knowledge graph. Forinstance, a higher number of nodes spread out around the central node (root key frame node) indicates a more detailed and comprehensive set of key frames. This suggests that the Al model has captured a wide range of information. Closely clustered nodes indicate a concentration of content, focusing on fewer, closely related frames. This can be useful for identifying key moments or themes within the data. Thus, methods of the present disclosure can comprise evaluating one or more vision language models (VLMs) by: processing one or more video files using said one or more VLMs. In one example, knowledge graphs are constructed using NetworkX and DiGraph. DiGraph is particularly useful when directionality matters. For example, it can represent the relationship from an Al model to the generated key frames, indicating the flow of information. NetworkX is a library that efficiently manages large graphs using dictionaries, allowing for the handling of complex and extensive data sets. In embodiments, nodes are automatically positioned using a spring layout (also known as a force directed graph drawing method). This improves clarity, ensuring that the graph is easily interpretable. In embodiments, nodes are coloured to help distinguish between different elements, such as Al models and key frames. For instance, nodes representing Al models can be colored differently from those representing key frames, enhancing the visual distinction.
[0089] Knowledge graphs offer a structured and visual representation of data relationships, providing an objective framework for comparing the performance of different models in terms of keyframe identification. By using a consistent format to represent data, knowledge graphs ensure that comparisons between models are based on the same criteria, eliminating subjective biases that might arise from different interpretation methods. The nodes and edges within a knowledge graph can be quantified, allowing for objective measurement of various aspects such as the number of key frames, the density of connections, and the distribution of nodes. These metrics provide concrete data points for comparison, making the evaluation process more reliable.
[0090] Furthermore, knowledge graphs visually map out the relationships between key frames and summaries, making it easier to see how each model processes and organizes information. This clarity helps in objectively assessing which model captures more relevant or detailed information. The process of creating knowledge graphs can be replicated, ensuring that comparisons are consistent across different datasets and scenarios. This reproducibility is crucial for objective evaluation, as it allows for the verification of results.
[0091] By reducing the reliance on subjective interpretation, knowledge graphs provide a visual and data-driven method for comparison, minimizing the influence of personal biases and preferences in evaluating model performance. They integrate multiple dimensions of data, such as thematic elements, narrative structures, and key frame details, ensuring that all relevant aspects of model performance are considered in the comparison. The visual nature of knowledge graphs makes the comparison process transparent, allowing stakeholders to easily understand and verify the basis of the comparison, fostering trust in the evaluation process.
[0092] In essence, knowledge graphs provide an objective and reliable method for comparing the performance of different models, ensuring that evaluations are based on clear, quantifiable, and reproducible criteria. For example, a distance metric may be used to assess summary quality. In some examples, both the number of key frame nodes and their spatial distribution are taken into account to estimate the level of detail and narrative coherence captured by each model in its summary. Specifically, the average distance from amodel's summary node to all its associated key frame nodes may be calculated, alongside the total number of key frames it selected. This approach has been validated by comparing these metrics, as well as ROUGE scores, against human-written summaries for the Video-MME dataset. ROUGE, or Recall-Oriented Understudy for Gisting Evaluation, is a set of metrics and a software package used for evaluating automatic summarization and machine translation software in natural language processing. The metrics compare an automatically produced summary or translation against a reference or a set of references (human-produced) summary or translation. ROUGE metrics range between 0 and 1 , with higher scores indicating higher similarity between the automatically produced summary and the reference. The inventors’ findings indicate that, while ROUGE scores might be comparable for long videos, the underlying process of summary creation differs significantly between models.
[0093] This knowledge graph method of comparison enables a deeper understanding of which types of key frames are prioritized by different VLMs. This comprehensive approach to analysis enhances the understanding of Al-generated summaries and key frames, facilitating detailed and unbiased comparisons. Consequently, in the future, this understanding can guide the selection of the most appropriate model for creating derivative videos from long-form content, depending on the specific key frame emphasis required for a given application.
[0094] While knowledge graphs may be used to monitor how the quality of summaries, particularly in relation to key frames, improves as different models and techniques are tested, knowledge graphs may also serve as communities form clusters and to create links across documents. This could help identify and measure the reuse of existing content within a database.
[0095] Figure 6 depicts an example knowledge graph for summary and key frames using Qwen and Gemini models. The knowledge graph visualizes the comparison between Gemini-2-Flash and Qwen-7B in summarizing a ’Snow White’ stage performance. Each key frame from Gemini-2 Flash is marked in light blue and video summary in dark blue, while Qwen-7B’s key frames are in light red, and video summary in red. The central node represents the key frames, with connections showing their relationships to each model’s summary. Gemini-2-Flash emphasizes narrative elements such as the Magic Mirror, the Evil Queen, and the climax involving Snow White’s revival, while Qwen-7B structures the story around broad thematic transitions like character introductions, forest scenes, and musical elements. This graph presents a structured comparison of the keyframes extracted by Gemini-2-Flash and Qwen-7B from a ’Snow White’ performance. The blue nodes represent Gemini-2-Flash’s emphasis on theatrical storytelling, focusing on individual character moments, while the red nodes highlight Qwen-7B’s broader narrative structure, including interactions between Snow White and supporting characters. Additionally, the red nodes are more widely distributed, whereas the blue nodes are clustered more closely, indicating a difference in granularity and focus.
[0096] In Figure 6, the knowledge graph visualizes the comparison between two vision language models, Gemini-2-Flash and Qwen-7B, in summarizing a 'Snow White' stage performance. The graph highlights the differences in key frame extraction and summary generation, providing insights into each model's approach to narrative structuring and thematic emphasis. The knowledge graph serves as a visual representation of the comparison between the vision language models (here illustrated as Gemini-2-Flash and Qwen-7B models). It is designed to illustrate how each model processes and summarizes key frames from a 'SnowWhite' stage performance. The graph can use color coding to differentiate between the outputs of the two models. In the example illustrated on Fig. 6, Gemini-2-Flash key frames are marked in light blue, and video summaries are marked in dark blue, and Qwen-7B key frames are marked in light red, and video summaries are marked in red.
[0097] At the center of the knowledge graph is a “key frames” root nodes to which all nodes representing the key frames extracted from the performance are connected. The connections illustrate how each model interprets and organizes the narrative elements of the performance.
[0098] In the example shown on Figure 6, the Gemini-2-Flash model emphasizes specific narrative elements within the 'Snow White' performance. Key frames and summaries generated by this model focus on individual character moments and pivotal scenes, such as:
[0099] Magic Mirror: Highlighting the scenes involving the Magic Mirror and its significance in the story.
[0100] Evil Queen: Emphasizing the actions and motivations of the Evil Queen.
[0101] Snow White's Revival: Capturing the climax of the performance where Snow White is revived. The blue nodes in the graph are clustered closely together, indicating a detailed and granular focus on theatrical storytelling and character-driven moments. The Qwen-7B model structures the 'Snow White' story around broader thematic transitions. Key frames and summaries generated by this model encompass larger narrative arcs, such as:
[0102] Character Introductions: Introducing the main characters and their roles in the story.
[0103] Forest Scenes: Depicting the scenes set in the forest, which are crucial to the plot development.
[0104] Musical Elements: Highlighting the musical performances and their contribution to the overall narrative. The red nodes in the graph are more widely distributed, reflecting a broader and less granular approach to storytelling. This model focuses on interactions between Snow White and supporting characters, providing a thematic overview of the performance.
[0105] The example knowledge graph on Figure 6 presents a structured comparison of the key frames and summaries extracted by Gemini-2-Flash and Qwen-7B. The visual differentiation between blue and red nodes allows for easy identification of each model's emphasis and narrative structuring. The clustering of blue nodes versus the distribution of red nodes indicates a difference in granularity and focus between the two models. This knowledge graph can be utilized in various applications, such as: performance analysis (e.g., providing insights into how different models interpret and summarize stage performances), content creation (e.g., assisting creators in understanding the strengths and weaknesses of each model for narrative development), educational tools (e.g., enhancing learning materials by visualizing complex narrative structures and model comparisons).
[0106] An example implementation of the knowledge graph approach to comparing the output quality of various VLMs is shown in the following algorithm (illustrated by reference to the two specific models for which the graph on Fig. 6 was produced, but any one or more VLMs may be used instead).
[0107] An example knowledge graph construction algorithm is as follows:
[0108] Input: ISON data with key frames from Gemini-2 Flash and Qwen-7BOutput: Visualized Knowledge Graph
[0109] Step 1 : InitiaLize Graph
[0110] Create directed graph G «- nx. DiGraph( )
[0111] • Nodes (V) -> Individual entities in the graph (e . g. , "Gemini-2 Flash" , "Snow White in rags" )
[0112] • Edges ( E) -> Directed connections between nodes (e .g. , "Gemini-2 Flash" -> "Key Frames" )
[0113] • Attributes -> Additional properties of nodes / edges (e. g. , color, size) DiGraph G = (V, E) is defined as :
[0114] • V=nodes
[0115] • E=(source, target) , where each edge has a direction
[0116] Step 2: Add Core Nodes
[0117] Add node KeyFrames with attributes (color: gray, size : 800)
[0118] Add node VideoSummary with attributes (color: gray, size : 600)
[0119] Step 3: Connect ModeLs to Core Nodes
[0120] Add edge (e . g. , GeminiFlash, KeyFrames)
[0121] Add edge (e . g. , GeminiFlash, VideoSummary)
[0122] Step 4: Add Key Frames for Each ModeL
[0123] for each (timestamp, description) in e . g. Gemini-2 Flash key frames do:
[0124] Add node description with attributes (color: light blue, size : 400)
[0125] Add edge (KeyFrames, description)
[0126] end for
[0127] Step 5: VisuaLize the Graph
[0128] Compute node layout pos «- spring_layout(G, seed=42)
[0129] Extract node colors and sizes for Knowledge Graph : Key Frames Evaluation
[0130] A further aspect of the present disclosure relates to a multi-modal search method and system designed to improve the accuracy and relevance of search results. The method and system employ techniques such as query regeneration using Large Language Models (LLMs), vectorization for semantic matching, and a dual search methodology integrating both vector-based and keyword-based searches. An advanced ranking algorithm optimally combines signals from both search methodologies to present the user with highly relevant search results.
[0131] Figure 7 depicts a flowchart for a method for searching digital content stored in vectorized form in a database. In some embodiments, the database may be a vector database.
[0132] At Step 710, the method begins when a user submits a search query, for example, via a user interface. This query can be in various forms, such as text, voice, or image input. The user interface may be on a user device from which the query is sent to a platform running the search engine. The user device may be, for example, a remote device (such as a mobile phone) operating outside of a cloud environment in which the platform operates and in communication with the platform via a network.
[0133] At Step 720, upon receiving the initial query, the method includes generating enhanced search queries based on the original search query using, for example, a query regeneration mechanism. This mechanism may leverage Large Language Models (LLMs) to reformulate and / or expand on the original query. The regeneration process may capture different aspects or interpretations of the user’s query and its intended meaning, thereby producing either a single refined query or multiple query variations. For example, if the user queries "best smartphones", the system might generate variations such as "top-rated smartphones", "latest smartphones", and "smartphones with best battery life". In one embodiment, the entire corpus ofdocuments available for search are segmented into a predefined taxonomy (hereinafter referred to as "tags"). In one example, the tags consist of twelve primary categories and over seventy subcategories. Upon receiving the initial query, the initial query may be enriched through the identification and expansion of abbreviations, synonyms, and / or related terminology. The enriched query may then be transformed into a structured representation that maps to the tags. In addition to tag-based structuring, semantic elements of the query that do not directly correspond to predefined tags are also identified and preserved as keyword phrases or semantic descriptors to ensure comprehensive query interpretation. For example, a user query such as "I want Tecentriq efficacy data for ESMO" may be transformed into a structured query such as: "Oncology; Tecentriq; Efficacy; All Formats; Patient Data; Presented at ESMO", or "Oncology; Tecentriq / Tecentriq SC; Efficacy / Safety-Efficacy Data; HOP Readouts; Latest Data; ESMO". This approach is designed to maximize recall and ensure the retrieval of relevant content across diverse document formats and terminologies.
[0134] At Step 730, the enhanced search query or queries undergo a vectorization process, transforming them into high-dimensional embeddings. This transformation enables semantic matching in the vector space against the vectorised digital content stored in the database, allowing for a vector search.
[0135] At Step 740, the vectorized search query or queries are utilized to perform a vector search over the database. In some embodiments, parallel searches are carried out across distinct vector collections grouped by data type. For example, a text collection may comprise unstructured text documents, such as articles, blogs, and reports. An image collection may contain visual content, and a search may be performed using the captions or metadata associated with the images. For instance, if the query is "sunset", the system searches for images with captions or tags related to sunsets.
[0136] At Step 750, a set of search results is obtained from the vector search. This set may include semantically matched content from the multi-modal collections, for example, providing a broader and more contextually relevant set of results. In some embodiments, search results may also be assigned a score based on different metrics, such as relevance, popularity, user preferences, historical data, etc. Then, a ranking algorithm may be used to rank the search results based at least in part on their scores. This may involve using machine learning techniques to optimally score and / or rank search results.
[0137] At Step 760, the search results are provided to the user. In some embodiments, the results are presented in the order in which they were ranked. In some examples, some search results may be omitted. For instance, search results that are ranked below a pre-determined threshold rank may be omitted or may have been pruned at step 750. In some other examples, search results may be curated, compressed, expanded, and / or reordered. The presentation may be designed to offer a comprehensive view of the search results.
[0138] Figure 8 depicts a flowchart for a further embodiment of the method shown in Figure 7 for searching digital content stored in vectorized form in a database. In this embodiment, a dual-search approach is employed to enhance search capabilities.
[0139] At Step 810, the method begins when a user submits a search query, for example, via a user interface. This query can be in various forms, such as text, voice, or image input. The user interface may be on a user device from which the query is sent to a platform running the search engine. The user device may be,for example, a remote device (such as a mobile phone) operating outside of a cloud environment in which the platform operates and in communication with the platform via a network.
[0140] At Step 820, upon receiving the initial query, the method includes generating enhanced search queries based on the original search query using, for example, a query regeneration mechanism. This mechanism may leverage Large Language Models (LLMs) to reformulate and / or expand on the original query. The regeneration process may capture different aspects or interpretations of the user’s query and its intended meaning, thereby producing either a single refined query or multiple query variations. For example, if the user queries "best smartphones", the system might generate variations such as "top-rated smartphones", "latest smartphones", and "smartphones with best battery life".
[0141] At Step 831 , the enhanced search query or queries undergo a vectorization process, transforming them into high-dimensional embeddings. This transformation enables semantic matching in the vector space against the vectorised digital content stored in the database, allowing for a vector search.
[0142] At Step 832, one or more keywords are extracted from the enhanced search query or queries. This extraction process involves identifying key terms and phrases that can be used for traditional keywordbased searches. In some implementations, relevant filters may also be inferred and applied based on the content of the original query.
[0143] Steps 831 and 832 may be executed in parallel.
[0144] At Step 841 , the vectorized search query or queries are utilized to perform a vector search over the database. In some embodiments, parallel searches are carried out across distinct vector collections grouped by data type. For example, a text collection may comprise unstructured text documents, such as articles, blogs, and reports. An image collection may contain visual content, and a search may be performed using the captions or metadata associated with the images. For instance, if the query is "sunset", the system searches for images with captions or tags related to sunsets.
[0145] At Step 842, the extracted keyword or keywords are used to carry out a keyword search over the database. In some implementation, the keyword search is executed applying one or more filters determined at step 832. The keyword search may be carried out over the vector-associated tags stored in the database, e.g., by matching keywords against tags, or over a separate database, e.g., a text-only database. This separate database may store metadata and textual summaries of various documents, enabling direct matching based on the keywords. Steps 841 and 842 may be executed in parallel.
[0146] At Step 851 , a set of search results is obtained from the vector search. This set may include semantically matched content from the multi-modal collections, for example, providing a broader and more contextually relevant set of results.
[0147] At Step 852, a set of search results is obtained from the keyword search. This other set may include directly matched documents based on traditional lookup techniques, such as keyword matching, synonym matching, and the like. Steps 851 and 852 may be executed in parallel.
[0148] At Step 860, an advanced ranking algorithm integrates and ranks both the vector-based and keywordbased search results. The algorithm may evaluate various factors such as relevance, user preferences, and historical data to combine and rank the search results effectively. In some embodiments, search resultsmay also be assigned a score based on different metrics, such as relevance, popularity, user preferences, historical data, etc. Then, a ranking may be determined based at least in part on their scores. One or more machine learning techniques may be used to optimally combine signals from both search methodologies. At Step 870, the search results are provided to the user. In some embodiments, the results are presented in the order in which they were ranked at step 860. In some examples, some search results may be omitted. For instance, search results that are ranked below a pre-determined threshold rank may be omitted or may have been pruned at step 860. In some other examples, search results may be curated, compressed, expanded, and / or reordered. The presentation may be designed to offer a comprehensive view that leverages both semantic understanding and explicit keyword matching.
[0149] The dual search method described in the present disclosure offers several specific technical advantages that significantly enhance the accuracy and relevance of search results. By integrating both vector-based and keyword-based searches, the system leverages the strengths of each approach to provide a comprehensive and nuanced understanding of user queries. One of the advantages of this method is the use of query regeneration through Large Language Models (LLMs). This technique allows the system to reformulate and expand the original query, capturing different aspects or interpretations of the user query and its intended meaning. This process ensures that the search query is semantically understood before the search is conducted, increasing the likelihood of retrieving highly pertinent results.
[0150] The vectorization process further enhances the system’s capabilities by transforming the enhanced query or queries into high-dimensional embeddings or vectors. This enables the system to search the database by semantic matching in the vector space in a computationally efficient manner. As a result, the system can perform parallel searches across multiple vector collections, such as image and text collections. Also executed in parallel is the keyword search, which complements the vector search with the use of relevant filters and keywords from the original query. This dual search methodology provides two sets of search results: vector search results, which include semantically matched content from multi-modal collections, and keyword search results, which comprise directly matched documents based on traditional information retrieval techniques. An advanced ranking algorithm plays a crucial role in integrating and evaluating both sets of search results. This algorithm, likely involving machine learning techniques, optimally combines signals from both search methodologies. It considers various factors such as relevance, user preferences, and historical data to rank the results effectively. The final ranked results are curated and presented to the user through the interface, offering a comprehensive view that leverages both semantic understanding and explicit keyword matching. This approach ensures that the user is provided with a diverse set of highly relevant search results, enhancing the overall search experience.
[0151] The methods and systems described herein have a wide range of applications across various fields, leveraging advanced Al and machine learning techniques to enhance the processing and analysis of digital content. In the realm of radiology and imaging analysis, these methods enable the analysis of full-length MRI or CT scans in motion, such as cardiac MRI or functional MRL Advanced Al algorithms can detect anomalies more rapidly and accurately than manual review, improving diagnostic efficiency and accuracy. Real-time video analysis powered by Al can assist in interpreting ultrasound images for fetal health assessments, echocardiography, and liver disease detection, providing immediate insights and reducing the reliance on manual interpretation.In endoscopy and surgical video analysis, Al can process extensive colonoscopy footage to detect polyps in real-time, significantly enhancing colorectal cancer screening by identifying potential issues that may be missed during manual review. Al-driven video processing can offer real-time insights to surgeons during laparoscopic and robotic surgeries, flagging anomalies or suggesting procedural adjustments to improve surgical outcomes.
[0152] For neurological and movement disorder detection, the methods can analyze EEG-video recordings to classify different types of epilepsy, aiding in precise diagnosis and treatment planning. Al can assess gait, facial expressions, and movement from patient videos, facilitating early diagnosis and monitoring the progression of neurological disorders such as Parkinson’s disease and ALS.
[0153] In ophthalmology and dermatology, Al models can process retinal scan videos to detect early signs of diabetic retinopathy or macular degeneration, enabling timely intervention and treatment. Dermatologists can utilize Al-enhanced dermoscopy video processing to detect melanoma with greater accuracy, improving early detection and patient outcomes.
[0154] Pathology and microscopy video processing benefit from Al's ability to analyze continuous microscopy footage to identify cancerous cells, bacterial infections, or rare hematological conditions in blood samples more quickly and accurately than human pathologists, enhancing diagnostic capabilities.
[0155] Remote patient monitoring and telemedicine are revolutionized by wearable devices that record and process patient videos, such as heart rate monitors and movement trackers. These devices can enable early diagnosis of conditions like heart arrhythmias or sleep apnea, reducing healthcare costs and improving patient care.
[0156] In agriculture and environmental monitoring, Al-driven video analysis can swiftly identify stressed crops, pest infestations, or nutrient deficiencies, aiding in efficient agricultural management. The methods can detect illegal logging or monitor ecosystem changes, providing faster and more cost-effective environmental surveillance. Rapid video analysis post-disasters such as hurricanes, earthquakes, or floods can help authorities assess damage and respond effectively.
[0157] Infrastructure and urban planning are enhanced by Al's ability to process high-resolution drone footage to detect structural issues like cracks, erosion, or weak points in infrastructure, improving maintenance and safety. Satellite video analysis can track congestion patterns and optimize urban development, improving traffic management and city planning.
[0158] Healthcare and epidemiology are supported by satellite video that can help detect standing water bodies where mosquitoes breed, aiding in the prevention of diseases like malaria and dengue. Fast video analysis can track pollution hotspots, correlating air quality data with disease outbreaks, supporting public health initiatives.
[0159] These applications demonstrate the versatility and potential impact of the described methods and systems across various domains, enhancing efficiency, accuracy, and outcomes in numerous fields.
[0160] In order to improve the performance of a document retrieval platform, particularly in the context of searching Veeva content and other structured materials, the inventors implemented, in some embodiments, a multistage search and ranking architecture designed to improve precision, recall, and contextual relevance ofreturned results. The architecture addresses several limitations associated with conventional semantic-only retrieval systems by combining one or more of: (i) sparse lexical retrieval, and dense semantic retrieval, (ii) large language model-based query transformation, (iii) metadata-driven filtering, and (iv) custom scoring / re-ranking mechanisms. Together, these components enable more accurate retrieval of documents in response to natural language user queries, while also supporting efficient refinement of result sets through contextual metadata.
[0161] In embodiments, the system and methods described herein use a BM42 algorithm for sparse search. For example, a BM42-based retrieval step may be configured to operate on document names and titles. This may , advantageously capture the high informational value of these fields in content repositories such as e.g. Veeva. In many enterprise and commercial content management environments, document names and titles contain highly discriminative terms, such as product names, campaign identifiers, approved material names, publication descriptors, or specialized terminology known to the user. The BM42 algorithm emphasizes exact term occurrence, term frequency, and keyword importance, thereby enabling the system to strongly prioritize documents whose names or titles contain exact or near-exact lexical matches to the input query. This is particularly advantageous where a user searches for a known product, a campaign asset, a particular document title, or a recognized technical or commercial term. By applying sparse retrieval over the name and title fields, the system improves precision by elevating documents exhibiting strong keyword relevance even where broader semantic similarity alone might not sufficiently distinguish among candidate documents.
[0162] Embodiments of the system and methods use a query optimization component (also referred to herein as query optimizer) implemented using a large language model. In many practical search scenarios, users submit queries in natural language, often using incomplete and / or incorrect phrases, conversational formulations, or requests that do not directly correspond to optimal retrieval syntax. In embodiment, the query optimizer is configured to receive such natural language search queries (also referred to herein as prompt) and transform them into structured, search-ready queries that are better suited for use with the downstream retrieval pipeline. In particular, the query optimizer may be configured to analyze the user’s prompt to infer search intent, identify predetermined entities or concepts, restructure and / or expand the query. The restructuring and / or expanding may be performed in a manner that improves recall while preserving relevance. For example, the query optimization component may reformulate an under-specified request into a more targeted query, for example by including additional semantically related search terms (e.g. synonyms, which can be from a predetermined data structure), and / or generate a representation optimized for use in hybrid retrieval across sparse and dense indexes. As a result, even loosely phrased or conversational queries can be converted into optimized search expressions that perform more effectively within the search system. This improves the overall robustness of the platform by reducing the dependence of retrieval quality on the user’s ability to formulate keyword-perfect input.
[0163] In embodiments, the system and / or methods also include an automatic filtering mechanism configured to generate contextual filters from document metadata. These may be presented to the user for iterative refinement of results. After an initial retrieval step, the method / system may analyze metadata associated with the retrieved content and derive relevant filter dimensions that may assist the user in narrowing the result set. Such auto-generated filters may include, by way of example, product, indication, language,country, scientific area, target audience, material intent, channel, content type, and content purpose. The filters may be generated in a contextual manner based on the retrieved documents and the search space, thereby enabling a user to progressively constrain the results without manually defining filter criteria from the outset. The filters may be presented to the user, which may provide user input to approve or disapprove the filters, This mechanism improves usability and search efficiency, especially in repositories containing a large number of related materials that differ only in regulatory region, audience type, content format, or intended commercial use. By allowing metadata-based refinement after retrieval, the system provides a guided narrowing workflow that helps users quickly locate the most relevant subset of documents.
[0164] Embodiments of the present disclosure use a hybrid search approach combining sparse and dense retrieval methods. Rather than relying solely on semantic vector search or solely on keyword search, the platform combines sparse and dense retrieval methods to exploit the complementary strengths of each. Sparse retrieval refers to the use of a search through data stored as sparse vectors indicative of keywords from a predetermined dictionary. Dense retrieval refers to the use of a search through data stored as dense vectors indicative of semantic content on an associated asset (e.g. text chunk, image caption, etc. - also referred to as indexed content). In one implementation, sparse retrieval is performed using BM42. In embodiments, dense semantic retrieval is performed using embeddings generated by the model mixbread-ai / mxbai-embed-large-v1. The dense retrieval stage captures semantic similarity between the user query and the indexed content, thereby allowing the system to retrieve relevant materials even when exact keywords are absent. Conversely, sparse retrieval preserves strong performance for exact-match and terminologysensitive use cases. Sparse retrieval (and associated sparse embedding) can be applied to asset names and titles. Dense retrieval (and associated dense embedding) can be applied to content such as e.g. image caption, text chunks, table captions, text chunks and summaries). In embodiments, a hybrid search approach can be applied combining sparse retrieval and dense retrieval applied to whole documents (i.e. indexed content associated with whole documents). Results from the sparse and dense retrieval processes may be combined using Reciprocal Rank Fusion (RRF). RRF merges ranked result lists from the different retrieval strategies into a unified ranking. The use of RRF allows the system to balance lexical precision against semantic relevance, reducing over-reliance on either method alone and improving the consistency and quality of the final result set.
[0165] The hybrid retrieval step implemented by the search engine (also referred to herein as “search module” and “search component”) may be configured to search across multiple content-bearing components of a document. In addition to the primary document content, the system may index and retrieve against generated summaries, image captions, and table captions. This enables the search engine to access information represented in different forms and at different levels of abstraction. For example, a concept that is not explicitly prominent in the body text may nonetheless appear in a generated summary, or relevant evidence may be captured in a figure caption or table caption rather than in the surrounding prose. By extending retrieval to these auxiliary representations, the system improves both semantic coverage and precision. The result is a richer retrieval process capable of identifying relevant documents on the basis of textual body content as well as derived or descriptive representations associated with visual and tabular elements.To further improve ranking quality, particularly for marketing materials and similar content collections in which document names and titles may have strong commercial or organizational significance, the inventors implemented a custom scoring function applied after the initial hybrid retrieval stage. In embodiments, the system / method first retrieves a set of candidate documents using the hybrid search pipeline described above. Thereafter, an additional scoring component computes a Jaccard similarity between the user query and selected document fields, such as document name and title. This Jaccard-based comparison provides an additional measure of overlap between the terms in the user’s query and the key identifying fields of the document, thereby rewarding documents whose naming structure more closely aligns with the query intent. Such post-retrieval scoring is particularly beneficial in environments where naming conventions, title tokens, or controlled commercial vocabulary are strong indicators of relevance.
[0166] In at least one implementation, a custom scoring is applied to sparse-search-derived results which uses a weighted formulation in which the BM42 score contributes a first predetermined weight (e.g. a weight of 0.7) and a fuzzy matching score between the query and the sparse search result contributes a second predetermined weight (e.g. a weight of 0.3). Such a weighted combination enables the ranking process to retain the strong lexical discrimination provided by BM42 while also accounting for approximate matches, term variations, or other fuzzy correspondences that may not be captured through exact term matching alone. By introducing this weighted post-processing stage, the system / method produces a more nuanced ranking in which documents are not only relevant according to sparse retrieval metrics, but also aligned with the textual form of the user’s search intent.
[0167] In embodiments, the disclosed retrieval framework provides a layered search architecture in which user queries are first optimized through a large language model-based query transformation process, candidate documents are then retrieved through a hybrid combination of BM42 sparse search and dense embeddingbased semantic search, results are fused using Reciprocal Rank Fusion, and relevance is further refined using metadata-driven filtering and a custom scoring mechanism incorporating Jaccard similarity and weighted sparse-result evaluation. This combination improves search precision for exact terminology, improves recall for semantically expressed intent, and provides end users with efficient mechanisms to refine and identify the most relevant content from complex enterprise document repositories.
[0168] In one example, a Query Optimizer is configured to transform raw user input into an optimized search query and a validated set of structured filters. The Query Optimizer is designed to improve both recall and precision in situations where users submit incomplete, misspelled, loosely phrased, or mixed-language search requests. In embodiments, the Query Optimizer operates as an intermediary processing layer between the user-facing search interface and the downstream retrieval system, and is configured to preserve user intent while converting the input into a predetermined form suited for hybrid search and metadata-based filtering.
[0169] In embodiments, the Query Optimizer is configured to implement a content ingestion step. This step may comprise combining a raw user query with a structured auto-filter object (e.g. a JSON object) and providing both inputs to a large language model (LLM). Filters are predetermined fields that are associated with a query, and which identify a semantic feature of the query. For example, a query may be associated with filters such as "product", "indication", "scientific_area”, "language", "country", ”content_type). Each filter may be associated with a string or set of strings (i.e. a value or set of values for the filter - also referred toas a “filter payload”) or an empty string, from a predetermined set of strings. The raw user query corresponds to the direct textual input received from the user at the search interface. The raw user query may be passed into the LLM prompt without modification so that the original wording, terminology, and intent expressed by the user are preserved. This is beneficial because the user query may contain domainspecific phrases, product names, abbreviations, scientific terminology, or partially specified requests that could be diluted or distorted if normalized prematurely. Alongside the raw query, the LLM may be provided with a structured auto-filter object that contains predefined filter categories and associated canonical values. These categories may include, for example, product, indication, scientific area, content type, language, and country. For each category, the auto-filter object may store canonical values together with corresponding synonyms or alternate forms. As a result, the LLM receives both the unstructured expression of user intent and a structured filter vocabulary against which the user’s request can be interpreted and normalized.
[0170] Following the content ingestion stage, the Query Optimizer may perform a spelling correction step. The step may comprise identifying misspelled words in the raw user query and correcting them before generating a final optimized search representation. The spelling correction may be performed in the context of the auto-filter object and the domain vocabulary made available to the LLM, thereby allowing the system to distinguish ordinary typographical errors from specialized branded or scientific expressions that should remain unchanged. For example, where a user enters a request containing a misspelled product name, such as “Okrevs” instead of “Ocrevus,” the Query Optimizer can correct the misspelling to the appropriate canonical term. This improves the likelihood that the downstream retrieval system will match the intended content, while still preserving the overall meaning and structure of the original user request.
[0171] After spelling correction, the Query Optimizer may perform a filter value identification based on user query intent. This step may comprise analyzing the corrected and contextualized user input to determine whether the query expresses one or more values corresponding to structured filter fields. Such fields may include product, indication, scientific area, language, and country. The purpose of this stage is to extract filter values that are implicit or explicit in the user’s request and represent them in a structured form usable by the search platform. For example, if a user requests “Give me Ocrevus materials,” the system identifies “Ocrevus” as a product value while leaving indication, scientific area, language, and country empty unless they are expressly or inferentially present in the query. By extracting structured filter values aligned with user intent, the system enables the retrieval engine to narrow or guide the search process in a controlled and explainable manner.
[0172] In embodiments, the Query Optimizer applies a rule that ensures the retention of technical, branded, and scientific terms in the optimized query. In enterprise search systems, especially those relating to regulated, scientific, or medical content, terms such as product names, indications, trial references, and scientific acronyms carry substantial meaning and must not be altered or omitted during normalization. Accordingly, the Query Optimizer may be configured to preserve predetermined terms or categories of terms in the optimized query string even while other portions of the query may be normalized, restructured, or expanded. This prevents degradation of search quality that could otherwise arise if critical domain-specific expressions were generalized, paraphrased, or removed. For example, where the input query includes “Ocrevus NSCLCtrial data,” the terms “Ocrevus” and “NSCLC” remain intact in the optimized query because they correspond to essential product and indication identifiers.
[0173] The Query Optimizer may enforce country-language dependency rules to ensure that references to language and geography are interpreted accurately and independently unless the user explicitly links them. In many search contexts, a language reference does not necessarily imply a country, and a country reference does not necessarily imply a language. The system may distinguish between these concepts and avoids automatically assigning one on the basis of the other. For example, a query requesting “Italian documents” can be interpreted as specifying a language filter of “Italian,” while acountry filter can remain empty unless the user explicitly refers to Italy as a geographic scope. Conversely, a query such as “Documents from Italy” may result in identification of a country filter of “Italy,” while the language filter remains empty unless separately stated. This separation reduces false constraints and improves the semantic correctness of the extracted filters.
[0174] In embodiments, the Query Optimizer may apply a rule predetermined content in the query may not be removed even if they have been identified as structured filters. For example, the method / system may ensure that product and indication values are not removed from the search query merely because they have been identified as structured filters. These values represent highly significant domain entities and serve not only as filter criteria but also as important lexical terms for retrieval. For example, where the system detects product and indication values in the user query, those values can be retained in both the optimized query string and the extracted filter representation. For example, in a request such as “Give me Ocrevus NSCLC materials,” both “Ocrevus” and “NSCLC” remain present in the optimized query while also being represented in the product and indication filters, respectively. This dual retention ensures that the retrieval engine can leverage both structured filtering and textual relevance matching without sacrificing either.
[0175] The system / method may include a step of synonym expansion for filter values identified from the user query. In this step, alternate forms, abbreviations, and synonymous expressions appearing in the user’s input may be mapped to canonical representations defined in a predetermined filter configuration data object. This improves consistency across search operations while maintaining fidelity to the user’s original language. For example, a query containing “UK oncology documents” may be normalized such that “UK” is mapped to the canonical country value “United Kingdom,” while “oncology” is retained or mapped to the scientific area value “Oncology.” This normalization permits more reliable filtering and aggregation across documents indexed with standardized metadata, even when users employ abbreviated or colloquial forms. In embodiments, the Query Optimizer applies one or more excluded filter rules governing filter categories that the LLM is not permitted to populate. These excluded filters can represent fields that are either controlled elsewhere in the system or considered unsuitable for inference from the user’s free-text input. Excluded filters may belong to excluded categories including general filters such as e.g. content complexity, document type, source file availability, non-clinical topic, and clinical topic, and / or date-related filters such as e.g. creation date, approved-for-distribution date, and expiration date. Even if the LLM suggests values for such categories, the system may enforce exclusion by clearing or suppressing those values in the final filter payload. For example, if a user query references materials created in a particular year and the LLM proposes corresponding creation date bounds, the final filter object will reset the excluded date fields toempty values. This restriction prevents unintended narrowing of results and ensures that system-controlled or disallowed filter dimensions are not populated through generative inference.
[0176] In embodiments, LLM-generated filters are validated against a predefined auto-filter configuration. For example, after the LLM proposes structured filter values, the system / method may compare those values against allowed values defined in an auto-filter schema (e.g. an auto-filter JSON schema). If a generated value does not appear in the configured filter vocabulary for the corresponding category, the system / method may exclude that value from the final filter set. This validation step ensures that only recognized products, indications, scientific areas, languages, countries, and content types are passed forward to the search engine. By way of example, if the LLM proposes a content type not present in the allowed schema, that content type can be removed during validation even if other generated values, such as recognized product or scientific area entries, are retained. This validation mechanism reduces noise and prevents the propagation of unsupported or hallucinated filter values.
[0177] In embodiments, the Query Optimizer performs a synonym expansion step for filters. For example, after a canonical product or indication has been identified and validated, the system / method may expand the corresponding filter payload to include associated synonyms drawn from a predetermined data object (also referred to as configuration source), such as a synonyms.json file. In this manner, the final filter set may include not only the detected canonical value but also alternate forms and related branded variants that should be considered during retrieval. For example, where the system identifies “Ocrevus” as a product, the expanded filter payload may include “Ocrevus,” “Ocrevus SC,” “Ocrevus IV,” and other configured synonyms. This expansion improves recall by ensuring that document metadata or indexed content using alternate product forms can still be matched, while the original query text itself remains faithful to the user’s phrasing and intent.
[0178] To support observability and operational monitoring, the Query Optimizer may include tracing and logging functionality. For each request processed by the Query Optimizer, the system may generate a unique trace identifier, for example in the form DEV_Finder_Optimizer, thereby enabling end-to-end tracking of the request across system components. In addition, the system may attach structured metadata such as requestjd, tool, module, and environment, thereby providing contextual information for debugging, monitoring, and auditability. The tracing and logging implementation is configured to support environmentbased monitoring across development, staging, and production deployments, and further supports latency tracking and error tracking for performance analysis and fault investigation. This observability framework enables maintainers to analyze how individual requests were processed, identify operational bottlenecks, and diagnose failures with greater precision.
[0179] Overall, the Query Optimizer may provide a structured workflow in which a raw user query is ingested together with an auto-filter configuration, corrected for spelling, analyzed for intent and filter extraction, normalized while preserving critical technical and branded terminology, constrained by country-language dependency rules, protected against removal of key product and indication terms, mapped through synonym handling, restricted by excluded filter enforcement, validated against a predefined auto-filter schema, expanded through synonym-based filter enrichment, and traced through a logging and observability framework. By combining these functions, the Query Optimizer produces an optimized searchquery and a validated structured filter set that together improve the quality, consistency, transparency, and operational reliability of the search process.
[0180] In order to provide an interactive assistant capable of answering user questions about documents returned by search, embodiments of the systems and methods of the disclosure include a Document Q&A module, also referred to as a Chat Al module, configured to generate responses that are grounded in retrieved document content and aligned with the user’s linguistic and conversational context. The purpose of the module may be to enable a user to interact with a corpus of retrieved documents in a conversational manner, such that the user may ask targeted questions, request summaries, seek comparisons, or otherwise interrogate the content of the documents without manually reviewing each document in full. The module can be designed to ensure that generated answers remain tied to the retrieved evidence, are produced in an appropriate response language, and preserve continuity across multiple turns of conversation, while also supporting observability, debugging, and auditability through tracing and logging mechanisms.
[0181] The Document Q&A module may implement a method comprising a content ingestion step in which the system prepares an input prompt for a large language model by combining three components: a user query, relevant document text identified by the search module, and a detected language value. The user query can correspond to the direct input submitted by the user through the conversational interface and represent the immediate informational request to be answered. For example, the query may seek a summary, factual explanation, or analysis of a particular product, clinical outcome, or regulatory issue discussed in the retrieved materials. The relevant document text may be obtained from the Search module based on the user query, and can comprise e.g. document text comprising passages, excerpts, or other textual content extracted from documents deemed relevant to the query. These passages may originate, for example, from clinical trial reports, product information documents, scientific communications, or other indexed enterprise materials. The relevant document text can be injected into the prompt so that the large language model is supplied with an evidentiary basis from which to formulate its answer. In addition, the system may inject a detected language parameter corresponding to the language associated with the user’s request. The inclusion of this language parameter allows the module to generate a response in a linguistic form that is appropriate for the user, thereby supporting multilingual interaction without requiring the user to explicitly configure a language preference at each turn.
[0182] Following content ingestion, the Document Q&A module may perform a query language detection step. This step may comprise automatically identifying the language of the user query. This enables the conversational response to be generated in an appropriate linguistic format. The step can be performed using a LLM. In one implementation, this language detection is performed using Amazon Comprehend, which provides automated natural language processing functionality for language identification. The use of an external language detection component at this stage enables robust classification even where the user query is short, domain-specific, or not syntactically complete. For example, where a user submits an Italian-language request relating to results of Ocrevus in clinical studies, Amazon Comprehend may identify the query language as Italian. This detected language is then made available to subsequent processing stages and may be injected into the prompt provided to the large language model. By programmatically identifyingthe language of the incoming request, the system improves the consistency of response generation and reduces the risk that an answer will be produced in an unintended language.
[0183] In embodiments, the Document Q&A module performs user intent classification. This step is configured to determine the functional purpose of the user’s query so that the system may generate a response aligned not only with the query topic, but also with the expected output type and intended linguistic context. In particular, intent classification may distinguish between requests for factual extraction, summarization, comparison, explanation, translation-oriented responses, or responses in a language explicitly requested by the user. This classification may be carried out by a large language model or other natural language understanding component that analyzes the form and content of the query. For example, where the user enters a query such as “Give me Ocrevus materials in Spanish,” the system determines that the user’s intent includes a language-specific request, namely that the response should be delivered in Spanish. The intent classification stage therefore may achieve more than identify subject matter; it may also detect whether the user is implicitly or explicitly directing the system as to the response mode, format, or language. The method may then comprise applying a language override rule to ensure that the final response language corresponds to user intent, even where that intent differs from the automatically detected query language. In many situations, a user may write a query in one language while requesting that the answer be produced in another. For example, a user may submit an English-language instruction requesting materials “in Spanish.” In such a case, automatic language detection may identify English as the language of the query text, but the actual intended response language is Spanish. The Document Q&A module is therefore configured to override the detected language when the intent classification stage identifies an explicitly requested target language. This override mechanism ensures that the response language is not determined solely by the literal language of the query text, but instead reflects the user’s intended output language. As a result, the system can accommodate multilingual interaction patterns in which the language of the prompt and the language of the desired answer are different.
[0184] To preserve continuity across multi-rounds interactions, the module may integrate previous chat history into the prompt. In particular, the system may summarize prior exchanges between the user and the Document Q&A module and inject the resulting summary into the large language model prompt alongside the current query, the retrieved document text, and the relevant language parameter. In one implementation, the summarization of prior exchanges is performed using Liunga, which condenses the last three user-assistant interactions into a compact representation. The summary is then used as contextual input for the current response generation process. By summarizing and incorporating recent conversational history, the system allows the assistant to maintain coherence across turns, resolve anaphoric references, and preserve topic continuity without requiring that the entire full-text transcript be supplied to the large language model. This approach improves efficiency while still allowing the assistant to respond as partof an ongoing conversation rather than as a sequence of isolated single-turn interactions.
[0185] In embodiments, the Document Q&A workflow applies a rule that prevents external knowledge integration. In This step may comprise constraining the assistant so that generated answers are grounded strictly in the provided document text and the conversational context supplied by the system, without incorporating uncited or external information from outside sources. This constraint is particularly important in regulated, scientific, medical, or compliance-sensitive settings, where the introduction of unsupported informationcould reduce trustworthiness or create regulatory risk. Accordingly, the prompt and response-generation logic may be structured so that a LLM operating as a chatbot in the Document Q&A module is directed to answer only on the basis of the retrieved documents and contextual inputs presented to it. This ensures that the resulting answers remain document-specific, evidence-based, and compliant with the intended role of the system as a document-grounded assistant rather than a general-purpose knowledge engine.
[0186] In embodiments, the Document Q&A module also stores compressed chat history in a database to support continuity in future sessions. In this process, recent interactions are summarized into a concise representation, again using Liunga in one implementation, and the summary is then persisted in a database rather than storing or repeatedly replaying full raw transcripts. By saving a compressed version of the last several exchanges, the system preserves key contextual information in a resource-efficient form. This stored compressed history can later be retrieved and incorporated into subsequent interactions, enabling the assistant to re-establish context over time while minimizing storage requirements and prompt length. Such a mechanism is particularly useful in enterprise conversational systems where repeated interactions about the same documents or projects may occur across multiple sessions and where efficient state persistence is desirable.
[0187] To support operational visibility and facilitate maintenance of the conversational pipeline, the Document Q&A module may include tracing and logging functionality. For each chat request processed by the system, a unique trace identifier may be generated, for example according to a pattern such as DEV_Finder_Chat_AL This trace identifier enables end-to-end tracking of individual conversational requests across the module’s workflow. In addition, the system may attach structured metadata including, for example, request identifiers, tool identifiers, module identifiers, and environment values. Such metadata enables contextual tracking and differentiation of requests across development, staging, and production environments. The tracing and logging layer may further be configured to capture latency and error information at the conversation level, thereby supporting performance monitoring, debugging, and auditability. By isolating Chat Al traces from traces associated with other system components, the system allows operators and developers to diagnose issues specific to the conversational workflow without conflating them with unrelated retrieval or optimization modules.
[0188] Overall, the Document Q&A module provides a structured conversational pipeline in which a user query is combined with retrieved document text and language information, the language of the query is automatically detected, the user’s intent is classified, explicit user-requested response languages override automatically detected languages when appropriate, recent conversational history is summarized and injected into the prompt, external knowledge sources are excluded so that answers remain grounded in the retrieved documents, compressed chat summaries are stored for future continuity, and tracing and logging mechanisms provide observability, performance monitoring, and debugging support. Through this architecture, the system enables a document-grounded conversational assistant that can answer user questions interactively, maintain contextual coherence across turns, operate in multiple languages, and do so in a controlled and auditable manner suitable for enterprise use.
[0189] In order to accelerate user review of documents returned by search, embodiments of the methods and systems of the disclosure comprise a Document Summary module configured to generate concise, query-focused summaries of a selected set of top-ranked documents. The purpose of this module is to reducethe time and effort required for a user to understand the substance of search results by synthesizing the most relevant information from multiple retrieved documents into a single coherent summary. Rather than requiring the user to manually open and inspect each result individually, the Document Summary module provides an aggregated overview that is specifically aligned with the user’s search intent and grounded in the retrieved evidence.
[0190] The Document Summary module may be configured to implement a method comprising a content ingestion step in which a user query is combined with a predetermined number of top ranked documents (e.g. the top five documents) returned by Search module. In embodiments, the system first identifies a ranked result set obtained by the search module responsive to the user’s query, and a predetermined number of the highest-ranked documents are selected as the basis for summarization. The content of these documents may then be injected into a prompt provided to a large language model, the prompt also including the user query itself. By including the user query in the summarization prompt, the system ensures that the resulting summary is not merely a generic abstract of the documents, but instead is focused on those aspects of the documents that are most relevant to the user’s expressed information need. By including the text of the top-ranked documents, the module further ensures that the summary is evidence-based and derived from the actual retrieved materials rather than from generalized or external knowledge. In this arrangement, the top documents may represent the documents that the Search module has determined to be most relevant according to the applicable ranking logic, which may include lexical matching, semantic similarity, metadata weighting, or other retrieval criteria. The use of the highest-ranked subset allows the summarization process to operate on a curated body of content that is likely to contain the most responsive information. The selected documents may be provided to the large language model as source material from which the summary is to be generated. In this manner, the Document Summary module functions as a downstream interpretive layer operating on the output of Search Services, converting a multi-document result set into a compact narrative form that is easier for the user to consume.
[0191] After the content ingestion stage, the Document Summary module may perform a summary generation step. In this step, a large language model synthesizes the content of the top documents into a single, unified summary. The generated summary can be configured to be concise while still conveying the core information relevant to the user query (i.e. the prompt provided to the LLM may request the generated summary to have these features). Rather than reproducing document text verbatim or providing separate summaries for each retrieved document in isolation, the Document Summary module can integrate the salient points across the selected documents into one coherent response. This synthesis may include identifying overlapping themes, consolidating repeated facts, and foregrounding the information that most directly addresses the user’s request. At the same time, unnecessary details, duplicative passages, and tangential content are omitted so that the output remains focused and efficient for review.
[0192] The summary generation process can therefore be query-specific in two respects. First, the user query is explicitly included in the prompt, thereby guiding the large language model toward the aspects of the retrieved documents that bear most directly on the user’s request. Second, the documents being summarized are themselves the top search results selected in response to that same query. The resulting summary is thus doubly aligned with user intent: it is based on documents retrieved because of the query, and it is generated under instructions that reflect the query. This architecture improves usability becausethe user receives a targeted synthesis of the most relevant retrieved evidence rather than a broad or indiscriminate condensation of source materials.
[0193] The Document Summary module may generate a summary prompt designed so that the summary avoids unnecessary detail. In many document collections, especially in scientific, regulatory, medical, or enterprise content repositories, retrieved documents may contain extensive background material, standard disclosures, or repeated information that is not central to the user’s immediate question. The Document Summary module can reduce this burden by instructing the large language model to generate an output that is compact and focused. In other words, the summarization process can emphasize relevance and brevity while preserving sufficient substance to allow the user to understand the key content of the top results. This makes the module particularly useful as a first-pass review aid, enabling a user to decide quickly whether to inspect particular documents in greater depth.
[0194] To support operational observability and facilitate maintenance of the summarization workflow, the Document Summary module may further include tracing and logging functionality. For each summarization request, the system can generate a unique trace identifier, for example according to a pattern such as DEV_Finder_Summary. This trace identifier enables end-to-end tracking of the summarization request through the module and allows individual executions to be distinguished from one another across system logs and monitoring tools. In addition, the system attaches structured metadata such as requestjd, tool, module, and environment, thereby supplying contextual information that can be used to analyze processing behavior and diagnose issues.
[0195] The tracing and logging implementation can support environment-based monitoring across multiple deployment environments, including development, staging, and production. By associating each summarization request with structured metadata that indicates the relevant environment, the system enables operators and developers to compare behavior across deployment stages and isolate environment-specific issues. The Document Summary module’s tracing functionality can be further configured to capture latency and error information so that the performance and reliability of the summarization process can be monitored over time. This allows for the identification of slow-running requests, failures in prompt construction or model invocation, and other operational anomalies that may affect the quality or responsiveness of the Document Summary module. The use of dedicated tracing for the Summary module also provides clear separation from other modules in the broader platform, such as search, query optimization, or chat-based question answering. By isolating Summary-related traces, the system improves module-level visibility and allows debugging efforts to focus specifically on summarization workflows without conflating those workflows with upstream or parallel components. This separation is advantageous in complex enterprise search architectures where multiple Al-enabled services may operate in sequence or in parallel and where precise attribution of performance metrics and error states is important for maintenance and auditing.
[0196] Overall, the Document Summary module provides a summarization workflow in which the top five documents returned by search are combined with the user query and supplied to a large language model, which then produces a single concise and coherent summary focused on the user’s request and grounded in the retrieved document content. The module thereby accelerates review of search results by highlighting the most relevant information without requiring the user to inspect each returned document individually. Inaddition, the module incorporates a tracing and logging framework that assigns a unique trace identifier to each request, attaches structured contextual metadata, supports environment-aware monitoring, tracks latency and errors, and maintains clear separation of Summary-related observability data from other system components. Through this combination of query-focused multi-document summarization and robust operational monitoring, the Document Summary module improves both the usability and maintainability of the overall search platform.
[0197] The features disclosed in the foregoing description, or in the following claims, or in the accompanying drawings, expressed in their specific forms or in terms of a means for performing the disclosed function, or a method or process for obtaining the disclosed results, as appropriate, may, separately, or in any combination of such features, be utilised for realising the invention in diverse forms thereof.
[0198] While the invention has been described in conjunction with the exemplary embodiments described above, many equivalent modifications and variations will be apparent to those skilled in the art when given this disclosure. Accordingly, the exemplary embodiments of the invention set forth above are considered to be illustrative and not limiting. Various changes to the described embodiments may be made without departing from the spirit and scope of the invention.
[0199] For the avoidance of any doubt, any theoretical explanations provided herein are provided for the purposes of improving the understanding of a reader. The inventors do not wish to be bound by any of these theoretical explanations. Any section headings used herein are for organizational purposes only and are not to be construed as limiting the subject matter described. Throughout this specification, including the claims which follow, unless the context requires otherwise, the word “comprise” and “include”, and variations such as “comprises”, “comprising”, and “including” will be understood to imply the inclusion of a stated integer or step or group of integers or steps but not the exclusion of any other integer or step or group of integers or steps. It must be noted that, as used in the specification and the appended claims, the singular forms “a,” “an,” and “the” include plural referents unless the context clearly dictates otherwise. Ranges may be expressed herein as from “about” one particular value, and / or to “about” another particular value. When such a range is expressed, another embodiment includes from the one particular value and / or to the other particular value. Similarly, when values are expressed as approximations, by the use of the antecedent “about,” it will be understood that the particular value forms another embodiment. The term “about” in relation to a numerical value is optional and means for example + / - 10%.Examples
[0200] Vision-Language Models (VLMs) are poised to revolutionize the digital transformation of the pharmaceutical industry by enabling intelligent, scalable, and automated multi-modality content processing. Traditional manual annotation of diverse data modalities — including text, images, video, audio, and web links — is prone to inconsistencies, quality degradation, and inefficiencies in content utilization. The sheer volume of video and audio data further exacerbates these challenges, making manual labeling prohibitively labor-intensive and cost-inefficient. By harnessing the power of GenAI, particularly VLMs and Audio-Language Models (ALMs), the inventors introduced an advanced large-scale production system capable of autonomously analyzing content, generating high-fidelity captions, and assigning semantically rich tags. Not only does it improve natural language search precision, but it also redefines reasoning within information retrieval paradigms, driving superior knowledge accessibility and operational efficiency across the pharmaceutical ecosystem.
[0201] In Example 1 , the inventors first demonstrate a large-scale GenAI platform that processed over 200,000 PDF data, 25,326 video data across 8 various formats (e.g., MP4, M4V etc.) and multilingual audio file covering > 20 languages. Additionally, the inventors present 4 key finding (Multi-modality, Attention Mechanism, Temporal Reasoning, Trade-off of Long Video Split), introduce a novel framework for video content reasoning using VLM / ALM, and open a new task domain on the SOTA Video-MME benchmark (Fu et al., 2024a). The results provided not only enhance technical understanding of VLM in video content reasoning, but also offer practical, scalable business solutions for managing vast amounts of video data under similar GPU constraints, ultimately facilitating video derivative creation.
[0202] Example 2 describes in more detail an example content finder system according to embodiments of the disclosure.
[0203] Example 1
[0204] A. Example implementation
[0205] An example implementation is shown in Figure 3. The following example implementation is built on the AWS Cloud platform, but any suitable platform or processing device may be used. Figure 3 illustrates a solution architecture blueprint of underlying tech stack on LLM and VLM for Natural Language (NL) search. This architecture enabled content search for 200,000 PDFs, 25,326 videos and 888 multilingual audio files covering more than 20 languages.
[0206] An example process begins with uploading documents to an S3 Source Bucket. This triggers a Lambda function that initiates jobs in AWS Batch. An Asset Extraction Pipeline uses Textract, Docling, and PymuPDF to extract text, images, tables, and metadata, storing results in S3 and MongoDB. The Captioning Pipeline then generates captions using Phi-3. Documents are translated to English using MBart and nllb models. The Tagging Pipeline then creates extractive and abstractive tags using Llama 3. Finally, the Vectorization Pipeline chunks data, embeds vectors, and stores them with tags in a Vector Database. The processed data is directed to a Target Bucket.
[0207] The process illustrated on Fig. 3 also differentiates between videos with and without transcription and includes a Video Length identifier. By using services like FFmpeg for Video Convert, Audio Language Model(Whisper turbo or Whisper Large V2), Qwen video model (VLM) to generate each video summary and key frame information. Using Claude 3 sonnet v2 to further refine the quality of summary output. Later embedding into Tagging pipeline.
[0208] The process can comprise the following steps:
[0209] 1. Upload documents to an S3 bucket, triggering an AWS Lambda function that submits jobs to AWS Batch.
[0210] 2. In the extraction pipeline, text, images, and tables are extracted using Docling and generating PDF metadata using PymuPDF. The results are stored in S3 and MongoDB. The jobs are then submitted to the captioning pipeline.
[0211] 3. Generate captions for images and tables using Microsoft Phi-3 Vision. The results are stored in S3 and MongoDB. The jobs are then submitted to the Translation pipeline.
[0212] 4. The document will be translated into English by using MBart and nllb model. Post translation the job is submitted to the tagging pipeline.
[0213] 5. In the tagging pipeline, two types of tags (extractive and abstractive) are created using Meta-Llama- 3-BB. The results are stored in S3 and MongoDB. The jobs are then submitted to the vectorization pipeline.
[0214] 6. In the vectorization pipeline, the data is split into chunks, embedded as vectors, and stored along with tags in a vector database.
[0215] 7. For video and audio data, it also differentiates between videos with and without transcription and includes a Video Length identifier. By combining services FFmpeg for Video Convert, Audio Language Model (Whisper turbo or Whisper Large V2), Qwen video model (VLM) to generate each video summary and key frame information, and using Claude 3.5 Sonnet v2 to further refine the quality of summary output. Then later embedding into Tagging pipeline.
[0216] An example auto-tagging pipeline architecture is illustrated on Figure 4. Extractive tags are generated through exact keyword or synonym matching within the text. They are further divided into two types: (1) Product Tags: Tags related to specific drugs, treatments, or products mentioned explicitly in the document. (2) Indication Tags: Tags representing medical conditions or diseases associated with the mentioned product or treatment. Example: If a document mentions “Ocrevus” or its synonyms, a Product Tag for “Ocrevus” is assigned. If it also mentions “cardiovascular” an Indication Tag for that document is added.
[0217] Abstractive tags are assigned based on the contextual meaning of the document and the definition of the tag categories. Unlike extractive tags, they do not rely solely on direct keyword matching. There are five types of abstractive tags: (1) Audience: Identifies the intended readership, such as Consumer, patients, Healthcare / Regulatory Authorities, Nurse, etc. (2) Content Purpose: Describes the objective of the document (e.g., Disease Awareness, Product Information, Clinical Trials). (3) Content Complexity: Indicates the level of technical detail (e.g., Introductory, Intermediate, Advanced, Expert). (4) Non-Clinical Topics: Captures themes not directly related to clinical or medical information (e.g., Lifestyle, E-Learning, legal). (5) Clinical Topics: Covers key medical or scientific subject matter (e.g., Mechanism Of Action, Disease Stage, Diagnostics).Example: A document outlining the legal implications of disclosing patient disease stage information under new data protection regulations.
[0218] Tags assigned: Audience: Administrator, Healthcare / Regulatory Authorities; Content Purpose: Market Access; Content Complexity: Advanced; Non-Clinical Topic: Legal; Clinical Topic: Disease Stage.
[0219] This hybrid tagging approach ensures comprehensive coverage of both surface-level mentions (extractive) and deeper semantic themes (abstractive), resulting in a richer and more meaningful categorization of documents.
[0220] Most leading vision language model are only reading static frame by frames images of video, and summarize the output (ChatGPT), or not encoding audio input at all due to capability (Qwen-2) or assembling them separately, in a very costly manner (Gemini). Current approaches do not fully cover visual, audio, metadata which leads to poor reasoning ability. For example, summaries may only cover very shallow information or logo, or very general information, writing low information content summarise and making mistakes in time stamps. The present methods avoid this using the following steps (where exact implementation details are only exemplary):
[0221] 1. Feed videos into Qwen model to get initial summary and their key frame information, write a draft version Json.
[0222] 2. Extract the metadata of this video by using tool like EXIF to obtain info like who created this video, what resolution, what length of this video etc - creating e.g. attributes of the video.
[0223] 3. Use FFMPEG tool to extract this video into a .mp3 audio fie, which basically got all the audio information, then feed this into an Audio Language Model (e.g. Whisper Large v2 or Whisper Turbo) to do voiceover transcription (if video has any voiceover or audio information).
[0224] 4. Feed those three Json files (one Json fie recorded all metadata; one Json file recorded audio transcription information, one Json file recorded a first draft summary and key frame information) into one LLM model, which takes in text and outputs text more comprehensively. In the present examples the inventors used Claudia Sonnet on AWS Bedrock Model API.
[0225] 5. Use additional negative prompting to force the LLM to not focus on shallow information like logos or font size or color etc., and set the tone for reasoning (e.g. pharma expert), do no hallucinate on fact etc. For example, the inventors used Claude LLM and asked the model to finetune the summary Json and output a better summary Json. For example, Claude LLM takes as input the VLM output, the ALM output, and the metadata and is prompted to output a more concise and improved summary.
[0226] Metadata - additional attributes
[0227] An example of additional attributes extracted from digital video content as part of the metadata extraction step, e.g., with tools such as EXIF, is provided below:
[0228] [ {"SourceFile": " / tmp / (M-ID-00000229-07-2021) HCC Series 2021 - Webinar 4. mp4", "ExifToolVersion": 12.4,"FileName": "(M-ID-00000229-07-2021) HCC Series 2021 - Webinar 4. mp4", "Directory":
[0229] " / tmp", "FileSize": "204 MiB","FileModifyDate": "2024:11 :26 13:57:22+00:00", "FileAccessDate":
[0230] "2024:11 :26 13:57:22+00:00", "FilelnodeChangeDate": "2024:11 :26 13:57:22+00:00", "FilePermissions": "-rw - ","FileType": "MP4","FileTypeExtension": "mp4","MIMEType": "video / mp4","MajorBrand": "MP4 v2 [ISO 14496-14]", "MinorVersion": "0.0.0","CompatibleBrands": [ "mp42", "mp41"],"MovieHeaderVersion":0, "TimeScale": 90000, "Duration": "0:29:17", "PreferredRate": 1 , "PreferredVolume":
[0231] "100.00%","PreviewTime": "0 s","PreviewDuration": "0 s","PosterTime": "0 s","SelectionTime": "0 s","SelectionDuration": "0 s","CurrentTime": "0 s","NextTracklD": 3,"TrackHeaderVersion":
[0232] O,"TrackCreateDate": "2021 :08:01 12:33:44", "TrackModifyDate": "2021 :08:01 12:33:44", "TrackID":
[0233] 1 ,"TrackDuration": "0:29:17", "TrackLayer": 0, "TrackVolume": "0.00%", "ImageWidth": 854,"lmageHeight": 480,"GraphicsMode": "srcCopy","OpColor": "0 0 0","CompressoriD": "avc1","SourcelmageWidth":
[0234] 854,"SourcelmageHeight": 480,"XResolution": 72,"YResolution": 72,"CompressorName": "AVC Coding", "BitDepth": 24, "Matrixstructure": "1 00 0 1 00 0 1","MediaHeaderVersion":
[0235] 0,"MediaCreateDate": "2021 :08:01 12:33:44", "MediaModifyDate": "2021 :08:01
[0236] 12:33:44", "MediaTimeScale": 48000, "MediaDuration": "0:29:17", "MediaLanguageCode": "eng", "Balance": 0,"HandlerType": "Alias Data","HandlerDescription": "Alias Data Handler", "AudioFormat":
[0237] "mp4a", "AudioChannels": 2,"AudioBitsPerSample": 16,"StartTimecode": "00;00;00;00","XMPToolkit": "Adobe XMP Core 7.0-c00079.217bca6, 2021 / 06 / 14-18:28:11 "."CreateDate": "2021 :08:01
[0238] 19:33:44+07:00", "ModifyDate": "2021 :08:01 19:33:46+07:00", "MetadataDate": "2021 :08:01
[0239] 19:33:46+07:00", "VideoFrameRate": 29.97003, "VideoFieldOrder":
[0240] "Progressive", "VideoPixelAspectRatio": 1 ,"AudioSampleRate": 48000, "AudioSampleType": "16-bit integer", "AudioChannelType": "Stereo", "StartTimeScale": 30000, "StartTimeSampleSize":
[0241] 1001 'Orientation": "Horizontal (normal)", "InstancelD": "xmp.iid:a25fd722-b1ce-d145-8022-06043dc9abd0","DocumentlD": "a63757ce-ba7f-1238-fd99-3ebd00000099","OriginalDocumentlD":
[0242] "xmp.did:d2731bc6-94a8-f848-8277-0450ce0e34a7", "Format": "H.264","DurationValue":
[0243] 158108160, "Durationscale": 0.000011111111111111."ProjectRefType": "Movie", "VideoFrameSizeW": 854,"VideoFrameSizeH": 480, "VideoFrameSizeUnit": "pixel", "StartTimecodeTimeFormat": "29.97 fps (drop)","StartTimecodeTimeValue": "00;00;00;00","AltTimecodeTimeValue":
[0244] "00;00;00;00","AltTimecodeTimeFormat": "29.97 fps (drop)","HistoryAction": [ "saved", "created", "saved", "saved"], "HistorylnstancelD": [ "8d91f338-1d1c-f542-97f8-37d3000000c6", "xmp.iid:e5cb31e1-598a-3149-b025-bf58e110cdc2", "xmp.iid:972dbdbb-3294-bb4d-b991-b037869dbae7", "xmp.iid:a25fd722-b1ce-d145-8022-06043dc9abd0"],"HistoryWhen": [ "2021 :08:01 19:33:46+07:00", "2021 :08:01 19:29:33+07:00", "2021 :08:01 19:33:45+07:00", "2021 :08:01
[0245] 19:33:46+07:00"], "HistoryChanged": [ " / ", " / ", " / metadata"], "IngredientsInstancelD": [ "142f2555-8a88-51 d8-0881 -e8d00000008c", "142f2555-8a88-51 d8-0881 -e8d00000008c", "8d0d6270-5978-2f6d-e802-689c00000096", "8d0d6270-5978-2f6d-e802-689c00000096", "xmp.iid:1fa014ce-f115-8b45-9d3f-07cc9b38b362", "xmp.iid:b3115d47-f390-694e-95de-9d66ac90af58", "xmp.iid:c152a4a0-e530-ff49-aacc-f496bb0ed6dc"],"lngredientsDocumentlD": [ "7dddf81 e-c552-54a4-b6c0-aa660000005f", "7dddf81e-c552-54a4-b6c0-aa660000005f", "44b342b5-3acf-423f-8676-f85f00000069", "44b342b5-3acf-423f-8676-f85f00000069", "adobe:docid:photoshop:0db1da5f-a4e7-9644-abee-01adda2e2af9", "adobe:docid:photoshop:76e82851-450e-b74b-98ea-52a1b8a7082e", "adobe:docid:photoshop:06df38f5-55ee-144a-a437-54db83cb4ec7"],"lngredientsFromPart": [ "time:0d4427942815296001254016000000", "time:0d442794281529600f254016000000",
[0246] "time:9314249712768001254016000000d11357394048001254016000000",
[0247] "time:9314249712768001254016000000d11357394048001254016000000", "time:0", "time:0", "time:0"],"lngredientsToPart": [ "time:7628100480001254016000000d4427942815296001254016000000", "time:762810048000f254016000000d4427942815296001254016000000", "time:445099663008000f254016000000d11357394048001254016000000", "time:445099663008000f254016000000d11357394048001254016000000",
[0248] "time:4435570915776001254016000000d7628100480001254016000000",
[0249] "time:4443199016256001254016000000d7628100480001254016000000", "time:0d762810048000f254016000000"], "IngredientsFilePath": [ "HCC Series - Webinar 4. mp4", "HCC Series - Webinar 4. mp4", "HCC Series 2021 - Webinar 2. mp4", "HCC Series 2021 - Webinar 2. mp4", "AE Disclaimer.psd", "Ml Disclaimer.psd", "Opening Disclaimer.psd"],"lngredientsMaskMarkers": [ "None", "None", "None", "None", "None", "None", "None"],"PantryProjectRefType":
[0250] "Movie", "PantrylngredientsFilePath": "VIDEO SIGNATURE. mp4", "PantryWindowsAtomExtension":
[0251] ".prproj", "Pantry WindowsAtomlnvocationFlags": " / L", "Pantry WindowsAtomUncProjectPath":
[0252] "\\\\?\\E:\\DDPRoche\\ROC-154845 - HCC Series 2021\\007_Edit\\Webcast 2\\V4\\Premiere\\ROC-154845-HCC Series 2021 Consolidating Care Webcast 2 V4a. prproj", "PantryMacAtomApplicationCode":1347449455, "PantryMacAtomlnvocationAppleEvent": 1129468018, "PantryVideoFrameRate":
[0253] 25,"PantryVideoFieldOrder": "Progressive", "PantryVideoPixelAspectRatio": 1 ,"PantryAudioSampleRate": 48000, "PantryAudioSampleType": "16-bit integer", "PantryAudioChannelType":
[0254] "Stereo", "PantryStartTimecodeTimeFormat": "25 fps","PantryStartTimecodeTimeValue":
[0255] "00:00:01 :00","PantryVideoFrameSizeW": 1920,"PantryVideoFrameSizeH":
[0256] 1080,"PantryVideoFrameSizeUnit": "pixel", "PantryOriginator": "Pro Tools", "PantryOriginatorReference": "aaOGK3B3ksPk","PantryOriginationDate": "2019-01-15", "PantryOriginationTime":
[0257] "14:55:33", "Pantry TimeReference": 10848000, "PantryVersion": O,"PantryStartTimeScale":
[0258] 3000,"PantryStartTimeSampleSize": 100,"PantryDurationValue": 6305900, "PantryDurationScale":
[0259] "0.000333333333333333", "PantryAltTimecodeTimeValue":
[0260] "00:00:00:00", "PantryAltTimecodeTimeFormat": "30 fps","PantryRenditionClass":
[0261] "proof:pdf',"PantryNPages": 1 ,"PantryHasVisibleTransparency": false, "PantryHasVisibleOverprint": true,"PantryType": "Document", "PantryThumbnailWidth": 256,"PantryThumbnailHeight":
[0262] 136,"PantryThumbnailFormat": "JPEG","PantryThumbnaillmage": "(Binary data 12324 bytes, use -b option to extract)", "PantryDerivedFromlnstancelD": "xmp.iid:73b0f4b4-e05a-4bfd-b838-f766eb07beaa","PantryDerivedFromDocumentlD": "xmp.did:73b0f4b4-e05a-4bfd-b838-f766eb07beaa","PantryDerivedFromOriginalDocumentlD":
[0263] "uuid:D2440A1961 BA11 DA89698D6D216BE137","PantryMaxPageSizeW":
[0264] 76.895217, "PantryMaxPageSizeH": 39.998928, "PantryMaxPageSizeUnit":
[0265] "Millimeters", "PantryPlateNames": "Magenta", "PantrySwatchGroupName": "Default Swatch
[0266] Group", "PantrySwatchGroupType": 0,"PantrySwatchColorantSwatchName":
[0267] "Schwarz", "PantrySwatchColorantMode": "RGB","PantrySwatchColorantType":
[0268] "PROCESS", "PantrySwatchColorantRed": 15,"PantrySwatchColorantGreen":
[0269] 15,"PantrySwatchColorantBlue": 15,"PantryCreatorTool": "Adobe Photoshop 22.4
[0270] (Windows)", "PantryCreateDate": "2021:07:3009:27:51 +07:00", "PantryModifyDate": "2021:07:30
[0271] 10:23:28+07:00", "PantryMetadataDate": "2021 :07:30 10:23:28+07:00", "PantryColorMode":
[0272] "RGB","PantrylCCProfileName": "sRGB IEC61966-2.1","PantryDocumentlD":
[0273] "adobe:docid:photoshop:06df38f5-55ee-144a-a437-54db83cb4ec7","PantryOriginalDocumentlD":
[0274] "xmp.did:88fd9bab-3270-cc45-8350-34b93cfdf65d", "PantryOrientation": "Horizontal
[0275] (normal)", "PantryXResolution": 72,"PantryYResolution": 72,"PantryResolutionUnit":
[0276] "inches", "PantryColorSpace": "sRGB","PantryExifl mageWidth": 1280,"PantryExiflmageHeight":
[0277] 720,"PantryTextLayerName": "Disclaimer", "PantryTextLayerText":
[0278] "Disclaimer", "PantryDocumentAncestors": "adobe:docid:photoshop:1d49338a-98e8-5f46-b3f3-d008059ac96c","PantryHistoryParameters": "from image / png to
[0279] application / vnd. adobe. photoshop", "PantryHistory Action": "saved","PantryHistorylnstancelD":
[0280] "xmp.iid:c152a4a0-e530-ff49-aacc-f496bb0ed6dc","PantryHistoryWhen": "2021 :07:30 10:23:28+07:00", "PantryHistorySoftwareAgent": "Adobe Photoshop 22.4
[0281] (Windows)", "PantryHistoryChanged": " / "."PantrylngredientsRenditionClass": "proof:pdf","PantryFormat": "application / vnd. adobe. aftereffects. layer", "PantrylnstancelD": "xmp.iid:fe1e9978-57cf-482e-b7f7-b0cfafde5a3d","PantryTitle": "20190115_Roche_Jingle.wav", "PantrylngredientsInstancelD": "c7abc7da-937e-00e9-c7b1 -77090000008a", "PantrylngredientsDocumentID": "4a086337-19f3-7277-58ec-af0c0000005d","PantrylngredientsFromPart": "time:25600f25600d128000f25600", "PantrylngredientsToPart": "time:25600f25600d128000f25600","PantrylngredientsMaskMarkers":
[0282] "AII","DerivedFromlnstancelD": "xmp.iid:e5cb31e1-598a-3149-b025-bf58e110cdc2","DerivedFromDocumentlD": "xmp.did:e5cb31e1-598a-3149-b025-bf58e110cdc2","DerivedFromOriginalDocumentlD": "xmp.did:e5cb31e1-598a-3149-b025-bf58e110cdc2","WindowsAtomExtension": ".prproj","WindowsAtomlnvocationFlags":
[0283] "ZL","WindowsAtomUncProjectPath": "\\\\?\\D:\\VIDEO PROJECTWHCC Webinar SeriesWHCC Webinar Series 4.prproj","MacAtomApplicationCode": 1347449455, "MacAtomlnvocationAppleEvent":
[0284] 1129468018, "MediaDataSize": 213066766, "MediaDataOffset": 832464, "ImageSize":
[0285] "854x480", "Megapixels": 0.41."AvgBitrate": "970 kbps", "Rotation": 0 }]
[0286] The implementation described in this example offers several objective technical advantages that further enhance performance and efficiency of the architecture.Firstly, the example architecture is designed for scalability. By utilizing batch processing and interconnected pipelines, it can handle large volumes of data efficiently. Each pipeline can be scaled independently based on the workload, ensuring optimal resource utilization and maintaining high performance even under heavy data loads.
[0287] The modularity of the implementation is another key advantage. With distinct pipelines dedicated to specific tasks such as asset extraction, captioning, translation, tagging, and vectorization, the system promotes modularity. This design allows for easier maintenance and updates, as well as the integration of new technologies without disrupting the entire system. Each pipeline can be modified or upgraded independently, enhancing the system's adaptability and longevity.
[0288] Parallel processing is a further key advantage of the implementation. By triggering jobs in a batch processing system, the system supports parallel processing, significantly reducing the time required to process large datasets. This capability enhances overall system performance, making it faster and more efficient.
[0289] Automation is seamlessly integrated into the workflow. The automated triggering of functions upon document upload streamlines the process, reducing the need for manual intervention. This increases efficiency and minimizes the potential for human error, ensuring a smooth and reliable operation.
[0290] B. Benchmark and experimental settings
[0291] Vision-Language Models (VLMs) have advanced significantly, with several state-of-the-art models demonstrating strong performance across multiple multimodal tasks (Fu et al., 2024a). Qwen-VL (Bai et al., 2023), a series of large-scale VLMs developed by Alibaba, has shown remarkable improvements in vision-language understanding through extensive pretraining and large-scale data alignment. The latest Qwen2-VL (Bai et al., 2023) model achieves competitive performance across various benchmarks, indicating the effectiveness of large-scale multimodal training. Recent multi-modal models, such as CLIP (Radford et al., 2021), BLIP (Li et al., 2022), FLAVA (Singh et al., 2022), and OFA (Wang et al., 2022), have demonstrated strong zero-shot and fine-tune performance in various vision-language tasks. However, newer architectures, such as Gemini 1.5 Pro (Gemini Team, Google: Petko Georgiev and 1135 other authors, 2024), AdaReTaKe (Lourentzou et al., 2021), and Qwen2-VL (Bai et al., 2023), have pushed the boundaries of video-based multimodal understanding by improving reasoning, temporal alignment, and multimodal fusion strategies. As shown in recent leaderboards (Fu et al., 2024a), Qwen2-VL (Bai et al., 2023) achieves state-of-the-art results in various video comprehension benchmarks, rein- forcing its role as a leading vision-language model.
[0292] Several benchmarks have been proposed to systematically evaluate the capabilities of MLLMs. Video-MME (Fu et al., 2024a) introduces an evaluation framework specifically designed for VLMs, assessing their understanding of dynamic and multi-modal content. Additionally, MME (Fu et al., 2023) provides a thorough benchmark for evaluating general multimodal LLMs, while MME-Survey (Fu et al., 2024b) offers a detailed review of existing evaluation methodologies. The MME-RealWorld benchmark (Zhang et al., 2024a) further extends this evaluation to real-world, high-resolution scenarios, testing the robustness and generalization of multi-modality beyond synthetic datasets.This example implementation primarily adopts two well-established MLLM benchmarks: Video-MME (Fu et al., 2024a) and MMBench (Yuan Liu, 2023), along with evaluations of more than 40+ VLMs.
[0293] Video-MME (Fu et al., 2024a) is the first full-spectrum multi-modal evaluation benchmark designed specifically for video-based MLLMs. It stands out from existing benchmarks with several key features: (1) Diversity in video types, covering six primary visual domains with 30 subfields to ensure broad scenario generalizability; (2) Temporal coverage, including short-, medium-, and long-term videos ranging from 11 seconds to 1 hour, allowing for robust contextual dynamics. Video-MME includes 900 manually selected videos totalling 254 hours, annotated with 2,700 question-answer pairs. Extensive evaluations of state-of-the-art MLLMs, including GPT-4 series (Ouyang et al., 2022), Gemini 1.5 Pro (Gemini Team, Google: Petko Georgiev and 1135 other authors, 2024), Intern VL-Chat-V1.5 (Chen et al., 2024), and LLaVA-NeXT-Video (Zhang et al., 2024b), reveal that Gemini 1.5 Pro is the best-performing commercial model and Qwen the best open-source model.
[0294] MMBench (Yuan Liu, 2023) is designed to assess the findings across diverse visual understanding tasks, including video recognition, captioning, visual question answering (VQA), and reasoning. Unlike conventional benchmarks, MMBench incorporates fine-grained difficulty annotations, allowing for a more precise evaluation of model strengths and weaknesses. The Video-MME dataset consists of 900 videos spanning six primary visual domains with 30 subfields categorized into 300 short-term (<2 min), 300 medium-term (4-15 min), and 300 long-term (30-60 min) videos. In contrast, the MMBench-Video dataset comprises approximately 609 videos across 16 major categories, with durations ranging all from 30 seconds to 6 minutes. An example Video-MME raw dataset structure is shown in Table 5.
[0295] Furthermore, the inventors’ findings have been tested on industry data, spanning 14 disease areas and encompassing over 25,326 industry-proprietary video data. These videos vary significantly in duration, ranging from less than 2 minutes to over 2 hours, and include more than eight different formats. Additionally, the dataset includes 888 industry-proprietary audio recordings (see Table 1).
[0296] C. Business and technical impact
[0297] In today’s information-intensive healthcare landscape, finding relevant and reliable content is a major challenge for healthcare professionals (HCPs) and patients. Traditional search methods are inefficient, leading to wasted time and under-utilized assets. The VLM for video reasoning streamlines content discovery, asset reusability, and information accessibility. Early evaluations show that, in this example, the GenAI assistance system according to aspects of the present disclosure reduced the time required to create patient materials by 66%, streamlining content generation and enhancing efficiency. The example implementation is benchmarked using Video-MME, consisting of 300 short, 300 medium-length and 300 longer videos. The end-to-end processing time, from voice abstraction by FFmpeg, Whisper Turbo transcription and Bedrock LLM improvement on VLM video summary & captions storage in the database, averaged 2.209 minutes per longer video, 1.752 minutes per medium-length video, 1.679 minutes per short video. According to the inventors’ findings, a 94.44% reduction in effort for longer videos compared to manual inspection and an 88% reduction for overall video categories was achieved. The total dataset size was 94.42GB GiB, and manual inspection would have required 252.55 hours. Scaling this approach to the entire dataset, assuming linear scalability, the inventors estimate a total processing time savings of c.a.
[0298] 224.25 hours.D. Evaluation
[0299] The example implementation was evaluated with the top two benchmarks, Video-MME (Fu et al., 2024a) and MMBench (Yuan Liu, 2023) for multimodal vision language models (MVLMs), focusing on widely recognized 40+ vision language models (VLMs). Furthermore, the inventors conducted over four iterations of experiments using their proprietary dataset, which includes the data presented in Table 1. Table 9 shows training details and hyper-parameters of experiments.
[0300] When evaluated on the video-MME dataset (900 videos, in total 300 short 1300 Longer I and 300 Medium), the time saving in processing videos with the reasoning-based framework described herein compared to manual processing can be summarized as follows:
[0301] Processing time for Longer Video avg / per each: 45.51s , Short Video avg / per each: 13.714s Voice Over abstraction by Ffmpeg avg / per each: 23.172s
[0302] Transcription by Whisper turbo avg / per each: 39.504s
[0303] Bedrock Claude Sonnet Improve finetune and improve summary quality avg: 24.365 s Total time cost avg: 2.2092 Mins Per Long Video *300 = 11.045H; 1.6792 Min Per Short Video * 300 = 8.396H; 1 .75201 Min Per Medium Video * 300 = 8.76H
[0304] Manual inspection avg: 198.79 H (Long) / 46.89 H (Medium) / 6.875 H (Short)
[0305] - Video total Length: 198.79 H (Long) 146.89 H (Medium) 16.875 H (Short)
[0306] - Total Video Video-MME in GiB: 94.43Gb
[0307] - Total Length in H: 252.55H = 10 Days!
[0308] - Total Savings = (252.55 - 11.045 - 8.396 - 8.76) H = 224.25 H (> 88 % Savings of Time)
[0309] In summary, the inventors have tested the framework with 94.43 GB of open-source video data which needed to be processed real time and extract the metadata when someone is querying from the front end. This process according to the external tools and tech available takes around 252.55H hours (there is no external tools / standard approach for long video processing (Key frame identification, captioning, Video summarization etc.) at scale, so Human Expert Processing is the only way to do this; manually processing will take almost same time as length of video to analyze, process and share meta information). With the advanced technical implementation above the effort is reduced by 94.44% for longer video (above 30 minutes long), 18.11 % shorter video (less than 2 minutes) and 81.32% for others (2 minutes < length < 30 minutes). This results in a total time saving of 88.83% for 900 Videos Processing.
[0310] This finding was replicated on in house data comprising 1087 videos, in total 663 short / 91 Longer / and 333 Medium (140.67Gb). The end-to-end processing time (from metadata generation to video caption storage in database) for long videos was avg / per each: 3.76 Minutes, for medium videos avg / per each: 2.84 Min. Total time cost avg: Long Video = 5.68 H; Medium Video = 15.76H. By contrast manual inspection would take on average: 74.7 H (Long) / 55.98 H (Medium) / 11.54 H (Short) - total Length in H: 199 H = 8.3 Days. This is equivalent to total savings= (199 - 5.68 - 15.76 - 11.54) H= 166.02 H (> 83.42 % Savings of Time).
[0311] Category Details
[0312] VLM models covered 42 models: GPT series include GPT-4, Gemini 1.5 Pro. 2.0-Flash, Qwen- 7B-VL, Qwen-72B, Owen VL Max, LLaVA-Video, Oryx-1.5, InternVL 2.5,
[0313]
[0314] Aria, VideoLLaMA series, VideoChat Flash, NViLA, GPT-4o, Claude 3.5
[0315] Sonnet, TimeMarker, MiniCPM-V 3.2, MiniCPM-V 2.6, InternVL series, ST- LLaMAVideo-XL, VITA-1.5, Kangaroo, Video-CCAM, ShareGemini, SIMM, Chat-Uni-VL 1.5,VideoChat2 Mistral, ShareGPT-4Wideo
[0316] ALM models covered Whisper-turbo and Whisper-large V2
[0317] Number of videos 25,326
[0318] Number of audios 888
[0319] Covered variants Disease area (specialty): number of audio files, number of video files Oncology: 208, 8934
[0320] Cardiovascular: 1 , 14
[0321] Dermatology: 0, 30
[0322] Hematology: 67, 3606
[0323] Immunology: 144, 510
[0324] Infectious disease: 1 , 239
[0325] Inflammatory disease: 20, 222
[0326] Metabolism: 0, 6
[0327] Movement disorder: 9, 289
[0328] Neuroscience: 202, 4914
[0329] NA: 59, 2853
[0330] Ophthalmology: 159, 2862
[0331] Respiratory disease: 16, 467
[0332] Nephrology: 1 , 380
[0333] Covered video format 8 formats: MP4, M4V, QuickTime, WMV, WebM, MSVideo, MPG, and types 3GPP
[0334] Covered audio formats 4 audio formats: ’.mp3’, ’.wav’, ’.m4a’, ’.flac
[0335] Covered video lengths <2 minutes to over 3 hours
[0336] Language types Over 20 languages including German, Italian, English, Mandarin, Hokkien,
[0337] Hindi, Korean, French, Dutch, Spanish and more.
[0338]
[0339] Table 1. Data and models used for evaluation.
[0340] Multimodality. Combining metadata and voiceovers using LLM / ALMs could improve VLM summaries and understanding, though ALM alone can still achieve good content summary results. The inventors observed that almost all 42 VLMs, including the GPT-4 model, exhibit a significant positive number of improvements. It is crucial to note this distinct performance difference also in relation to video length. Overall, with voiceover transcription, the overall percentage increased from 58.4% to 62.3% (+3.9 points). For short videos, it rose from 67.7% to 70.0% (+2.3 points in acc.), for medium videos from 69.3% to 74.3% (+5.0 points), and for long videos from 61.7% to 69.6% (+7.9 points). As video length increases, performance significantly declines (from over 80% to less than 50%) highlighting the challenges in long video understanding. This trend underscores the difficulties models face in processing content. Notably, incorporating subtitle and audio information can significantly enhance performance, with improvements up to 7.9% observed in longer videos, as shown in Figure 10a, b,c, e. When further examining multimodalitywith subtask improvements, it was seen to improve in 8 / 12 task domains, such as action reasoning, action recognition, information synopsis, OCR problems, and object reasoning, with improvements ranging from 2% to 20%. Here, the benefits of ALMs are clear, though negative impacts can also be observed when incorporating ALMs, as VLMs still struggle significantly with temporal-related tasks, as shown in Figure 10 d. Overall, the accuracy of the Qwen-7B model (Bai et al., 2023) improved from 58.14% to 61.8%. However, the inventors noticed a decline in the completeness of answered questions, dropping from 817 to 699, as CUDA out-of-memory (OOM) issues frequently occurred when reasoning with both audio and visual information. The data on Fig. 3 shows that multimodality matters: combining metadata and voiceovers using LLM / ALMs can improve VLM summaries and understanding. This is especially effective for longer videos, and for tasks like action recognition, object reasoning, and OCR.
[0341] Table 8 shows examples results for summary and key frame tasks with and without using ALM. The data show that using ALM for voiceover transcription improves the relevance and precision of the summary content (see the italicised marked texts). However, timing remains a significant issue.
[0342] In summary, the inventors’ data show that combining metadata / voiceovers with LLM / ALMs enhances Vision Language Model (VLM) video understanding (specifically tasks like action recognition, object reasoning, and OCR). Voiceover transcription boosts accuracy from 58.4% to 62.3% (+3.9 points).
[0343] Attention Mechanism. Efficient attention mechanisms have been crucial for scaling large multimodal models (Face, 2024). FlashAttention-2 (Dao, 2023) and Scaled Dot-Product Attention (SDPA) (Vaswani et al., 2017) have played significant roles in improving efficiency in transformer-based architectures. FlashAttention reduces memory overhead and computational costs by optimizing key-query-value matrix operations, making it well-suited for large-scale applications. Similarly, SDPA, widely implemented in frameworks like Hugging Face’s Transformers, optimizes inference performance on GPUs, particularly with hardware accelerations e.g. AMD ROCm and NVIDIA TensorRT (Face, 2024). The VLM Community has shown significant interest in FlashAttention-2 (Dao, 2023), as it offers a most efficient implementation of standard attention mechanism by parallelizing computation over sequence length and optimizing GPU memory usage. While FlashAttention can significantly accelerate inference, most leading VLMs have been developed and benchmarked on high-end GPUs, such as NVIDIA A100, which inherently benefits from optimized tensor operations and memory bandwidth. However, FlashAttention-2 (Dao, 2023) also has strict GPU architecture dependencies, requiring Torch >3.6 and being optimized for NVIDIA A100 or select AMD GPUs (Instinct MI210, MI250 etc.). Additionally, dtype constraints limit usage to fp16 or bf16, making it less practical for many real-world industrial applications. The reality of deploying these mechanisms on standardized A10G Tensor Core GPUs — instead of NVIDIA A100 UltraCluster setups — presents significant cost and compatibility challenges. For instance, an A100 UltraCluster costs 40.96 Dollar per hour, whereas AWS G5 instances (A10G GPUs) range from 5.67 to 8.14 Dollar per hour, making it 8* more expensive. In general, FlashAttention-2 (Dao, 2023) performs best on A100 GPUs, making it a very popular choice. Here, Scaled Dot-Product Attention (SDPA) (Vaswani et al., 2017) proves to be a better fit for G5 instances, showing up to a 4* speed improvement. Notably, SDPA delivered higher accuracy (58.73%) than FlashAttention (54,81%) when FPS=0.1 for 2,700-Q&A test on Video-MME Benchmark, SDPA required only 4 hours and 37 minutes, compared to 7 hours and 40 minutes for FlashAttention-2, as shown in Table 2.These findings suggest that VLMs exhibit strict GPU architecture and FPS dependencies, meaning the optimal attention mechanism varies based on the hardware environment. SDPA can also invoke FlashAttention and memory-efficient attention kernels as needed, and native SDPA (Vaswani et al., 2017) support is expanding in Transformers, now being enabled by default for Torch versions > 2.1.1.
[0344] Experiments Processing time Total answered (%) Correct Answered (%) SDPA (0.1 FPS) 4h 37m 2s 37 58.73%
[0345] SDPA (0.01 FPS) 2h 12m 37s 87 48.40% FlashAttention (0.1 FPS) 7h 40m 17s 100 54.81% FlashAttention (0.01 FPS) 1h 50m 11s 100 48.55%
[0346]
[0347] Table 2. Comparison of speed, accuracy, and completeness across leading attention mechanisms on all length videos. Results from tests on Video-MME (which is a benchmark that includes 900 manually selected videos totalling 254 hours, annotated with 2,700 question-answer pairs).
[0348] The inventors implemented an ablation study on frames per seconds (FPS) to investigate attention mechanism dependence of FPS. Table 3 shows a comparison of speed, accuracy and completeness across leading attention mechanisms on short videos (<120s). This study highlights the strong dependence of FPS on each model’s performance. For short videos, FlashAttention is recommended over SDPA. Experiments on short videos Processing time Total answered (%) Correct answered (%) SDPA (1FPS) 44m 12s 6% 64.81% FlashAttention (1FPS) 1h 30m 12s 100% 70.78% Experiments on all length Processing time Total answered (%) Correct answered (%) videos
[0349] SDPA (0.1 FPS) 4h 37m 2s 37% 58.73% FlashAttention (0.1 FPS) 7h 40m 17s 100% 54.81%
[0350] SDPA (0.01 FPS) 2h 12m 37s 87% 48.40% FlashAttention (0.01 FPS) 1h 50m 11s 100% 48.55%
[0351]
[0352] Table 3. Ablation study on FPS. ‘Default setting as per the Video-MME benchmark (FPS=1), with no additional audio ALM transcription fed into the Qwen model.
[0353] Figure 5 depicts a schematic representation of the inventors’ findings in relation to the choice of attention mechanism used by the visual models for processing digital video content. As shown, Scaled Dot-Product Attention (SDPA) unexpectedly outperforms FlashAttention-2 when adopted in models processing video content longer than 120 seconds. Additional findings on the choice of graphics processing unit (GPU) chip, number of GPU instances, and the video’s frame rate per second are also included in Figure 5.
[0354] Time Localization Challenges in VLMs. The performance of VLMs was evaluated on speed, cost, and completeness. Accuracy scores for time localisation were assigned as follows:
[0355] Scores
[0356] liS(x) where S(x) = {1 if Sa>i= ga>i0 if Sa i\ = g
[0357] aa>iwhere S is the matching node score, a e A refers to a key frame or summary scenarios, g refers to the ground truth, and nais the total number of valid video output (e.g., if 74 videos have valid JSON outputs, we match key frames to verify timestamp accuracy and compare summary accordingly).
[0358] Fig. 11 shows results demonstrating that both top open-source and commercial models struggle with key frame detection, showing low accuracy (5-35%) and incorrect timestamps. Summaries are much more accurate, ranging from 75-95%. The findings indicate that Gemini-2-Flash is slower and more expensive, requiring 7-8 hours to process 300 videos at a cost of EUR 46.73 on Google Cloud. In contrast, Qwen-7B (Bai et al., 2023) demonstrated significantly faster processing, completing each video in under 1 minute when running on AWS EC2 g5.24xlarge, a GPU instance with 96 vCPUs, 384 GiB of memory, priced at USD 8.144 per hour. However, while Qwen-7B offers a more efficient processing speed, when running at 0.01 fps with SDPA attention, processing 300 videos took approximately 3-4 hours. In terms of output completeness, Gemini-2-Flash outperformed Qwen-7B, generating valid summaries for 297 / 300 videos (with acc. 94.6%), while Qwen-7B produced only 74 valid summaries (with acc. 74.3%). Additionally, key frame extraction remained challenging for both models, with Gemini-2-Flash achieving quality acc. 35.1% (26 / 74), while Qwen-7B reached only quality acc. 5.4% (4 / 74). For longer videos, Gemini-2-Flash processing required 3 minutes per video, whereas Qwen-7B was under 1 minute. Despite these differences, both top open & closed source VLMs struggle with accurate temporal reasoning, particularly in handling key frame alignment.
[0359] Table 6 shows a comparison of video summarisation between Gemini-2-Flash and Qwen-7B. In this example the timestamps are all wrong.
[0360] Gemini-2-Flash Qwen-7B
[0361] Summary
[0362] A stage performance of Snow White. The Evil A fairy tale performance, likely Snow White and the Queen consults the Magic Mirror, instructs Snow Seven Dwarfs. The video introduces characters, a White to clean the castle, and the story unfolds as forest scene, a confrontation between a queen and Snow White meets the Seven Dwarfs, receives the a prince, interactions between Snow White and the poisoned apple, collapses, and is revived by the dwarfs, and ends with a song.
[0363] Prince.
[0364] Key frames
[0365] 00:08 Magic Mirror reveals an angry face 00:00 Introduction of characters and setting 00:42 Snow White in rags looking at her 02:00 Scene with group of people in a forest stepmother 04:00 Confrontation between a queen and a 00:46 The Evil Queen on a castle balcony prince
[0366] 01 :27 Dopey Dwarf dancing in silk costume 06:00 Introduction of the dwarfs as Snow White’s 01 :37 Ethereal dancer twirling with a deer friends
[0367] 02:17 Snow White with basket approaching 08:00 Scenes of the dwarfs working and animals interacting with Snow White
[0368] 02:57 Snow White collapses onto a stage of 10:00 Snow White singing a song with the dwarfs rocks
[0369]
[0370] 03:42 Snow White at a wishing well
[0371] 05:09 The Evil Queen on a balcony speaking to
[0372] a soldier
[0373] 05:37 Snow White dancing in her new dress
[0374] 06:09 Snow White and her prince hold hands
[0375] 07:02 Snow White falls, animals mourn her
[0376] 07:26 The Prince awakens Snow White with a
[0377] kiss
[0378] 08:00 Snow White is held up for celebration
[0379] 08:07 Evil Queen standing on castle balcony
[0380] 09:05 Snow White lies in a glass coffin as
[0381] prince kneels
[0382]
[0383] Table 6. Comparison of Video Summarization: Gemini-2-Flash vs Qwen-7B. VideolD= P69idA8JO98, duration: long.
[0384] Trade-off of Long Video Split and Cut. Processing long videos presents significant challenges, particularly on GPU-constrained instances. Out-of-Memory (OOM) issues frequently occur, even when reducing frames per second (FPS) or lowering resolution. A common approach to mitigating OOM is video splitting, compression, or cutting. However, the analysis reveals that these methods do not result in meaningful speed gains but introduce new challenges related to temporal alignment. Splitting videos into multiple segments disrupts time consistency, making it even more difficult for VLMs to maintain coherent event sequences. Additionally, compressing video files using standard tools such as FFmpeg consumes substantial preprocessing time. Although shorter video segments can be processed individually, the lack of contextual continuity causes VLMs to focus on superficial elements such as logos, text color, or general stylistic attributes, leading to redundant descriptions and weaker key frame alignment.
[0385] The inventors performed an ablation study on frames per seconds to determine whether reducing FPS would help the Qwen model answer more questions correctly. The results are shown in Table 4. These show that reducing FPS does not necessarily help the model answer more questions correctly, and can have a negative impact as lower frames per second can lead to missing information. Here, the completeness percentage increased from 37% to 87%, but the accuracy dropped from 58.73% to 48.40%. However, with the support of audio ALM transcription, accuracy is maintained, improving from 58.73% to 61.80% when FPS=0.1. This further validates the findings above on multimodality from a different perspective.
[0386] Experiments on all lengths videos Total answered (%) Correct answered (%) SDPA (0.1 FPS) without audio transcription 37% 58.73%
[0387] SDPA (0.1 FPS) with audio transcription 26.78% 61.80%
[0388] SDPA (0.01 FPS) without audio transcription 87% 48.40%
[0389] SDPA (0.01 FPS) with audio transcription 68.92% 49.63%
[0390]
[0391] Table 4. Ablation study on FPS.
[0392] Field Details
[0393]
[0394] Video ID 001
[0395] Duration Short
[0396] Domain Knowledge
[0397] Sub-category Humanity & History
[0398] URL Youtube. com / watch?v=fFjy93ACGo8
[0399] VideolD fFjy93ACGo8
[0400] Question ID 001-2
[0401] Task Type Information Synopsis
[0402] Question What is the genre of this video?
[0403] Options
[0404] A It is a news report that introduces the history behind Christmas decorations.
[0405] B It is a documentary on the evolution of Christmas holiday recipes.
[0406] C It is a travel vlog exploring Christmas markets around the world.
[0407] D It is a tutorial on DIY Christmas ornament crafting.
[0408] Answer A
[0409]
[0410] Table 5. Video-MME raw dataset structure. Q / A details in Json format.
[0411] Component Output text
[0412] Prompt "Focusing on the content and workflow rather than specific logos or the color of text? (Qwen-7B After summarizing, list the key frames with brief captions in the format (00:00, caption). Model) Ensure the analysis is accurate and avoid including any assumptions or extrapolations.
[0413] Use a pharmaceutical expert domain perspective to enhance relevance and precision." Whisper " "id": 0, "start": 7.72, "end": 13.6, "text": " PCR of course refers to pathological complete Model (Voice- response where once the patient has surgery" ...
[0414] over "Full Transcription": " PCR of course refers to pathological complete response where Transcription) once the patient has surgery the pathologist does not find any cancer at all and pleasingly over the last sort of 15-20 years we’ve seen improvements in systemic treatment to such an extent that certainly for HER2 positive breast cancers we are now able to expect 50-60% of patients who have a PCR following the neoadjuvant treatment and we know that the patients who have a PCR are the ones who are likely to do the best in the long term."
[0415] Qwen Model VLM Output: "The video appears to be a medical demonstration, likely related to the Output Roche brand, which is known for its involvement in healthcare and diagnostics. The (Before video showcases a 3D model of a human body with various organs highlighted, including Transcription) the liver and brain. The model is displayed within a transparent box, possibly representing a medical device or a simulation environment.
[0416] Key Frames with Captions: 00:00 - Introduction: The video begins with a close-up of a transparent box containing a 3D model of a human body. 01 :00 - Organ Highlighting: The liver and brain are highlighted in the model, indicating a focus on these organs.
[0417] 02:00 - Interactive Display: The model rotates, providing a better view of the highlighted organs. 03:00 - Data Overlay: Digital text and graphics appear, likely representing
[0418]
[0419] medical data or information related to the highlighted organs. 04:00 - Conclusion: The video ends with a summary or conclusion, possibly emphasizing the importance of the highlighted organs or the role of the device in medical diagnostics. The video likely aims to educate viewers about the importance of the liver and brain in the human body and how the device or simulation can be used to study or diagnose related conditions." " Qwen Model VLM Output: "Here is a summary focusing on the content and workflow from a Output (After pharmaceutical expert’s perspective, along with key frames and brief captions" Transcription Summary: The video provides an explanation of the term "pathological complete Integration) response" (pCR) and its significance in cancer treatment, particularly for HER2-positive breast cancer. The speaker, a medical professional, defines pCR as the absence of any detectable cancer cells in the surgical specimen after neoadjuvant (pre-operative) systemic treatment. He highlights the remarkable progress made in systemic therapies over the past 15-20 years, enabling 50-60% of HER2-positive breast cancer patients to achieve pCR following neoadjuvant treatment. The speaker emphasizes that patients who achieve pCR have the best long-term outcomes, underscoring the importance of this endpoint in cancer management.
[0420] Key Frames: (00:07) Introduction to the concept of pathological complete response (pCR). (00:19) Progress in systemic treatments improving pCR rates. (00:24) Specifically mentioning 50-60% pCR rates in HER2-positive breast cancer. (00:33) Linking pCR to better long-term outcomes for patients."
[0421]
[0422] Table 8. With and Without ALM for Summary and Key Frame.
[0423] Token
[0424] Model description Type API price in dollars Modality limit
[0425] GPT-4 Turbo, the latest GPT-4 g pt-4- Input token Input 0.06 / K Tokens. Text Generation, model with im- proved 1106- limit:128K Output 0.12 / K Chat Completion, instruction, reproducible outputs, preview Tokens Image to Text parallel function calling. Returns
[0426] max of 4,096 output tokens.
[0427] Training data up to Apr 2023
[0428] Qwen2-VL. updated on Qwen2-VL- Input token Open source Text Generation, Huggingface Jan. 2025 7B limit:32K Video to Text Gemini 2.0 Flash Model, Gemini 2.0 Input token Per 1M tokens in Audio, images, released data 05th Feb 2025 Flash 001 limit:1048K USD: API Cost 0.10 video, text and Output (text / image / video) PDF to text token limit: 0.70 (audio). Output
[0429] 8K 0.40. e.g. 300 Long
[0430] Videos from Video- MME costed 56.62
[0431]
[0432] Table 9. Backbones, hyper-parameters, and prompt settings of samples of state-of-the-art VLMs used in the above experiments. Example prompt: Could you please provide a summary of this video, focusing on the content and workflow rather than specific logos or the color of text? After summarizing, list the keyframes with brief captions in the format (00:00, caption). Ensure the analysis is accurate and avoid including any assumptions or extrapolations. Use an expert domain perspective to enhance relevance and precision. Do not repeat sentences or focus on QR codes or logos.
[0433] As of now, processing long-form video content contextually on edge devices is largely limited to basic tasks such as object detection. However, the integration of multimodal technologies, specifically Vision-Language Models (VLMs), Large Language Models (LLMs), and Audio-Language Models (ALMs) as described herein, has the potential to revolutionize this space by enabling deeper, context-aware analysis directly at the edge. For instance, in prenatal ultrasound imaging, static image-based assessments are often prone to errors and heavily reliant on expert interpretation. With the application of these advanced models, not only can the analysis be automated and improved, but the way data are stored, used, and processed can become seamless. Furthermore, this capability can be extended to remote and underserved geographical regions through edge devices connected to the cloud, enabling high-quality diagnostics without the need for on-site specialists. Similarly, in satellite imaging, where data typically consists of thousands of images without accompanying audio, current methods focus on object detection in individual frames. By treating this data as a continuous video stream, these models could enable contextual path mapping and more comprehensive environmental analysis. The ability to process long-form video content at the edge opens up transformative possibilities, particularly in healthcare diagnostics and drone-based applications.
[0434] Examples areas of impact related to Al-driven video processing for medical diagnosis. The present inventors identified many such areas including: (1) Radiology and Imaging: MRI / CT Scan Video Processing: Advanced Al can analyze full-length MRI or CT scans in motion (e.g., cardiac MRI or functional MRI), detecting anomalies faster than manual review; Ultrasound Interpretation: Al-powered real-time video analysis can help with fetal health assessments, echocardiography, and liver disease detection; (2) Endoscopy and surgery: Colonoscopy Polyp Detection: Al can process hours of colonoscopy footage to detect polyps in real-time, improving colorectal cancer screening; Robotic Surgery Assistance: Al-driven video processing can provide real-time insights to surgeons, flagging anomalies or suggesting procedural adjustments; (3) Ophthalmology and dermatology: Retinal Scan Analysis: Al models can process retinal scan videos to detect early diabetic retinopathy or macular degeneration; Skin Cancer Detection: Dermatologists can use Al-enhanced dermoscopy video processing to detect melanoma more accurately.; (4) pathology and microscopy: Al can analyze continuous microscopy footage to identify cancerous cells, bacterial infections, or rare hematological conditions in blood samples faster than human pathologists; (5) remote patient monitoring: Wearable devices that record and process patient videos (e.g., heart rate monitors, movement trackers) can enable early diagnosis of conditions like heart arrhythmias or sleep apnea at lower costs.
[0435] Example areas of impact for Al-driven video processing for drone / satellite data analysis. The present inventors identified many such areas including: (1) Agriculture and environment: Crop Health Monitoring: Al-driven video analysis can quickly identify stressed crops, pest infestations, or nutrient deficiencies; Deforestation and Land Use: Detecting illegal logging or monitoring ecosystem changes becomes faster and cheaper; Disaster Assessment: Rapid damage assessment after hurricanes, earthquakes, or flood helps authorities respond effectively; (2) Infrastructure and urban planning: Road and Bridge Inspections: Al can process high-resolution drone footage to detect cracks, erosion, or weak points; Traffic and UrbanPlanning: Satellite video can track congestion patterns and optimize urban development; (3) healthcare and epidemiology: Mosquito-Borne Disease Prevention: Satellite video can help detect standing water bodies where mosquitoes breed, aiding in malaria / dengue prevention; Air Pollution and Public Health: Fast video analysis can track pollution hotspots, correlating air quality data with disease outbreaks.
[0436]
[0437] E. Search
[0438] Figure 9 depicts an example of a system and method for processing search queries to provide relevant search results to users. The system comprises multiple modules that enhance, search, score, rank, and rerank documents based on the user's query. Briefly, the methods described above enable the implementation of a natural language search process across the entire datasets described above which begins with (a) query initiation (user submits a query through an interface); (b) query regeneration (also referred to herein as search query enhancement) - this consists of two substeps: glossary addition (which takes care of abbreviations and synonyms, for example expanding abbreviations into corresponding full expressions, and providing multiple synonyms known for an expression used), and autofilter creation (which finds the filters mentioned in the query, e.g. query after refinement: tecentriq documents for USA, this will be converted into, tecentriq documents for USA, filter; country=United States); (c) query embedding (the regenerated queries undergo vectorization, transforming them into high-dimensional embeddings, which enables semantic matching in the vector space) - this can be done using 'sentence-transformers / all- MiniLM-L6-v2' embedding model which is a lightweight and good model for embedding generation; (d) Vector search: the embedded query is utilized to perform parallel searches across distinct vector collections: image Collection: For visual content matching queried with the caption of the images; and Text Collection: For unstructured text document search; (e) Parallel Keyword Processing: Concurrently with steps (b)-(d), the system extracts relevant filters and keywords from the original query; (f) Keyword-Based Search: Using the extracted filters and keywords, a traditional keyword search is executed against the Document database, which can contain e.g. metadata and textual summaries of various documents (a text search on the document database and some of the text indices created, which extracts all the documents which are lexically similar to the search query based on the veeva_name and veeva_description. As explained further herein, for every document a summary (e.g. well representative structured summary) is created as part of meta information creation to facilitate document communities creation and thereby maximising recall. There is a 100% match search on the file name and veeva_document_number).
[0439] Next, (g) Dual Search Results. The system now has two sets of search results: (i) Vector Search Result: Semantically matched content from the multi-modal collections; (ii) Keyword Search Result: Directly matched documents based on traditional information retrieval techniques; Specifically, the search process as described herein uses a document search which is keyword based, and an information search which is a semantic search. The document search identifies ant document that includes keywords in their metadata, and can give priority to documents that contain all of the keywords in their metadata, over documents identified in a semantic search. The former may be referred to as class 1 results and the latter may be referred to as class 2 results. Keyword searches may be used for finding documents with exact matches on a field, without any analysis or processing of the text / content of the document. This means the search term is compared directly against the indexed term, ensuring precise matches. Traditional keyword search often relies on finding exact matches for the terms entered. This is very rigid and fails in many commonscenarios, such as when typos are present in the search terms, different conjugated forms of verbs are used, synonyms or different word orders are used, etc. Thus, Jaccard Index and / or Fuzzy matching may also be used to identify matches to keywords. The use of the Jaccard Index focusses on keyword overlap. The Jaccard Index (or Jaccard Similarity Coefficient) measures the similarity between two sets. It is calculated as the size of the intersection of the sets divided by the size of the union of the sets. The Jaccard Index ignores the word order. For example, "cheap canon ink" and "canon ink cheap" produce the same set and thus match perfectly with each other using the Jaccard Index. Further, the Jaccard Index handles partial matches naturally, since it measures the proportion of shared keywords. A document containing all query keywords plus others will score highly, but less than 1.0, reflecting the presence of extra terms. Additionally, the Jaccard Index can quantify relevance, since documents with a higher Jaccard score relative to the query set share more keywords proportionally, suggesting higher relevance based on keyword overlap. Fuzzy Matching focusses on approximate string similarity. Fuzzy matching encompasses techniques designed to find strings that match a pattern approximately rather than exactly. It often works at the character level. A very common metric used is the Levenshtein distance. The Levenshtein distance is the minimum number of single-character edits (insertions, deletions, or substitutions) needed to change one word into the other. Fuzzy matching can be used to enable typo tolerance, and can directly addresses misspellings and typos in either the query or the source text. For example, "Jacard" can match "Jaccard". Fuzzy matching can handle slight differences like pluralization ("cartridge" vs "cartridges" - distance 1) or minor variations in product codes. Fuzzy matching can enable increased recall by finding relevant documents that would be missed by exact matching due to small errors. The use of both Jaccard Index and Fuzzy matching may be complementary. For example, fuzzy matching can be used first to identify potential term matches, even if misspelled. The set of matched terms (identified via exact or fuzzy matching) from the query and the document can then be used to calculate a Jaccard Index (or a more sophisticated ranking score like TF-IDF or BM25) to assess the overall relevance of the document based on keyword overlap. Preprocessing steps like stemming / lemmatization can also be applied before either technique to handle different word forms. By using Jaccard Index and Fuzzy Matching, keyword search becomes more robust, user-friendly, and capable of retrieving relevant information even when queries aren't perfectly formed. (h) Advanced Ranking Algorithm: a ranking algorithm integrates and evaluates both the vector-based and keyword-based search results. For example, a semantic weighting can be used which assigned weights based on experiments. For example, keyword search results may be weighted higher than semantic search results based on predefined weights. In some embodiments, the similarity scores calculated for each search result found within each collection (e.g., text collection, image collection, table collection, video collection, etc.) are weighted. In some implementations, weights may be assigned based on experiments conducted over a large number of queries (250+). This means that through careful testing and analysis, the system may have learned which types of results (text, images, etc.) tend to be more relevant or important for different kinds of search queries. For example, the experiments might have shown that for certain types of queries, the text content is generally more crucial than the filename. In such a case, the "Text" modality would be assigned a higher weight (e.g., 0.7) than the "FileName" modality (e.g., 0.3). In any case, the sum of the weights for all modalities would be equal to 1 . In some embodiments, the final relevance score for a search result may not just be a simple average of its similarity scores across different collections. Instead, after calculating how similar that result is to the query within each relevant collection (text, image, etc.),each of these similarity scores is then multiplied by a pre-determined weight that reflects the importance of that particular collection type (or modality). Finally, these weighted similarity scores are added together to produce the overall semantic relevance score. For example, for the search query "best apple pie recipe", a search is carried out over three collections: text, image, file name. A search result responsive to this query is a document of a recipe article with an image of an apple pie. The text of the article, from the text collection, has a similarity score of 0.9; the image of the apple pie, from the image collection, has a similarity score of 0.7; and the document’s name, from the file name collection, has a similarity score of 0.5. In this example, previous experiments may have determined the following weights: text modality weighted at 0.6; image modality weighted at 0.3; file name modality weighted at 0.1. The weighted average, which represents the final semantic relevance score for this search result, may then be calculated as: (Similarity Score for Text * Weight for Text) + (Similarity Score for Image * Weight for Image) + (Similarity Score for Filename * Weight for Filename) = (0.9 * 0.6) + (0.7 * 0.3) + (0.5 * 0.1) = 0.54 + 0.21 + 0.05 = 0.80. The search result may therefore be assigned a relevance score of 0.80 and then be ranked based at least in part on this score. This score takes into account how well the search result matched the search query across different modalities (i.e. text, image, filename) and also gives more importance (higher weight) to the text of the recipe compared to the image or just the filename. This weighted approach allows the search engine to provide more accurate and relevant results by prioritizing the information sources that are most likely to be helpful, for example, based on experience or user feedback.
[0440] (i) Result Presentation: the final ranked results are curated (based on intent, ranking order and maximum relevance for a query based on recency, asset type, language, country and objective.) and presented to the user, offering a comprehensive view that leverages both semantic understanding and explicit keyword matching.
[0441] This process is illustrated on Figure 9. When a user submits a search query, the following method is carried out to process the query and provide search results to the user:
[0442] 1. Search Query Enhancement
[0443] • Glossary Addition: This sub-module addresses abbreviations within the search query by expanding them to their full forms.
[0444] • Autofilter Creation: This sub-module identifies and applies relevant filters to the search query. For example, the query "Tecentriq documents for USA" is enhanced to include the filter "country=United States".
[0445] • Automatic query classification: This sub-module classifies generated queries into one of two distinct categories:
[0446] i. Keyword-specific queries, typically aimed at retrieving documents based on exact matches.
[0447] ii. Semantic-specific queries, intended for extracting information based on contextual understanding.
[0448] 2. Text Search on DocumentDBThe system performs a text search on DocumentDB using predefined text indices. Documents lexically similar to the search query are retrieved based on attributes such as veeva_name and veeva_description. Additionally, a 100% match search is conducted on the file name and veeva_document_number.
[0449] 3. Scoring and Ranking
[0450] Retrieved documents are scored using an algorithm that combines custom Jaccard similarity and fuzzy similarity metrics. Documents are then ranked according to their scores.
[0451] 4. Semantic Search
[0452] Query embeddings are generated using the 'sentence-transformers / all-MiniLM-L6-v2' model. The query is searched across multiple asset collections within the vector database, including text assets, tables, image captions, and document summaries. The semantic search results are aggregated using a weighted average method.
[0453] 5. Re-ranking
[0454] In the final step, documents from both text search and semantic search are merged and re-ranked. A weighting (e.g., 0.6) may be applied to search results from keyword-specific queries, and a complementary weighting (e.g., 0.4) may be applied to search results from semantic-specific queries. The re-ranked search results may be further curated before they are presented to the user along with their respective similarity scores.
[0455] In one example where a user searches for “Herceptin MOA email”, and the terms “Herceptin”, “MOA” (Mechanism of Action), and “email” are present in the metadata of certain documents, the system may prioritize those documents in the re-ranking of search results. This keyword-based prioritization of search results may be used to ensure that documents with explicit keyword matches are surfaced first.
[0456] Query classification
[0457] As mentioned, the system may also classify generated queries into two distinct categories:
[0458] Class 1 : Keyword-specific queries, typically aimed at retrieving documents based on exact matches. Class 2: Semantic-specific queries, intended for extracting information based on contextual understanding. From a pool of multiple generated queries, the system may then filter and select a number of top candidates (e.g. the top 2 or top 3) by evaluating their classification and alignment with the user’s intent, i.e. whether it is document retrieval or information extraction.
[0459] Keyword-based search techniques
[0460] Conventional keyword search techniques rely on exact matches between the search term and indexed content. These methods do not involve any linguistic analysis or text processing, which means the system compares the input query directly against stored terms. While this ensures precision, it also introduces significant rigidity, often failing in common real-world scenarios such as:
[0461] • Typos and Misspellings: A query like “Canonm” will not match “Canon”.
[0462] Word Variations: “Running” will not match “run”.Synonyms: “Cheap” will not retrieve documents containing “inexpensive” or “affordable”.
[0463] • Word Order Sensitivity: “Ink cartridge Canon” may not match “Canon ink cartridge”.
[0464] • Partial Matches: A search for “Canon ink” might miss documents that mention “cheap Canon ink cartridges”.
[0465] To address these limitations, advanced techniques such as Jaccard Index and Fuzzy Matching are employed.
[0466] The Jaccard Index, also known as the Jaccard Similarity Coefficient, quantifies the similarity between two sets by dividing the size of their intersection by the size of their union. When applied to keyword sets from queries and documents, it offers several advantages:
[0467] • Order Invariance: The index treats “cheap Canon ink” and “Canon ink cheap” as identical sets, ensuring consistent matching regardless of word order.
[0468] • Partial Match Handling: Documents that contain most — but not all — query terms still receive a high similarity score, reflecting their relevance.
[0469] • Relevance Scoring: The Jaccard score provides a proportional measure of keyword overlap, helping rank documents based on how closely they match the query.
[0470] Fuzzy matching techniques are designed to identify approximate matches between strings, often at the character level. A widely used metric in this domain is the Levenshtein Distance, which calculates the minimum number of single-character edits (insertions, deletions, or substitutions) required to transform one string into another. Key benefits include:
[0471] • Typo Tolerance: Misspelled queries like “Jacard” can still match “Jaccard”.
[0472] • Handling Minor Variations: Differences such as “cartridge” vs “cartridges” (edit distance of 1) are effectively managed.
[0473] • Improved Recall: Fuzzy matching retrieves relevant documents that would otherwise be missed due to small textual discrepancies.
[0474] In our system, fuzzy matching and Jaccard similarity are used in tandem to enhance the robustness and flexibility of keyword search. For example, the following approach may be adopted:
[0475] 1. Fuzzy matching is applied to identify potential term matches, even in the presence of typos or slight variations.
[0476] 2. The resulting matched terms, whether exact or approximate, are used to construct keyword sets for both the query and candidate documents.
[0477] 3. The Jaccard Index (or optionally, more advanced ranking metrics like TF-IDF or BM25) is then computed to assess the overall relevance based on keyword overlap.
[0478] 4. Preprocessing techniques such as stemming or lemmatization are applied to normalize word forms before matching, further improving accuracy.By integrating these techniques, the system transforms rigid keyword search into a more intelligent, user-friendly engine capable of retrieving relevant information even when queries are imperfectly formed. Semantic Weighting for Multimodal Relevance Scoring
[0479] Semantic weighting refers to the process of assigning differentiated importance to various content modalities (e.g., text, images, tables, summaries, filenames, videos, audio, major segment tags, document recency, etc.) when evaluating the relevance of a document or asset. Each modality contributes uniquely to the overall semantic understanding of a document, and thus, must be weighted accordingly.
[0480] To determine these weights, the inventors conducted experiments using a dataset of over 250 queries. For each query, similarity scores were calculated across the different modalities and tags. These individual scores were then multiplied by their respective weights, empirically derived from the experimental results, and aggregated to produce a final relevance score. The sum of all weights is normalized to 1 , ensuring a balanced and interpretable scoring framework.
[0481] This approach serves as a foundational or "cold start" mechanism for a recommendation engine, enabling it to make informed decisions even in the absence of user interaction data. As the system matures, user feedback, such as binary relevance signals (e.g., yes / no responses), may be incorporated into the scoring model. This will allow the engine to dynamically adjust weights based on real-world usage patterns, further refining its ability to deliver contextually relevant recommendations.
[0482] Results Curation
[0483] The curation of search results before presentation may be based on one or more of the following criteria: the inferred user intent, the ranking order determined by the ranking algorithm, and maximum relevance for a query based on recency, asset type, language, country and / or objective.
[0484] F. Conclusion
[0485] The work above introduces a novel industrial VLM framework for large-scale video processing tailored to the healthcare industry. It enables researchers to rigorously assess long video content reasoning. Through their extensive evaluation, the inventors generated the following key findings: (1) while current research predominantly focuses on static image inference, incorporating ALMs, voiceover transcriptions, and metadata improves video reasoning; (2) selecting attention mechanisms based on the GPU architecture can yield better efficiency, challenging the assumption that the latest state of the art methods always provide the best results; (3) VLMs perform well in video summarization but continue to struggle with key frame detection and time-sensitive reasoning; (4) while splitting long videos into shorter segments appears to be an intuitive solution, it was found that cutting and compressing videos often consumes even more processing time and increases errors in key frame alignment and summary.
[0486] Beyond these insights, the inventors contribute by extending the Video-MME dataset to subtasks for video summarization and key frame extraction, incorporating an enhanced evaluation. With the above framework, users can perform natural language search, extract key information via chat-based queries, generate structured outlines, and efficiently skim documents. Further, built-in translation enables seamless retrieval across languages.2
[0487] The present example describes an embodiment of a search engine of the disclosure (referred to in the present example as “Content Finder”), such as e.g. as described in section E of Example 1 .
[0488] The Content Finder application provides users with an intuitive search experience for exploring information. Users can start by selecting a relevant filter to narrow down their topic of interest. They can then craft their search query by typing in their question or keywords. The application retrieves a diverse range of outputs, including pertinent images, informative tables, and comprehensive textual explanations, making it easy for users to find the information they need.
[0489] System overv / ew
[0490] Content Finder is an intelligent medical search application engineered for high-precision retrieval of specialized information. The system architecture is designed to retrieve relevant multi-modal assets, including PDFs, images, videos, and audio. Beyond basic retrieval, the application implements advanced features such as Al-driven summarization, automated filter generation, document previews, and interactive Al chatbots, ensuring seamless discovery across the global content library.
[0491] The example system is configured to process material and reference type documents. Once the search retrieves results based on sophisticated scoring algorithms (see below), the user interface (Ul) presents them in the following categorized formats: (i) Materials (Search Type): processes material documents that have expired, Approved For Use, or Approved For Distribution status; and (ii) References (Search Type): a specialized search mode executing the same query processing, optimization, and search mechanisms as Materials, but restricted to reference documents with an Approved For Use status. This can be achieved by automatically applying a Veeva Document Type filter (Veeva Document Type = References). It returns results across all content formats in a single Unified Result Set, meaning all reference documents are displayed together regardless of format (although separation in a similar way as for the Materials results is also possible).
[0492] The Materials results can be divided into intuitive tabs, such as, in the present example:
[0493] Materials :: Overview: This is a primary results dashboard presenting a unified, consolidated view of all content types retrieved from the search (e.g. PDFs, Word documents, PowerPoint presentations, Excel spreadsheets, standalone images, audio files, and video files). Users can quickly scan through top matches across all formats without switching tabs.
[0494] Materials :: Documents: This is a filtered view of the Overview tab, pulling from the same comprehensive search results but displaying only text-based document formats (PDFs, Word, PPT, Excel, etc.) while maintaining the same similarity score ranking.
[0495] Materials :: Image: This is a filtered view showing only visual content from the complete result set. This includes standalone image files from the overview tab as well as images directly extracted from PDFs, presentations, and Word documents.
[0496] Materials :: Audio, Video etc: This is a filtered view displaying only multimedia (audio and video) formats, maintaining the same underlying ranking logic.The search lifecycle is initiated by the Query Optimizer, which outputs refined queries and auto-generated metadata filters. These outputs are ingested by the Search Module, the central engine for cross-domain retrieval. The architecture allows users to bypass the Query Optimizer to execute raw query searches. In this scenario, the raw query is dispatched directly to the Search Module, adhering to the standard execution pipeline from that entry point.
[0497] The Search Module performs comprehensive indexing across both asset-level metadata and extracted components: (i) Asset Metadata: Filenames and file descriptions; (ii) Textual Components: Full-text content extracted from documents; (iii) Visual / Structured Components: Image captions and table captions; (iv) Temporal Components: Audio and video transcripts / segments; and (v) Synthesized Data: Al-generated document summaries.
[0498] When a search query is entered, for text search, the query is embedded after which both semantic search and keyword search is performed on the existing text chunk embeddings collection, filename and title collection in the vector database. The text chunks with the highest similarity scores are returned along with their respective payloads, which include the docjd, assetjd, language, tags of the respective document, the text chunk itself, and the similarity score. For image and table search, the query is embedded and a semantic search is performed on the existing caption embeddings of the respective images (captions extracted during the captioning stage) in the image and table collection in the vector database. The query and captions with the highest similarity scores are returned along with their respective payloads, which include the docjd, assetjd, language, tags of the respective document, and the similarity score. Semantic search results come from relevant documents, which may include images, tables, and text. These results are provided along with a summary and a score.
[0499] Search methodologies
[0500] Content Finder can employ any of three distinct retrieval strategies tailored to specific data structures: (i) Keyword Search: Utilizes lexical matching for rapid, exact, or near-match retrieval of identifiers; (ii) Semantic Search: Employs vector embeddings to capture conceptual meaning and context in visual and structured data; (iii) Hybrid Search: A multi-vector approach combining lexical and semantic scores to optimize full-text retrieval.
[0501] The primary goal of the vector search system is to deliver highly relevant and contextually accurate results. The vector search implementation can be performed using Qdrant for multi-modal retrieval across different content types. The system can manage specialized vector databases (e.g. Qdrant collections) configured for specific search modalities. In a specific example, the system employs three distinct search strategies optimized for different data types and use cases. Examples are provided in Table 10.
[0502] Content Type Search Method Use Case
[0503] Veeva Name BM42 (Sparse) Exact / fuzzy name matching
[0504] Veeva Title BM42 (Sparse) Exact / fuzzy title matching
[0505] Image Captions Dense Vectors Visual content search
[0506] Table Captions Dense Vectors Structured data search
[0507] Document Summary Dense Vectors Summary semantic search
[0508]
[0509] Full Text Content BM42 + Dense Full-text semantic search
[0510] Table 10. Example vector database collections, search methods and use cases.The architecture can leverage two different optimized models for vector generation, including a model for generating BM42 embeddings or other sparse vectors, and a model for generating dense embeddings. For example, FastEmbed (BM42) can be used for sparse vector generation. Dimensions can be mapped to token IDs via the FastEmbed tokenizer, with values representing modified TF-IDF weights. The system executes searches using the modifier=idf parameter for enhanced lexical precision. For dense vector generation (e.g. 1024-dimensions), mixbread-ai / mxbai-embed-large-v1 can be used, which is a sentencetransformer optimized for retrieval tasks with robust multilingual capabilities. TF-IDF weights are a vectorized text representation mechanism in which each dimension of an embedding vector corresponds to a specific token generated by the FastEmbed tokenizer, and the numerical value assigned to that dimension represents a weighted importance score derived from a TF-IDF computation. In the present example a modified TF-IDF computation is used. In traditional TF-IDF (Term Frequency-Inverse Document Frequency), the importance of a token is determined based on: TFIDF(t,d)=TF(t,d)*IDF(t) where TF(t,d) measures how frequently token t appears in document d; and IDF© measures how rare or discriminative the token is across the corpus. Here the inventors used a modified TF-IDF formulation using BM42, This advantageously integrates transformer tokenization, contextual term weighting, adaptive inverse document frequency scaling, and sparse semantic embedding generation. Instead of representing documents using purely lexical frequency statistics, BM42 produces semantically enriched sparse vectors in which each dimension corresponds to a tokenizer-generated token ID and each value represents a modified TF-IDF relevance weight calibrated using corpus-aware and context-sensitive weighting mechanisms. This modified formulation is defined by: TFmodified=1+log(ft,d) and IDFmodified(t)=log(1+nt+0.5N-nt+0.5). This improves semantic retrieval precision, token discriminability, and robustness across noisy, heterogeneous, and domain-specific enterprise datasets, equation
[0511] The keyword search aims to perform fast, exact or near by matching of name and title. The semantic search aims to understand the meaning and context in visual and structured contents of documents searched. The hybrid search aims to achieve both aims, for full-text content.
[0512] Execution process
[0513] The Search Module of the present embodiment executes a deterministic five-step pipeline: filter consolidation, glossary-based query expansion, embedding generation, vector search execution, and score normalization and merging. These are explained in more detail below.
[0514] Filter Consolidation: The search process begins by taking both user-defined filters and auto-generated filters and merging them into a single, unified filter base. These filters leverage metadata fields, which help refine the search results based on specific metadata attributes. Once consolidated, the system translates these filters into the exact query formats required by the underlying databases, e.g. MongoDB and Qdrant. Glossary-Based Query Expansion: To ensure no relevant information is missed, the system checks the user's query against a predefined glossary to identify any abbreviations. If abbreviations are found, the system automatically creates multiple query variants to account for every possible meaning of that abbreviation. This expansion ensures comprehensive coverage, meaning the search won't fail just because a user searched for an acronym instead of the full term.Embedding Generation: Once the query variants are established, the system processes them through a dual-embedding strategy to capture both exact terminology and underlying meaning. Using sparse embedding, the query is tokenized and converted into a sparse vector using the BM42 lexical model, which specializes in exact keyword matching. For dense embedding, the query is (e.g. simultaneously) converted into a 1024-dimensional dense vector using the mixbread-ai / mxbai-embed-large-v1 semantic model, which captures the conceptual meaning of the query
[0515] Vector Search Execution: With both sets of embeddings generated, the system performs parallel vector searches across multiple content types — including text, images, tables, audio, video, filenames, and file descriptions. The Sparse Search specifically targets the Name and Title collections to output raw keyword relevance scores. The Dense Search specifically targets the Image Caption, Table Caption, and Summary collections to output raw semantic similarity scores (using Cosine Similarity). The Hybrid Search specifically targets the text content in Hybrid Search collections, which uses both keyword relevance score and cosine similarity, which are merged using a Reciprocal Rank Fusion (RRF) algorithm.
[0516] Score Normalization & Merging: Aggregates disparate scores into a single ranked list of the most relevant documents.
[0517] Keywords filed (e.g. Name, Title) are routed directly to sparse embedding to populate the Name and Title collections. Semantic fields (e.g. Captions, Summaries) are routed through Dense Embedding for conceptual mapping. Hybrid Fields (e.g. Document Text) undergo both Sparse and Dense embedding. The text is cleaned and chunked before being processed by both models to feed the Hybrid Text Collection, where results are merged via Reciprocal Rank Fusion (RRF). Chunking transforms large documents into semantically coherent segments, using a splitter configured to preserve semantic coherence by respecting natural document boundaries and structures, maintain context by ensuring informational continuity through configurable chunk overlaps; prevent information fragmentation by avoiding mid-sentence splitting to maintain local context; and perform automated ingestion, handling various document formats automatically. An overview generation process can also be applied, where relevant text data is retrieved from the vector database. This data is then filtered to identify the most pertinent pieces of information. These top relevant texts are passed to a Language Learning Model (LLM), which processes and synthesizes the content to generate a comprehensive summary. The final output is a concise summary, typically ranging between 200 to 300 words, capturing the key points and insights from the original texts.
[0518]
[0519] Keyword scoring: Finder retrieves relevant documents from the database, scored by the BM42 model on a 0-5 relevance scale. BM42 scores are normalized to a 0-100 scale to ensure consistent weighting across queries. The normalization formula is: normalized_bm42_score = - jf??n42 sco e
[0520] max _bm =4 -2_Score * 100 where bm42 score is the raw score produced by the BM42 algorithm for a given document, and max_bm42_score is the highest BM42 score among all retrieved documents for the same query. This normalization ensures the top-scoring document receives a score of 100, and all other documents are scaled proportionally. A weighted scoring formula is applied to combine the normalized BM42 score with a secondary similarity metric fuzzy matching. The final score is computed as: finalscore= (0.7 *
[0521]
[0522] normalize dbm42scor^ + (0.3 *similarity _s cor e) where similarity score is a score calculated using fuzzy logic which indicates the lexical similarity between the query and the retrieved document name / title.
[0523] The similarity score may be derived using one or more fuzzy matching algorithms. In this example implementation, the inventors used a Jaccard Similarity algorithm. More than one algorithm can be used. Each algorithm produces a partial similarity estimate. These estimates may then be combined into a unified fuzzy confidence score. For example, any one or more of the following methods may be used Token overlap, Edit-distance similarity, Phrase alignment, Synonym mapping. The resulting scores may then be aggregated using a weighted sum with predetermined weights. The scores may be between 0 and 1. The weights may sum to 1. The fuzzy logic system can therefore produces a continuous confidence value between 0 and 1 instead of a binary match / no-match result. For example, given an example user query is: "breast cancer treatment" and the retrieved document title is: "breast carcinoma therapeutic methods", although the wording differs, the phrases are conceptually and lexically related. Therefore, the fuzzy matching should give it a middle to high score (e.g. Token overlap: 0.80, Edit-distance similarity: 0.72, Phrase alignment: 0.83, Synonym mapping: 0.77, leading to SimilarityScore=0.78 after weighted fuzzy aggregation).
[0524] Semantic scoring: semantic scoring is performed based on individual entity semantic score multiplied with predetermined weights by Entity type (e.g. 40% for text, 30% for summary, 15% for image, 15% for table) , , , . , . , , XlL, semantic entity score *entity weiqht and normalize by}sum of weights used accordin9g to the formula: - -
[0525]
[0526] ent -i -ty wei -g -hti - —2—L. An example of the scoring process described above is illustrated on Fig. 12.
[0527]
[0528] The tagging pipeline is structured around three primary components: text, tables, and images. These components form the foundation of the tagging process, ensuring that all relevant information within the documents is systematically processed and analyzed. The tagging process is divided into three distinct stages: extraction, captioning, and tagging. Each stage plays a role in transforming raw document data into meaningful tags.
[0529] During the extraction stage, the system reads the document page by page and pulls out all important elements such as text, tables, images, and metadata. Tools like Docling and PyMuPDF work together to accurately identify and isolate these components. This stage forms the foundation for all later pipelines processing.
[0530] In the captioning stage, the images and tables extracted from the document are analyzed by Al models to generate clear and meaningful captions. These captions describe the visual content in simple text form, allowing the system to understand and process visuals effectively. This step is essential because it converts non-text elements into searchable and interpretable information.
[0531] During the translation stage, the system checks whether the extracted content is in English. If not, the text is translated into English using advanced language models. Both the original and translated versions are preserved. This ensures consistency across documents and enables accurate tagging and vectorization regardless of the document’s original language.The tagging stage involves generating two types of tags for each document. Extractive tags are created from specific keywords found directly in the text, while abstractive tags are generated by Al models based on the overall meaning and context of the document. Together, these tags provide both factual and conceptual insights, enabling better classification and understanding of the content.
[0532] In the vectorization stage, the extracted text, captions, filenames, and summaries are converted into vector embeddings. These vectors capture the semantic meaning of the content, allowing documents to be searched and matched based on similarity rather than exact words. This stage powers advanced applications like semantic search (as described above).
[0533]
[0534] The Query Optimizer module uses an LLM (e.g. Amazon Bedrock Claude 4 Sonnet) to transform raw user queries into a standardized, machine-ready format that preserves user intent. When a user submits a query, the system processes it through the optimizer, which interprets the intent (e.g. performing any spelling corrections to correctly interpret the user intent; e.g. with the raw user input of “Give me Okrevs materials”, the model may interpret that “Okrevs” refers to the drug product “Ocrevus”), enriches the query with domainrelevant context, and generates all applicable filters. These filters are automatically derived based on the content of the query such as scientific area, product, indication, content type, or other metadata attributes and are packaged together with the optimized query into a structured payload.
[0535] The optimizer prompt is designed to ensure that all generated filters strictly adhere to a controlled schema. As part of this process, the system loads a file (e.g. .json file) that defines every valid filter field along with its allowed values. When a user query is optimized through the LLM, the model not only restructures the query into a standardized format but also generates the relevant filters exclusively from the entries present in the file defining the valid filters. If the model returns any filter fields or values that fall outside this predefined set, the system performs an additional validation step to remove invalid or noisy filters before creating the final payload. This guarantees that the optimized query and its associated filters remain clean, consistent, and fully compliant with the approved filter schema.
[0536] The purpose of the query optimizer is to convert raw user queries (including misspellings, unstructured intent, and mixed-language inputs) into an optimized search query plus structured filters that improve recall and precision. It performs the following tasks: (i) content ingestion, (ii) spelling correction, (iii) filter values identification based on user query intent, (iv) Retain Technical, Branded and Scientific Terms in the query, (v) check country language dependencies, (vi) check that “Product” and “Indication” are maintained in the search query, (vii) synonym handling, (viii) enforce excluded filters, (ix) LLM generated filter validation with Auto Filters, (x) synonym expansion for filters, (xi) tracing and logging.
[0537] Content Ingestion: Content Injection is the initial step in the Query Optimizer workflow. At this stage, the raw user query and the structured auto-filter JSON are combined and passed to the Large Language Model (LLM). This ensures that the LLM has both the unstructured user intent and the structured filter context available for query optimization. The User Query represents the direct input provided by the user during search (for example: “Give me Ocrevus materials”). This query is injected into the LLM prompt without modification to preserve the original intent. The Auto-Filter JSON is a structured JSON object containing key-value pairs for filters such as product, indication, scientific area, content type, language, and country.Each filter includes canonical values and synonyms to support accurate extraction and normalization. An example structure for an auto-filter json is shown below:
[0538] {
[0539] "product" : {
[0540] "ocrevus" : ["Ocrevus" ] ,
[0541] "tecentriq" : [ "Tecentriq" ]
[0542] } ,
[0543] "indication" : {
[0544] "nsclc" : ["Non Small Cell Lung Cancer" , "NSCLC" ]
[0545] } ,
[0546] "scientific_area" : {
[0547] "oncology" : ["Oncology" ]
[0548] }
[0549] }
[0550] Spelling Correction: After the user query has been injected into the LLM prompt along with the auto-filter JSON, the system automatically identifies and corrects misspelled words. This ensures that the optimized query maintains accuracy and improves the likelihood of retrieving relevant results. For example, the system may correct a user Query: “Give me Okrevs materials” to a corrected query: “Give me Ocrevus materials”.
[0551] Filter Values Identification Based on User Query Intent: At this stage, the system analyzes the user query intent and extracts relevant filter values such as product, indication, scientific area, language, and country. This ensures that the search process is guided by structured filters aligned with the user’s request. For example, for the query: “Give me Ocrevus materials, the system may output the following Identified Filters json:
[0552] {
[0553] "product" : ["Ocrevus" ] ,
[0554] "indication" : [ ] ,
[0555] "scientific_area" : [ ] ,
[0556] "language" : [ ] ,
[0557] "country" : [ ]
[0558] }
[0559] Retain Technical, Branded and Scientific Terms in the query: This step ensures that domain-specific terminology, product names, and scientific expressions remain intact during query normalization and optimization. Preserving these terms prevents loss of meaning and maintains the accuracy of search results. For example, in a user Query: “Give me Ocrevus NSCLC trial data”, the system may ensure that the terms Ocrevus (Product), and NSCLC (Indication) are preserved.
[0560] Country-Language Dependencies: At this step, the system applies a rule that ensures that language references in user queries are correctly interpreted without automatically assigning a corresponding country. The distinction prevents misclassification when a query specifies a language but does not explicitly mention a geographic location. For example, for a user query: “Italian documents”, the filters identified may be
[0561] {
[0562] "language" : ["Italian" ] ,
[0563] "country" : [ ]
[0564] }
[0565] whereas for a user query: “Documents from Italy”, the filters Identified may be:
[0566] {
[0567] "language" : [ ] ,"country" : ["Italy" ]
[0568] Do not remove 'Product' and 'Indication' from the Search Query: At this step, the system applies a rule that ensures retention of Product and Indication terms (or any other values that represent critical domainspecific entities that directly influence the accuracy and relevance of search results). The example implementation can ensure that they remain present in both the optimized query string and the extracted filter set, even if they are already mapped to structured filters. For example, for the user query “Give me Ocrevus NSCLC materials”, an optimised query may be “Give me Ocrevus NSCLC materials” and filters identified may include:
[0569] {
[0570] "product" : ["Ocrevus" ] ,
[0571] "indication" : ["NSCLC" ]
[0572] }
[0573] Synonyms Handling for Filter from User Query: Synonyms Handling ensures that variations, abbreviations, and alternate forms of filter values in user queries are correctly mapped to their canonical representations. This step improves consistency and accuracy in search results by standardizing filter values while retaining the original user input for transparency. For example, for a user query: “UK oncology documents”, the following filters may be identified:
[0574] {
[0575] "country" : ["United Kingdom" ] ,
[0576] "scientific_area" : [ "Oncology" ]
[0577] }
[0578] LLM Excluded Filter: Excluded Filters represent fields that are set to remain empty when the LLM generates filter values. These exclusions are enforced to prevent unintended or irrelevant filter assignments that could compromise search accuracy or conflict with system-controlled parameters. In an example implementation there are two categories of excluded filters: general filters (e.g. content complexity, document type, source file availability, non-clinical topic, clinical topic), and date filters (e.g. creation date (from / to), approved for distribution date (from / to), expiration date (from / to)). For example for the user query: “Give me Ocrevus NSCLC materials created in 2020”, the LLM suggested filters may be:
[0579] {
[0580] "product" : ["Ocrevus" ] ,
[0581] "indication" : ["NSCLC" ] ,
[0582] "creation_date_f rom" : "2020-01-01" ,
[0583] "creation_date_to" : "2020-12-31"
[0584] }
[0585] and the final auto-filters (after exclusion enforcement) may be:
[0586] {
[0587] "product" : ["Ocrevus" ] ,
[0588] "indication" : ["NSCLC" ] ,
[0589] "creation_date_f rom" :
[0590] "creation_date_to" :
[0591] "approved_f or_distribution_date_f rom" : " " , "approved_for_distribution_date_to" : " " ,
[0592] "expiration_date_f rom" :
[0593] "expiration_date_to" :
[0594] "content_complexity" :
[0595] "document_type" :
[0596] "source_file_available" :
[0597] "non_clinical_topic" :"clinical_topic" : ""
[0598] }
[0599] LLM generated filter validation with Auto Filters: When the LLM generates filter values, they are validated against the predefined Auto Filter JSON. If the LLM suggests a value that is not present in the auto-filter configuration (e.g., a product or indication not recognized), that value is excluded during validation. For example, for a user query: “Show me oncology materials in csv ”, the LLM Generated Filters (before validation) may be:
[0600] {
[0601] "product" : ["Ocrevus" ] ,
[0602] "scientific_area" : ["Oncology" ] ,
[0603] "content_type" : [ ' csv ' ]
[0604] }
[0605] an auto filter json schema may be used to specify allowed filters such as e.g.:
[0606] {
[0607] "product" : ["ocrevus" , "tecentriq" . . . etc] ,
[0608] "indication" : [ "nsclc" . . etc] ,
[0609] "scientific_area" : ["oncology" , . . etc] ,
[0610] "language" : [ ] ,
[0611] "country" : [ ] ,
[0612] "content_type : [ ' xlsx ' , 'mp3 ' . . . etc]
[0613] }
[0614] and a validated filter (after enforcement) may be:
[0615] "product" : ["ocrevus" ] ,
[0616] "indication" : [ ] ,
[0617] "scientific_area" : ["oncology" ] ,
[0618] "language" : [ ] ,
[0619] "country" : [ ] ,
[0620] "content_type" : [ ]
[0621] }
[0622] Synonym Expansion for Filters: This process ensures that when a product or indication is detected in the user query, all associated synonyms are also included in the final payload. Synonyms are managed through a dedicated configuration file (synonyms.json) that maps canonical values to their alternate forms. This guarantees comprehensive coverage during search and improves recall without altering the original query intent. For example, for a user query: “Find Ocrevus materials”, the system may use a Synonyms.json Configuration (simplified) such as:
[0623] {
[0624] "Ocrevus" : ["Ocrevus" , "Ocrevus SC" , "Ocrevus IV" , . . . ] ,
[0625] }
[0626] to produce a final filter (after synonym expansion) such as:
[0627] {
[0628] "product" : [ "Ocrevus " , "Ocrevus SC" , "Ocrevus IV" ] ,
[0629] "indication" : [ ] ,
[0630] "scientific_area" : [ ] ,
[0631] "language" : [ ] ,
[0632] "country" : [ ]
[0633] }
[0634] Tracing and Logging: For the Query Optimizer module, logging and tracing ensure observability, performance monitoring, and easier debugging. Each request generates a unique tracejd (e.g., DEV_Finder_Optimizer_<timestamp>) and attaches structured metadata (requestjd, tool, module, env) via Portkey. This enables end-to-end tracking and better monitoring across environments.Hybrid Embedding strategyThis pipeline uses a Hybrid Embedding strategy, which combines Sparse and Dense vectors to leverage the strengths of both approaches. In a data preparation phase, source data is extracted (e.g. from a "Document DB Meta and Asset Collection") and categorized into two categories: (i) Keyword Fields (Sparse Only): name and title (e.g. Veeva Name and Veeva Title) - these are short, key identifiers suited for exact match / keyword search; (ii) Text / Asset Fields (Hybrid): Document Text, Image Caption, Table Caption, and Document Summary - these fields contain rich semantic content, benefitting from a more complex embedding strategy.
[0635] In a text chunking and cleaning stage, assets that are “document text” that are too large for effective embedding and search are process. Through a chunking process, the text is split into smaller, overlapping segments (chunks) of a predefined maximum length. This ensures that the retrieved results are manageable and highly relevant. Through a cleaning process, the text in all relevant fields is pre-processed to remove noise (e.g., punctuation, stop words for sparse models, special characters, formatting artifacts) to improve embedding quality.
[0636] As explained above in relation to the search query, different embeddings are used depending on the type of content. A sparse embedding is used for keywords fields (names and titles), which is used for keyword / lexical searches. This can use a model such as Qdrant BM42 (Best Match 42). It creates a highdimensional vector where only a few indices (corresponding to specific vocabulary words) have non-zero values. This embedding enables excellent for matching exact keywords and is highly efficient for filtering and retrieving documents based on specific names, titles, or jargon with a semantic edge. This process outputs two collections: a name collection and a title collection (e.g. Veeva Name Collection; Veeva Title Collection).
[0637] A dense embedding is used for image captions, table captions and summaries. The dense embeddings are used for semantic search. This process can use a transformer-based language model (such as e.g. mixbread-ai / mxbai-embed-large-v1 or others) to generate vectors. It creates a fixed-size, low-dimensional vector where all elements are non-zero (dense). The position of this vector in the vector space represents the semantic meaning of the text. This is important for semantic search — finding documents that are conceptually related to the query, even if they don't share exact keywords. This process outputs 3 collections: Image Caption Collection, Table Caption Collection, Summary Collection
[0638] A hybrid embedding is also created, which combines the Sparse and Dense vectors generated from the Document Text chunks. During a search query, the system uses both the keyword-matching (sparse) and the concept-matching (dense) scores, using a Reciprocal Rank Fusion (RRF) algorithm to merge the results lists. This provides the highest retrieval quality by ensuring the search captures both lexical (keyword) and semantic (meaning) relevance. This process outputs a Hybrid Text Collection (stores both sparse and dense components for each chunk).
[0639] A text collection can store metadata and details for text assets within documents. This can be associated with e.g. following fields: _id (Unique identifier for the document), documentjd (Identifier of the associated document), tagjd (Identifier for the tag associated with the text), index (Index of the text within the document), asset_type (Type of the asset (e.g., text)), asset_path (Path where the text asset is stored), page (Page number where the text is located), text (Content of the text asset), title (Title of the text asset).A table collection can store metadata and details for table assets within documents. This can be associated with e.g. the following fields: Jd (Unique identifier for the document), documentjd (Identifier of the associated document), tagjd (Identifier for the tag associated with the table), index (Index of the table within the document), asset_type (Type of the asset (e.g., table)), asset_path (Path where the table asset is stored), page (Page number where the table is located), text (Content or description of the table), title (Title of the table asset), caption (Caption of the table), asset_aspect_ratio (Aspect ratio of the table asset), asset_height (Height of the table asset), asset_size (Size of the table asset), asset_width (Width of the table asset), thumbnail_aspect_ratio (Aspect ratio of the thumbnail image), thumbnail_height (Height of the thumbnail image), thumbnail_path (Path where the thumbnail image is stored), thumbnail_size (Size of the thumbnail image), thumbnail_width (Width of the thumbnail image). The same fields may be used for images in an Image collection.
[0640]
[0641] Document
[0642] This module provides an interactive assistant that answers user questions about documents returned by Search Services.
[0643] Content Injection is the first step in the Document Q&A workflow. At this stage, the system prepares the prompt for the LLM by combining three components: the user query, the relevant document text, and the detected language. This ensures that the assistant’s responses are grounded in the correct context and tailored to the user’s intent. The user query represents the direct input provided by the user (example: “Summarize the safety profile of Ocrevus.”). The document text contains the relevant content retrieved from Search Services (e.g. extracted passages from clinical trial reports or product information documents). This text is injected into the prompt to ensure that answers are evidence-based and document-specific. Detected language is the language of the user query automatically identified by the system. The detected language is injected into the prompt to ensure that responses are generated in the appropriate language.
[0644] At the Query Language Detection step, the language of the user query is automatically identified to ensure that responses are generated in the correct linguistic format. This detection can be performed using Amazon Comprehend, or any other natural language processing service that provides reliable language identification.
[0645] User Intent Classification determines the purpose of the user’s query and ensures that the response is generated in alignment with the intended language and context. Intent classification helps the system decide whether the user is requesting factual information, a summary, a comparison, or a language-specific response. For example, for a user query: “Give me Ocrevus materials in Spanish”, the system interprets that the user here wants the response to be in “Spanish” language.
[0646] In the Override Detected Language step, the system ensures that the final response language aligns with the user’s explicit intent, even if the automatic detection service (e.g., Amazon Comprehend) identifies a different language. This guarantees that the assistant respects user preference and delivers answers in the requested language. For example, for a user query: “Find me Ocrevus materials in Spanish”, the User Intended Language detected by the LLM is “Spanish”, even though the query language detected is English. The response language is set to the user-intended language.Previous Chat History Integration ensures that the assistant maintains conversational continuity by summarizing prior interactions and injecting them into the LLM prompt. Summarization can be performed e.g. using Liunga, which condenses the last three user-assistant exchanges into a compact representation. This summary is then added to the prompt along with the current query, document text, and detected language.
[0647] Preventing External Knowledge Integration applies a rule that ensures that the assistant’s responses are strictly grounded in the provided documents and contextual data, without introducing information from outside sources. By enforcing this constraint, the system guarantees that answers remain accurate, compliant, and relevant to the user’s query.:
[0648] Compressed Chat History Saving ensures that the conversational context is preserved efficiently by summarizing recent interactions and storing them in the database. Summarization can be performed using the LLM Liunga, which condenses the last three exchanges into a concise representation. This enables continuity in future sessions without overloading the system with full transcripts.
[0649] Forthe Chat Al module, logging and tracing provide strong observability, real-time performance monitoring, and streamlined debugging of conversational workflows. Each request generates a unique tracejd (e.g., DEV_Finder_Chat_AI_<timestamp>) and attaches structured metadata (requestjd, tool, module, env) via Portkey. This ensures end-to-end visibility ofchat interactions and consistent tracking across environments.
[0650] Document Summary
[0651] The Document Summary step produces concise, query-focused summaries of the top documents retrieved by Search Services. This accelerates user review by highlighting the most relevant information aligned with the user’s query. The top e.g. five documents returned by Search Services are injected into the LLM prompt along with the user query. This ensures the summary is query-specific and evidence-based. The LLM synthesizes the content of the top e.g. five documents into a single, coherent summary .The summary is concise, focused on the user query, and avoids unnecessary details.
[0652] Forthe Summary module, logging and tracing ensure complete observability, performance monitoring, and simplified debugging of summarization workflows. Each request generates a unique tracejd (e.g., DEV_Finder_Summary_<timestamp>) and attaches structured metadata (requestjd, tool, module, env) via Portkey. This enables end-to-end request tracking, clear module-level visibility, and consistent monitoring across different environments.
[0653] References
[0654] A number of publications are cited above in order to more fully describe and disclose the invention and the state of the art to which the invention pertains. Full citations for these references are provided below. The entirety of each of these references is incorporated herein.
[0655] Jinze Bai, et al. 2023. Qwen-vl: A versatile vision-language model for understanding, localization, text reading, and beyond. arXiv preprint arXiv:2308.12966.
[0656] Tom Brown, et al. 2020a. Language models are few-shot learners. In Advances in Neural Information Processing Systems.Tom B Brown, et al. 2020b. Language models are few-shot learners. arXiv preprint arXiv:2005.14165. Zhe Chen, et al 2024. How far are we to gpt-4v? closing the gap to commercial multimodal models with open-source suites. arXiv preprint arXiv:2404.16821.
[0657] Tri Dao. 2023. Flashattention-2: Faster attention with better parallelism and work partitioning. arXiv preprint arXiv:2307.08691.
[0658] Anujkumarsinh Donvir, et al. 2024. The role of generative ai tools in application development: A comprehensive review of current technologies and practices. In 2024 International Conference on Engineering Management of Communication and Technology (EM- CTECH), pages 1-9.
[0659] Hugging Face. 2024. Transformers performance inference on gpu. Accessed: 2024-03-10.
[0660] Chaoyou Fu, et al. 2023. Mme: A comprehensive evaluation benchmark for multimodal large language models. arXiv preprint arXiv:2306.13394.
[0661] Chaoyou Fu, et al. 2024a. Video-mme: The first-ever comprehensive evaluation benchmark of multi-modal Urns in video analysis. arXiv preprint arXiv:2405.21075.
[0662] Chaoyou Fu, et al. 2024b. Mme-survey: A comprehensive survey on evaluation of multimodal Urns. arXiv preprint arXiv:2411.15296.
[0663] Gemini Team, Google: Petko Georgiev and 1135 other authors. 2024. Gemini 1.5: Unlocking multimodal understanding across millions of tokens of context. arXiv preprint arXiv:2403.05530.
[0664] Ben Goodrich, et al. 2019. Assessing the factual accuracy of generated text. In Proceedings of the 25th ACM SIGKDD International Conference on Knowledge Discovery Data Mining, pages 166-175.
[0665] Roberto Gozalo-Brizuela and Eduardo C. Garrido-Merchan. 2023. A survey of generative ai applications. Preprint, arXiv:2306.02781 .
[0666] Patrick Lewis, et al. 2021.Retrieval-augmented generation for knowledge-intensive NLP tasks. Preprint, arXiv:2005.11401.
[0667] Junnan Li, et al. 2022. Blip: Bootstrapped language-image pre-training for unified vision-language understanding and generation. In Proceedings of the 39th International Conference on Machine Learning (ICML), pages 12888-12900.
[0668] Qiang Li, et al. 2024. XIMAGENET-12: An explainable visual benchmark dataset for model robustness evaluation. In Synthetic Data for Computer Vision Workshop @ CVPR 2024.
[0669] Zongxia Li, et al. 2025. Benchmark evaluations, applications, and challenges of large vision language models: A survey. Preprint, arXiv:2501.02189.
[0670] Ismini Lourentzou, Chen Chen, and ChengXiang Zhai. 2021. Adarenet: Adaptive reweighted semisupervised active learning. In Proceedings of the 14th ACM International Conference on PErvasive Technologies Related to Assistive Environments (PETRA), pages 1-8.
[0671] Sandra C. Matz, et al. 2024. The potential of generative ai for personalized persuasion at scale. Scientific Reports, 14:4692.
[0672] OpenAI. 2023. ChatGPT for intelligence briefing: A case study. Accessed: March 2025.
[0673] OpenAI. 2024. ChatGPT: Optimizing language models for dialogue. Accessed: March 2025.
[0674] Long Ouyang, et al. 2022. Training language models to follow instructions with human feedback. In Advances in Neural Information Processing Systems, volume 35, pages
[0675] 52527730-27744.
[0676] Xiaoye Qu, et al. 2025. Look, compare, decide: Alleviating hallucination in large vision-language models via multi-view multi-path reasoning. In Proceedings of the 31st International Conference on Computational Linguistics. Association for Computational Linguistics.Alec Radford, et al. 2021. Learning transferable visual models from natural language supervision. In Proceedings of the 38th International Conference on Machine Learning (ICML), pages 8748-8763.
[0677] Aditya Ramesh, et al. 2021 . Dall e: Creating images from text. OpenAI Blog.
[0678] Robin Rombach, et al. 2022. Stable diffusion: High-resolution image synthesis with latent diffusion models. arXiv preprint arXiv:2112.10752.
[0679] Amanpreet Singh, et al. 2022. Flava: A foundational vision and language model for generalized multimodal learning. In Proceedings of the IEEE / CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 15638-15650.
[0680] Karan Singhal, et al. 2022. Large language models encode clinical knowledge. arXiv preprint arXiv:2212.13138.
[0681] Romal Thoppilan, et al. 2022. Lamda: Language models for dialog applications. arXiv preprint arXiv:2201.08239.
[0682] Ashish Vaswani, et al. 2017. Attention is all you need. In Advances in Neural Information Processing Systems.
[0683] Peng Wang, et al. 2022. Ofa: Unifying architectures, tasks, and modalities through a simple sequence-to-sequence learning framework. arXiv preprint arXiv:2202.03052.
[0684] Kaixun Yang, et al. 2024. Modifying ai, enhancing essays: How active engagement with generative ai boosts writing quality. Preprint, arXiv:2412.07200.
[0685] Yuanhan Zhang et al. 2023. Mmbench: Is your multi-modal model an all-around player? arXiv:2307.06281. Yi-Fan Zhang, et al. 2024a. Mme-realworld: Could your multimodal LLM challenge high-resolution real-world scenarios that are difficult for humans? arXiv preprint arXiv:2408.13257.
[0686] Yuanhan Zhang, et al. 2024b. Llava-next: A strong zero-shot video understanding model.
[0687] All references cited herein are incorporated herein by reference in their entirety and for all purposes to the same extent as if each individual publication or patent or patent application was specifically and individually indicated to be incorporated by reference in its entirety.
Claims
Claims:
1. A computer-implemented method for processing video content, the method comprising:generating one or more captions for the video content using a vision language model; extracting metadata associated with the video content;generating, using an audio language model, a transcript from audio content extracted from or associated with the video content;generating a vector representation of the video content based at least in part on the metadata, the transcript, and the one or more captions using a large language model; andstoring, in a database, the vector representation in association with one or more tags generated based on the captions and the metadata.
2. The method of claim 1 , wherein the method further comprises:processing one or more digital document files comprising text, image content and / or tabular content, said processing comprising extracting, from said digital document files, reference data comprising one or more of: text, image content, tabular content;generating a vector representation of the reference data; andstoring the vector representation in the database,optionally wherein the method further comprises extracting metadata associated with the digital document files, the metadata comprising one or more tags based on the reference data.
3. The method of claim 1 or claim 2, wherein the one or more captions comprise one or both of: a summary of the video content, and information identifying one or more key frames of the video content.
4. The method of claim 3, wherein the information identifying one or more key frames of the video content comprises for each of one or more key frames: (i) information uniquely identifying the key frame location in the video content, optionally comprising a timestamp, and (ii) information describing the content of the keyframe.
5. The method of any preceding claim, wherein:the method further comprises processing the video content with a video converter to extract audio content from the video content; and / orthe method further comprises translating the one or more captions and / or the transcript from a first language to a second language using a machine translation model; and / orthe vision language model is an attention-based model that uses scaled dot product attention.
6. The method of claim 5, wherein generating a vector representation of the video content based at least in part on the metadata, the transcript, and the one or more captions using a large language model comprises:generating, with the large language model, revised captions based on the metadata, the transcript, and the one or more captions,wherein the revised captions optionally include one or both of: a revised summary and revised information identifying one or more key frames.
7. The method of any preceding claim, wherein the metadata comprises a plurality of attributes of the video content selected from: a length of the video, a resolution of the video, a creator of the video, an orientation of the video, a rotation of the video, a frame rate of the video, a frame size of the video, a graphics mode of the video, a format of the video, a compression of the video, a creation date of the video, and / orwherein extracting the metadata comprises generating one or more tags based on the one or more captions by processing, with a large language model (LLM), the one or more captions.
8. The method of any preceding claim, wherein each of the one or more tags is one of:an extractive tag, which is a tag generated using a keyword matching process and a list of keywords and optionally synonyms thereof, optionally wherein extractive tags include one or both of a product tag and an indication tag, wherein a product tag is a tag that identifies a specific therapeutic product in a list of products, and an indication tag is a tag that identifies a clinical indication in a list of clinical indications; oran abstractive tag, which is a tag generated using contextual semantic analysis and a large language model, optionally wherein abstractive tags include one or more of: an audience tag, a content purpose tag, a content complexity tag, a non-clinical topic tag, and a clinical topic tag, wherein an audience tag is a tag that identifies an intended audience, a content purpose tag is a tag that identified a purpose of the document, a content complexity tag is a tag that indicates a level of technical detail, a non-clinical topics tag is a tag that identifies a specific non-clinical category of content, and a clinical topics tag is a tag that identifies a specific category of medical or scientific content.
9. The method of any of claims 6 to 8, further comprising:generating a knowledge graph representing the one or more revised captions, wherein each element of the revised caption is associated with a node, and nodes associated with elements of the revised caption are connected with a root node indicating the type of the element,optionally wherein the elements comprise key frames and the knowledge graph is displayed to a user through a user interface using a layout that places nodes associated with key frames located closer to each other in the video content closer together than nodes associated with key frames located further from each other in the video content, and / orwherein the method comprises generating revised captions using captions generated with two or more different vision language models thereby obtaining corresponding two or more sets of revised captions, and generating a knowledge graph for each set of revised captions.
10. The method of any preceding claim, wherein:the video content is a video of at least 30 minutes, and / orthe method does not comprise cutting or compressing the video content, and / orthe method comprises reducing the number of frame per second as the only data reduction step applied on the video content prior to generating the one or more captions for the video content using the vision language model.
11. The method of any preceding claim, further comprising:receiving a first search query from a user;generating one or more enhanced search queries based on the first search query; generating a vector representation for each of the one or more enhanced search queries; performing, with each vector representation, a vector search over the database; and obtaining a first set of search result from the vector search.
12. The method of claim 11 , further comprising:extracting one or more keywords from the one or more enhanced search queries; performing, with the one or more keywords, a keyword search over the database; obtaining a second set of search results from the keyword search;integrating the first set of search results with the second set of search results according to a ranking algorithm; andproviding the integrated search results to a user through a user interface.
13. The method of claim 11 or claim 12, wherein generating one or more enhanced search queries based on the first search query comprises:processing, with a large language model, the first search query and generating, with the large language model, one or more related search queries based at least in part on the processing; and / or generating one or more search filters based on the first search query, optionally wherein said generating comprises identifying one or more filters from a predetermined set of filters that apply to the search query and / or matching one or more keywords and synonyms associated with respective filters present in the first search query.
14. The method of any one of claims 11 to 13, wherein performing a keyword search over the database further comprises:matching the one or more keywords against one or more tags stored in the database; and / or matching the one or more keywords against the one or more captions or revised captions for the video content.
15. The method of any one of claims 12 to 14, wherein performing the keyword search over the database comprises applying a BM42 sparse retrieval algorithm to one or more document-identifying fields; and / orwherein performing the vector search over the database comprises retrieving documents based at least in part on semantic similarity to indexed content, the indexed content comprising one or more of: document body text, document summaries, image captions, and / or table captions; and / orwherein integrating the first set of search results with the second set of search results comprises combining ranked results from the vector search and ranked results from the keyword search using reciprocal rank fusion.
16. The method of any one of claims 11 to 15, wherein generating one or more enhanced search queries based on the first search query comprises:providing a raw user query and a structured auto-filter object to a large language model, the structured auto-filter object comprising predefined filter categories and corresponding canonical values and / or synonyms; andgenerating, by the large language model, an optimized search query and a structured set of filter values, wherein one or more predetermined semantic concepts, optionally a detected product value and / or indication value, are retained in both the optimized search query and the structured set of filter values;optionally wherein the structured set of filter values is a set of filter values validated against a predefined filter configuration such that unsupported filter values are excluded, and wherein one or more excluded filter categories are prevented from being populated by the large language model, and / or optionally wherein the structured set of filter values comprises a language filter and a country filter, wherein a language filter value and a country filter value are populated independently unless the first search query explicitly links the language value and the country value.
17. The method of any one of claims 12 to 16, further comprising:deriving, from metadata associated with documents retrieved in the integrated search results, one or more contextual filters for presentation to a user for iterative refinement of the search results; and re-ranking at least a subset of the documents based on a similarity between terms of the first search query and terms of the documents name and / or title,optionally wherein said similarity comprises a Jaccard similarity, and / oroptionally wherein a sparse-search-derived ranking score is determined from a weighted combination of a BM42 score and a fuzzy matching score.
18. A system comprising:one or more processors; andmemory storing instructions that when executed by the one or more processors cause the one or more processors to carry out the method of any preceding claim.
