Systems and methods for ai-assisted infringement assessment and patent portfolio enforcement ranking

LLMs automate the identification of patent infringers and portfolio ranking, addressing inefficiencies in conventional methods by providing efficient and precise infringement monitoring and portfolio evaluation.

WO2026232226A1PCT designated stage Publication Date: 2026-11-12PIONEERIP INC
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
PCT/CA2026/050652
Authority / Receiving Office
WO · WO
Patent Type
Applications
Current Assignee / Owner
Priority Date
2025-07-25
Filing Date
2026-05-04
Publication Date
2026-11-12

AI Technical Summary

Technical Problem

Conventional methods for identifying patent infringers are manual, costly, and time-consuming, lacking efficient and automated solutions for discovering potential infringers and assessing patent portfolio value.

Method used

Utilizing Large Language Models (LLMs) to automatically identify products or services that infringe patents and rank patent portfolios based on relevancy scores, generating textual summaries and refining search results through probabilistic approaches and vector embeddings.

Benefits of technology

Facilitates efficient and automated identification of potential infringers and enhances patent portfolio assessment by reducing manual effort and computational resources, enabling precise infringement monitoring and portfolio evaluation.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CA2026050652_12112026_PF_FP_ABST
    Figure CA2026050652_12112026_PF_FP_ABST
Patent Text Reader

Abstract

Systems and methods for AI assisted infringement assessment and patent portfolio enforcement ranking are disclosed herein. A method receives a patent document identifier associated with a target patent claim and identifies search results for products or services relevant to the target claim. A large language model (LLM) generates textual summaries corresponding to the identified products or services, and a relevancy score indicating similarity to the target claim is determined for ranking the identified products or services. A method receives multiple patent identifiers and for selected claims of interest of each patent, identifies relevant product or service search results and determines relevancy scores. Claim scores are determined based on the relevancy scores, for ranking the patents accordingly.
Need to check novelty before this filing date? Find Prior Art

Description

SYSTEMS AND METHODS FOR AI-ASSISTED INFRINGEMENT ASSESSMENT AND PATENT PORTFOLIO ENFORCEMENT RANKINGCROSS-REFERENCE TO RELATED APPLICATIONS

[0001] This application claims the benefit of United States Provisional Patent Application No. 63 / 800,191 filed May 5, 2025 and United States Provisional Patent Application No.63 / 851,199 filed July 25, 2025. The entire contents of United States Provisional Patent Application No. 63 / 800,191 and United States Provisional Patent Application No. 63 / 851,199 are hereby incorporated by reference.FIELD

[0002] The present embodiments relate generally to systems and methods for Al assisted infringement assessment and patent portfolio enforcement ranking, and more specifically, to systems and methods for Al-assisted infringement searching using LLM-generated textual summaries of identified products and services and Al-assisted patent portfolio ranking based on infringement.BACKGROUND

[0003] An issued patent offers a patentee enforceable rights to exclude others from making, using, selling, or importing products and services that fall within the bounds of the claims of the issued patent.

[0004] The discovery of potential infringers of the claims of a patent, however, has conventionally been performed based on market surveillance including private investigations of competitors and the analysis of commercially available products and services. This can be done by searching online marketplaces, product catalogs, and patent databases to identify products that may be in violation of their patents.

[0005] Such monitoring and discovery of potential infringers is complex, costly, and time consuming since it is driven by manual search methodologies. There remains an unmet need for efficient and automated solutions for the discovery of alleged infringers.SUMMARY

[0006] The present embodiments are directed to systems, methods and computer-readable media for Al-assisted infringement assessment and patent portfolio enforcement ranking. The present embodiments involve automatically identifying potential patent infringers by performing patent analysis using Large Language Models (LLMs), identifying products or services that mayinfringe a patent, generating textual summaries corresponding to the identified products and services and ranking the identified products and services based on a relevancy to the textual summaries and ranking patent documents in a patent portfolio based on the products and services identified as infringing the patent.

[0007] In accordance with a first aspect of this disclosure, there is provided a computer-implemented method for identifying products or services relevant to a target patent claim. The method involves operating at least one processor to: receive a patent document identifier associated with the target patent claim from a computing device in communication with the at least one processor via a network; obtain the target patent claim; retrieve, from a database in communication with the at least one processor via the network, patent document details associated with the target patent claim, using the patent document identifier; generate a set of queries, based on the retrieved patent document details and the target patent claim; search remote databases using the set of queries to identify products or services satisfying the query set in a set of returned search results; generate, by a large language model (LLM), textual summaries respectively corresponding to each of the identified products or services in the set of returned search results; score each of the identified products or services in the set of returned search results, based on the corresponding LLM-generated textual summaries, to provide a respective relevancy score for each identified product or service; generate a ranked list of the identified products or services having a relevancy score greater than a predetermined threshold; and display the ranked list.

[0008] In an example of the first example aspect of the method, wherein generating a set of queries comprises operating the at least one processor to: generate a first set of queries, based on the retrieved patent document details; search the remote databases using the first set of queries to identify products or services satisfying the first set of queries in a first set of returned search results; generate a second set of queries based on the first set of returned search results; and search the remote databases using the second set of queries to identify products or services satisfying the second set of queries in a second set of returned search results.

[0009] In an example of the first example aspect of the method, wherein the method further comprises operating the at least one processor to: analyze the first set of returned search results using a probabilistic approach to determine features of the preliminary search results that appear frequently in the first set of returned search results; validate one or more frequently appearing features in the first set of search results to generate a list of validated key features by operating the at least one processor to: refine the first set of search results to include only the one or more frequently appearing features representing a genuine entity; and process the refined first set of search results to consolidate the one or more frequently appearing features which represent thesame genuine entity; and generate the second set of queries, based on the list of validated key features.

[0010] In an example of the first example aspect of the method, wherein generating the textual summaries corresponding respectively corresponding to each of the identified products or services in the set of returned search results comprises operating the at least one processor to: provide a prompt to the LLM including the second set of search results, to instruct the LLM to generate a respective detailed description for each of the identified products or services in the second set of returned search results.

[0011] In an example of the first example aspect of the method, wherein scoring each of the identified products and services in the set of returned search results comprises operating the at least one processor to: transform, using a vector embedding model, the target claim into a corresponding target claim vector; transform, using the vector embedding model, the LLM-generated summary for each identified product or service into a corresponding summary vector; and compare the target claim vector with the summary vector for each identified product or service to generate a relevancy score for each identified product or service based on a similarity of the corresponding LLM-generated textual summary to the target claim.

[0012] In an example of the first example aspect of the method, further comprising operating the at least one processor to: extract one or more claim elements from the target claim; transform, using a vector embedding model, the one or more claim elements into corresponding one or more claim element vectors; transform, using the vector embedding model, the LLM-generated summary for the one of the identified products and services in the ranked list into a corresponding summary vector; compare the summary vector with each of the one or more claim element vectors to identify a relevance of the one of the identified products and services to each of the one or more claim elements; generate a claim chart for the one of the identified products and services in the ranked list, the claim chart identifying the relevance of the one of the identified products and services to each of the one or more claim elements; and display the claim chart.

[0013] In accordance with a second aspect of this disclosure, there is provided a system for identifying products or services relevant to a target patent claim. The system comprises: one or more processors; and a memory storing machine-executable instructions, which, when executed by the one or more processors, cause the system to: receive a patent document identifier associated with the target patent claim from a computing device in communication with the at least one processor via a network; obtain the target patent claim; retrieve, from a database in communication with the at least one processor via the network, patent document details associated with the target patent claim, using the patent document identifier; generate a set of queries, based on the retrievedpatent document details and the target patent claim; search remote databases using the set of queries to identify products or services satisfying the query set in a set of returned search results; generate, by a large language model (LLM), textual summaries respectively corresponding to each of the identified products or services in the set of returned search results; score each of the identified products or services in the set of returned search results, based on the corresponding LLM-generated textual summaries, to provide a respective relevancy score for each identified product or service; generate a ranked list of the identified products or services having a relevancy score greater than a predetermined threshold; and display the ranked list.

[0014] The following embodiments are directed to systems, methods and computer-readable media for ranking patent documents in a patent portfolio based on an automatic identification of potential patent infringers. The present embodiments involve automatically identifying potential patent infringers by performing patent analysis using Large Language Models (LLMs), identifying products or services that may infringe a patent and ranking patent documents based on the products and services identified as potentially infringing the patent documents.

[0015] In accordance with a third of this disclosure, there is provided a computer-implemented method for patent portfolio ranking. The method involves operating at least one processor to: receive a plurality of patent numbers corresponding to a plurality of patents; for each claim of interest of each patent of the plurality of patents, identify at least one search result corresponding to at least one product or service relevant to the claim of interest; for each identified search result, determine a relevancy score that includes an indication of a similarity between the product or service corresponding to the search result and the claim of interest; determine a claim score based on the relevancy score of at least one identified search result; for each patent of the plurality of patents, determine a patent score based on the claim score of at least one claim of interest; rank the plurality of patents based at least in part on the patent scores of the plurality of patents; and display a ranked list of the plurality of patents.

[0016] In an example of the third example aspect of the method, wherein the claim of interest is an independent claim and identifying the at least one product or services relevant to the claim of interest comprises operating the at least one processor to: obtain a text of the claim of interest; retrieve, from a database in communication with the at least one processor via a network, patent document details associated with the claim of interest, using the patent number; generate a set of queries based on the retrieved patent document details; and search remote databases using the set of queries to identify search results corresponding to the products or services satisfying the query set.

[0017] In an example of the third example aspect of the method, further comprisingoperating the at least one processor to: for each patent of the plurality of patents, determine a weighted patent score by assigning a weight to each patent of the plurality of patents based on one or more of: a financial reward associated with the patent, defense patents associated a potential infringer of the patent, a litigation history of the patent, an importance of a product or service associated with the patent; and rank the plurality of patents based on the weighted patent scores of the plurality of patents.

[0018] In an example of the third example aspect of the method, wherein determining the claim score based on the relevancy score of the at least one identified search result comprises operating the at least one processor to compute a sum of the relevancy score of the at least one identified search result and / or determining the patent score comprises operating the at least one processor to compute an average of the claim score of the at least one claim of interest.

[0019] In an example of the third example aspect of the method, further comprising operating the at least one processor to: for each patent of the plurality of patents, determine a normalized patent score; and display the normalized patent scores in association with the ranked list of the plurality of patents.

[0020] In an example of the third example aspect of the method, wherein determining the relevancy score for the at least one identified search result comprises operating the at least one processor to: calculate a vector embedding of the claim of interest; and calculate a vector embedding of the search result and comparing the vector embedding of the claim and the vector embedding of the search result to obtain the relevancy score for the search result.

[0021] In accordance with a fourth of this disclosure, there is provided a system for patent portfolio ranking. The system comprises: one or more processors; and a memory storing machineexecutable instructions, which, when executed by the one or more processors, cause the system to: receive a plurality of patent numbers corresponding to a plurality of patents; for each claim of interest of each patent of the plurality of patents, identify at least one search result corresponding to at least one product or service relevant to the claim of interest; for each identified search result, determine a relevancy score that includes an indication of a similarity between the product or service corresponding to the search result and the claim of interest; determine a claim score based on the relevancy score of at least one identified search result; for each patent of the plurality of patents, determine a patent score based on the claim score of at least one claim of interest; rank the plurality of patents based at least in part on the patent scores of the plurality of patents; and display a ranked list of the plurality of patents.

[0022] In accordance with a fifth aspect of this disclosure, there is provided a non-transitory computer readable medium having a set of executable instructions stored thereon that, when executed by at least one processor, cause the at least one processor to perform any of the method steps described herein.DRAWINGS

[0023] An illustrative embodiment of the present invention will now be described in detail with reference to the diagrams, in which:

[0024] FIG. 1 shows a patent analytics and infringement monitoring system in communication with external components, in accordance with one or more embodiments;

[0025] FIG. 2 shows a device diagram for a server of the patent analytics and infringement monitoring system of FIG. 1, in accordance with one or more embodiments;

[0026] FIG. 3 shows a method diagram of a method of operating the patent analytics and infringement monitoring system in accordance with one or more embodiments.

[0027] FIG. 4 shows an input / output diagram in accordance with one or more embodiments;

[0028] FIG. 5 shows another method diagram of a method of operating the patent analytics and infringement monitoring system in accordance with one or more embodiments;

[0029] FIG. 6 shows a flowchart of an example method for fine-tuning a machine-learning model, in accordance with one or more embodiments;

[0030] FIG. 7 shows another method diagram of operating the patent analytics and infringement monitoring system in in accordance with one or more embodiments;

[0031] FIG. 8 shows another method diagram of a method of operating the patent analytics and infringement monitoring system in accordance with one or more embodiments;

[0032] FIG. 9 shows a data diagram in accordance with one or more embodiments;

[0033] FIG. 10A shows a screenshot of graphical user interface (GUI) illustrating an example embodiment of an application of a system for patent analytics and for monitoring patent infringement;

[0034] FIG. 10B shows another screenshot of graphical user interface (GUI) illustrating an example embodiment of an application of a system for patent analytics and for monitoring patent infringement;

[0035] FIG. 11 shows another screenshot of a GUI illustrating an example embodiment ofan application of a system for patent analytics and for monitoring patent infringement;

[0036] FIG. 12 shows a flowchart of an example method for providing updated search results by fine-tuning of machine learning models using a feedback dataset, in accordance with one or more embodiments;

[0037] FIG. 13 shows another data diagram in accordance with one or more embodiments;

[0038] FIG. 14 shows a flowchart of an example method for identifying products or services that may infringe a patent, in accordance with one or more embodiments;

[0039] FIG. 15 shows a flowchart of an example method for identifying one or more patents that may be infringed by a product or service, in accordance with one or more embodiments;

[0040] FIG. 16 shows a flowchart of an example method of ranking products or services that may infringe a patent, in accordance with one or more embodiments;

[0041] FIG. 17 is a schematic representation of another method of operating the patent analytics and infringement monitoring system to identify products or services that may infringe a patent, in accordance with one or more embodiments;

[0042] FIG. 18 is a flowchart of an example method for identifying a list of products or services potentially infringing a target patent, in accordance with one or more embodiments;

[0043] FIG. 19A is a schematic representation of functionalities of a patent analytics and infringement monitoring system, in accordance with one or more embodiments;

[0044] FIG. 19B is a schematic representation of functionalities of a patent analytics and infringement monitoring system, in accordance with one or more embodiments;

[0045] FIG. 19C is a schematic representation of functionalities of a patent analytics and infringement monitoring system, in accordance with one or more embodiments;

[0046] FIG. 19D is a schematic representation of functionalities of a patent analytics and infringement monitoring system, in accordance with one or more embodiments;

[0047] FIG. 19E is a schematic representation of a patent analytics and infringement monitoring system, in accordance with one or more embodiments;

[0048] FIG. 20 shows another screenshot of a GUI illustrating an example embodiment of an application of a system for patent analytics and for monitoring patent infringement;

[0049] FIG. 21 shows another screenshot of a GUI illustrating an example embodiment of an application of a system for patent analytics and for monitoring patent infringement;

[0050] FIG. 22 is a schematic representation of another method for operating the patent analytics and infringement monitoring system to identify patents or patent applications that may be infringed by products or services, in accordance with one or more embodiments;

[0051] FIG. 23 is a schematic diagram showing components of the patent analytics and infringement monitoring system;

[0052] FIG. 24 is a schematic diagram showing a training model for a machine learning framework, in accordance with one or more embodiments;

[0053] FIG. 25 is a flowchart of an example method for ranking a patent portfolio, in accordance with one or more embodiments;

[0054] FIG. 26 is a schematic diagram of the method of FIG. 25;

[0055] FIG. 27 is a table showing example scores that may be computed for a patent;

[0056] FIG. 28 is a table showing functions that may be used to compute patent scores;

[0057] FIG. 29 is a table showing example patent scores and weighted patent scores computed for a patent portfolio;

[0058] FIG. 30 is a screenshot of a GUI illustrating an example ranked patent portfolio, in accordance with one or more embodiments.

[0059] FIG. 31 is a schematic representation of another method of operating the patent analytics and infringement monitoring system to identify products or services that may infringe a patent, in accordance with one or more embodiments;

[0060] FIG. 32 is a schematic diagram of the method of FIG. 31;

[0061] FIG. 33 is a schematic representation of another method of operating the patent analytics and infringement monitoring system to generate a claim chart, in accordance with one or more embodiments; and

[0062] FIG. 34 is a table showing an example configuration of a claim chart, in accordance with one or more embodiments.DESCRIPTION OF VARIOUS EMBODIMENTS

[0063] Various embodiments will now be described below to provide an example of the claimed subject matter. No example described below limits any claimed subject matter and any claimed subject matter may cover embodiments such as systems or methods that differ from those described below.

[0064] Furthermore, it will be appreciated that for simplicity and clarity of illustration, where considered appropriate, reference numerals may be repeated among the figures to indicate corresponding or analogous elements. In addition, numerous specific details are set forth in order to provide a thorough understanding of the examples described herein. However, it will be understood by those of ordinary skill in the art that the examples described herein may be practiced without these specific details. In other instances, well-known methods, procedures, and components have not been described in detail so as not to obscure the examples described herein. Also, the description is not to be considered as limiting the scope of the examples described herein.

[0065] It should also be noted that, as used herein, the wording “and / or” is intended to represent an inclusive-or. That is, “X and / or Y” is intended to mean X or Y or both, for example. As a further example, “X, Y, and / or Z” is intended to mean X or Y or Z or any combination thereof.

[0066] It should be noted that terms of degree such as "substantially", "about" and "approximately" as used herein mean a reasonable amount of deviation of the modified term such that the end result is not significantly changed. These terms of degree may also be construed as including a deviation of the modified term if this deviation would not negate the meaning of the term it modifies.

[0067] Furthermore, the recitation of numerical ranges by endpoints herein includes all numbers and fractions subsumed within that range (e.g., 1 to 5 includes 1, 1.5, 2, 2.75, 3, 3.90, 4, and 5). It is also to be understood that all numbers and fractions thereof are presumed to be modified by the term "about" which means a variation of up to a certain amount of the number to which reference is being made if the end result is not significantly changed.

[0068] Some elements herein may be identified by a part number, which is composed of a base number followed by an alphabetical or subscript-numerical suffix (e.g., 112a, or 1121). Multiple elements herein may be identified by part numbers that share a base number in common and that differ by their suffixes (e.g., 1121, 1122, and 1123). All elements with a common base number may be referred to collectively or generically using the base number without a suffix (e.g., 112).

[0069] The example systems and methods described herein may be implemented in hardware or software, or a combination of both. In some cases, the examples described herein may be implemented, at least in part, by using one or more computer programs, executing on one or more programmable devices comprising at least one processing element, a data storage element (including volatile and non-volatile memory and / or storage elements), and at least one communication interface. These devices may also have at least one input device (e.g., a keyboard,a mouse, a touchscreen, and the like), and at least one output device (e.g., a display screen, a printer, a wireless radio, and the like) depending on the nature of the device. For example, and without limitation, the programmable devices (referred to below as computing devices) may be a server, network appliance, embedded device, computer expansion module, a personal computer, laptop, personal data assistant, cellular telephone, smart-phone device, tablet computer, a wireless device or any other computing device capable of being configured to carry out the methods described herein.

[0070] In some examples, the communication interface may be a network communication interface. In examples in which elements are combined, the communication interface may be a software communication interface, such as those for inter-process communication (IPC). In still other examples, there may be a combination of communication interfaces implemented as hardware, software, and a combination thereof.

[0071] Program code may be applied to input data to perform the functions described herein and to generate output information. The output information is applied to one or more output devices, in known fashion.

[0072] Each program may be implemented in a high-level procedural, declarative, functional or object-oriented programming and / or scripting language, or both, to communicate with a computer system. However, the programs may be implemented in assembly or machine language, if desired. In any case, the language may be a compiled or interpreted language. Each such computer program may be stored on a storage media or a device (e.g., ROM, magnetic disk, optical disc) readable by a general or special purpose programmable computer, for configuring and operating the computer when the storage media or device is read by the computer to perform the procedures described herein. Examples of the system may also be considered to be implemented as a non-transitory computer-readable storage medium, configured with a computer program, where the storage medium so configured causes a computer to operate in a specific and predefined manner to perform the functions described herein.

[0073] Furthermore, the example system, processes and methods are capable of being distributed in a computer program product comprising a computer readable medium that bears computer usable instructions for one or more processors. The medium may be provided in various forms, including one or more diskettes, compact disks, tapes, chips, wireline transmissions, satellite transmissions, internet transmission or downloads, magnetic and electronic storage media, digital and analog signals, and the like. The computer useable instructions may also be in various forms, including compiled and non-compiled code.

[0074] Various examples of systems, methods and computer programs products are described herein. Also, in the various user interfaces illustrated in the figures, it will be understood that the illustrated user interface text and controls are provided as examples only and are not meant to be limiting. Other suitable user interface elements may be used with alternative implementations of the systems and methods described herein.

[0075] Conventionally, patent infringement is discovered through market surveillance, including through monitoring the commercial activities of competitors, or by monitoring the Web through tools such as search engine alerts. These techniques, however, are time-consuming and costly, and contribute to the majority of patents being unenforced. Further, monitoring the web often requires specialized knowledge of search terms to be used and requires patent owners to manually create alerts or other monitoring tools. Because of the difficulty in monitoring patent infringement, assessing the value of patent portfolios can be difficult and typically requires expert knowledge.

[0076] Similarly, assessing whether a product or service is likely to infringe a patent is often costly and time-consuming, since the process typically requires specialized patent databases, expert knowledge, and a manual assessment of claims.

[0077] The disclosed systems and methods can identify products or services that may infringe claims of a patent and patents that may be infringed by a product or service and patent applications that may be relevant to a product or a service (e.g., patent applications containing claims, that if issued, could be infringed by a product or service. The disclosed systems and methods can be used for actively monitoring for patent infringement and for ranking patent portfolios. Ranking patent portfolios according to the potential for infringement can enable patent owners or patent portfolio managers to assess the value of patent portfolios.

[0078] The described embodiments can automatically identify products or services that may infringe patent claims and patents that may be infringed by a product or service without requiring the user to have specialized knowledge of patents and patent terminology and without requiring the user to have competitive knowledge. When compared to conventional methods, the described embodiments can more efficiently monitor for patent infringement. The described embodiments can also more efficiently use computational resources, since they do not require continuous monitoring of the web, as is required with conventional search engine alerts, and can reduce the number of searches performed (e.g., searches conducted by non-expert users who may not be using adequate search terms).

[0079] At least some of the embodiments described herein can identify products or servicesthat may be relevant to a claim being drafted. For example, at least some of the embodiments described herein can enable a user, such as a patent practitioner to evaluate a draft claim drafted and / or amended (e.g., a mock claim) by the patent practitioner and identify products or services that would infringe the proposed, if the proposed claim was a granted patent claim to assist the patent practitioner in drafting or amending the claim. In this manner, the language of a claim may be adjusted prior to the filing of an application (e.g., a continuation application, a divisional application, a continuation-in-part application) or during its prosecution in order to take into account potential infringement targets.

[0080] At least some of the embodiments described herein can analyze a patent to identify its validity. For example, at least some of the embodiments described herein can enable a user, such as a patent practitioner, a patent owner, or a person seeking to invalidate a patent to evaluate whether the validity of a given patent may be challenged due to products or services that existed prior to the priority date of the patent.

[0081] The embodiments described herein can involve fine-tuning a large language model (LLM) used for identifying products or services that may infringe claims of a patent using a litigation dataset.

[0082] As used herein, the term “product” includes any subject matter that involves physical elements, including but not limited to, apparatuses, devices, systems, formulations that may be patented and is not limited to physical, software or any other type of products. The term “service” includes any subject matter that involves including methods, processes and that may be patented.

[0083] As used herein, the term “patent”, unless otherwise indicated, includes utility models.

[0084] As will be described, at least some of the embodiments described herein can determine a relevancy score that is indicative of the similarity between claims of a patent and a product or service, as described by a product or service description. The relevancy score can enable patent owners, product manufacturers or service providers to assess the risk of infringement. The disclosed systems and methods may rank patent documents in a patent portfolio based on the relevancy scores. Patent portfolio ranking can assist patent owners and patent portfolio managers in evaluating the strength, value, potential and potential litigation value of a collection of patents. Patent portfolio ranking can assist patent owners in assessing whether to maintain their patents. For example, patent owners may favor maintaining patents that have a high potential litigation value.

[0085] The disclosed systems and methods allow patent owners to automatically identify products that may infringe their patents and allow product manufacturers to automatically identify patents that may be infringed by their products, reducing the need for manual searches.

[0086] At least one of the embodiments described herein can determine a relevancy score that is indicative of a risk of a patent being invalidated.

[0087] At least some of the embodiments described herein can involve fine-tuning the LLM using a feedback dataset so that improved search results can be obtained.

[0088] Reference is first made to FIG. 1, which illustrates an example block diagram 100 of a patent analytics and infringement monitoring system 108 in communication with an external data storage 102 and a computing device 106 via a network 104. Although only one computing device 106 is shown in FIG. 1, the patent analytics and infringement monitoring system 108 may be in communication with a greater number of computing devices 106. The patent analytics and infringement monitoring system 108 communicate with the computing device(s) 106 over a wide geographic area via the network 104. While the patent analytics and infringement monitoring system 108 and the computing device 106 are shown as separate components, in some cases, the patent analytics and infringement monitoring system 108 or one or more components of patent analytics and infringement monitoring system 108 may be implemented within the computing device 106.

[0089] The patent analytics and infringement monitoring system 108 includes a storage component 110, a processor 112, and a communication component 114. The patent analytics and infringement monitoring system 108 can be implemented with more than one computer server distributed over a wide geographic area and connected via the network 104. The storage component 110, the processor 112 and the communication component 114 may be combined into a fewer number of components or may be separated into further components. The patent analytics and infringement monitoring system 108 can be configured to identify products or services that may infringe patent of interest or identify patents that may be infringed by a product or service, depending on the implementation of patent analytics and infringement monitoring system 108.

[0090] The patent analytics and infringement monitoring system 108 can be configured to rank the products or services, or patents, according to their relevance, as will be described in further detail below. For example, the patent analytics and infringement monitoring system 108 can implement a subsystem for ranking product / services-patent evaluation search results. The patent analytics and infringement monitoring system 108 can also include an LLM fine-tuning subsystem that can fine-tune one or more LLM subsystems for identifying products or services that mayinfringe a patent or interest.

[0091] The LLM subsystem(s) may use a pre-trained model or fine-tuned model (e.g. GPT-4 or similar) to process inputs from the computing device 106 via server. The LLM subsystem may process the inputs from the computing device 106, collect patent data and analyze it using the LLM subsystem(s), generate one or more search queries (for example, Google searches, requests to web servers, or requests to user remote devices). The results of said searches may be used to generate search results that may be provided in user interfaces (see e.g. FIG. 11) for display on the computing device 106. Detailed prompts and predefined crafted prompts provided by a server may enable the LLM subsystem(s) to adapt to the volume and type of input data without making any modifications to its base model. For example, prompts may be created to translate patent applications in non-English languages, to caption patent application drawings or formulae in patent documents, to parse the structure of different national patent documents, and the like. The LLM subsystem(s) may include one or more Large Language Models (LLMs).

[0092] The prompts sent to the LLM subsystem(s) may include additional information identified in a vector database in the storage component 110 of the patent analytics and infringement monitoring system 108 prior to the prompt being executed against an LLM subsystem as described herein.

[0093] The LLM subsystem(s) may include a first party LLM trained and / or fine-tuned based on the litigation dataset (see e.g. FIG. 9), the feedback dataset, and other patent data or product or service catalogs or any other dataset as are known.

[0094] The one LLM subsystem(s) may include one or more of the following (it is noted that the following are non-limiting examples, and that other models may be used):GPT-2 made by OpenAI®GPT-3 made by OpenAI®GPT-3.5 Turbo made by OpenAI®GPT-4 made by OpenAI®GPT-4 Turbo made by OpenAI®GPT-4o made by OpenAI®Claude 1 made by Anthropic®Claude 2 made by Anthropic®Claude 3 made by Anthropic®LaMDA (Language Models for Dialog Applications) made by Google®LLaMA (Large Language Model Meta Al) made by Meta ®PaLM 2 (Pathways Language Model 2) Google®Llama 2 made by Meta®SDXL made by Stability Al®A BART trained model, i.e., a model trained using a denoising autoencoder for pretraining sequence-to-sequence models;ALBERT, A Lite BERT for Self-supervised Learning of Language Representations.

[0095] The processor 112 can be implemented with any suitable processor, controller, digital signal processor, graphics processing unit, application specific integrated circuits (ASICs), and / or field programmable gate arrays (FPGAs) that can provide sufficient processing power for the configuration, purposes and requirements of patent analytics and infringement monitoring system 108. The processor 112 can include more than one processor with each processor being configured to perform different dedicated tasks. For example, the processor 112 can include a one or more processors for fine-tuning the LLM subsystem, one or more processors for identifying products or services that may infringe a target patent claim (e.g., a claim of a patent of interest, a draft claim of a patent application) and / or one or more processors for ranking products or services identified as potentially infringing a target patent claim.

[0096] The communication component 114 can include any interface that enables the patent analytics and infringement monitoring system 108 to communicate with various devices and other systems. For example, the communication component 114 can receive inputs (e.g., patent numbers, patent publication numbers, patent application numbers, utility model numbers, claims of interest including draft claims, URL, product or service description) from the computing device 106 and store the inputs in the storage component 110 or external data storage 102. The processor 112 can then process the inputs according to the methods described herein.

[0097] The communication component 114 can include at least one of a serial port, a parallel port or a USB port, in some embodiments. The communication component 114 may also include an interface to component via one or more of an Internet, Local Area Network (LAN), Ethernet, Firewire, modem, fiber, or digital subscriber line connection. Various combinations of these elements may be incorporated within the communication component 114. For example, the communication component 114 may receive input from various input devices, such as a mouse, a keyboard, a touch screen, a thumbwheel, a trackpad, a track-ball, a card-reader, voice recognitionsoftware and the like depending on the requirements and implementation of the patent analytics and infringement monitoring system 108.

[0098] The storage component 110 can include RAM, ROM, one or more hard drives, one or more flash drives or some other suitable data storage elements such as disk drives. The storage component 110 can include one or more databases for storing information extracted from patents, vector embeddings of patent claims, vector embeddings of product or service descriptions or product / service website contents, pre-defined search queries, search results, litigation datasets, user feedback information, a feedback dataset, previous product-patent pairs or service-patent pairs evaluated, models for extracting information from patents and / or product / service websites, models for generating search queries, search parameters or filters and ranking refinement models. The litigation dataset can be saved according to the method in FIG. 6. The database(s) may store a vector database populated with document embeddings based on the litigation dataset as described herein. The storage component 110 can include databases including a Structured Query Language (SQL) such as PostgreSQL® or MySQL® or a not only SQL (NoSQL) database such as MongoDB™. The vector database(s) can include a Pinecone database.

[0099] The external data storage 102 can store data similar to that of the storage component 110. The external data storage 102 can, in some embodiments, be used to store data that is less frequently used and / or older data. In some embodiments, the external data storage 102 can be a third-party data storage that stores data that can be analyzed by the patent analytics and infringement monitoring system 108. For example, the external data storage 102 can store patents or patent data derived from patents (e.g., subject matter, industry, classifications). The external data storage 102, for example, can be the patent database of a national patent office, such as, for example, the United States Patent and Trademarks Office or commercially available databases, for example, by Questel™, LexisNexis™, ReedTech™, Google Patents™, and the like.

[0100] The external data storage 102 can include data servers that store data uploaded on the internet. The data stored in the external data storage 102 can be retrieved by the computing device 106 and / or patent analytics and infringement monitoring system 108 via the network 104. For example, the patent analytics and infringement monitoring system 108 can retrieve data via remote systems 120 using search engines (e.g., Google®), retrieve websites locatable using search engines and / or data otherwise available from publicly available websites. Although only one external data storage 102 is shown, it will be understood that the external data storage can include multiple data storages that can be distributed over a wide geographic area. In at least one embodiment, the external data storage 102 is implemented as a cloud storage.

[0101] The remote systems 120 may include web servers and may be reachable usingHTTP requests, or other network requests as known. Alternatively, the remote systems 120 may be located at a client venue (not shown). This may include a private network controlled by the user for example, the end-user of the patent analytics and infringement monitoring system 108. The user at computing device 106 may specify one or more remote systems 120 to be searched when a user submits a search request to the server. The remote system(s) 120 may include web servers and may be reachable using HTTP requests, or may be accessible using a connector to a proprietary client storage system. The remote system(s) 120 may be created by the user and may describe evidence of third-party products or services. The remote system(s) 120 may include public websites, websites for news publications, websites for product monographs, and the like. The remote system(s) 120 can include one or more remote devices.

[0102] The computing device 106 can include any device capable of communicating with other devices through a network such as the network 104 through a wireless connection. The computing device 106 can include a processor and memory, and may be an electronic tablet device, a personal computer, workstation, server, portable computer, mobile device, personal digital assistant, laptop, smart phone, WAP phone, an interactive television, video display terminals, gaming consoles, and portable electronic devices or any combination of these. The computing device 106 can be a user device that can transmit patent numbers of patents of interest, claims of interest, and / or product or service descriptions and search parameters and receive ranked search results.

[0103] The computing device 106 can be a user device used by an end-user such as a patent owner, a lawyer, a patent practitioner, an engineer, a product manufacturer, a product designer, or any other intended end-user of the patent analytics and infringement monitoring system 108, i.e., an end-user of software application for evaluating patent analytics, including but not limited to patent portfolio review and analysis of potential infringing products or services, or infringed patents. The computing device 106 can be used to access a software application (not shown) running on a server (not shown) over network 104, which may be provided by a downloadable application on user device 102 that connects to the server using an API, or using a web-based application in a browser on computing device 106. The one or more computing devices 106 may be any two-way communication device with capabilities to communicate with other devices. A computing device 106 may be, for example, a mobile device such as mobile devices running the Google® Android® operating system or Apple® iOS® operating system. A computing device 106 may also be, for example, a personal computer operating the Windows® or MacOS® operating system.

[0104] The application provided to the end-user either via an application at computingdevice 106 or via a web interface from the server may include a user interface that accepts inputs (e.g., patent numbers, claims, product descriptions, service descriptions, other search parameters) in various forms, such as text or image-based inputs.

[0105] The software application running on the one or more computing devices 106 may communicate with the server using an Application Programming Interface (API) endpoint, and may send various inputs and other requests for processing by the patent analytics and infringement monitoring system 108.

[0106] The software application running on the one or more computing devices 106 may display one or more user interfaces on a display device of the user device computing device 106, including, but not limited to, the graphical user interface shown in FIGS. 10 and 11. A browser may be used at the computing device 106 to access the web application running on the server.

[0107] The network 104 may be any network or network components capable of carrying data including the Internet, Ethernet, fiber optics, satellite, mobile, wireless (e.g. Wi-Fi, WiMAX), SS7 signaling network, fixed line, local area network (LAN), wide area network (WAN), a direct point-to-point connection, mobile data networks (e.g., Universal Mobile Telecommunications System (UMTS), 3GPP Long-Term Evolution Advanced (LTE Advanced), Worldwide Interoperability for Micro wave Access (WiMAX), and the like.) and others, including any combination of these.

[0108] Referring next to FIG. 2, there is shown a device diagram 200 of a server of the patent analytics and infringement monitoring system 108 of FIG. 1.

[0109] The network unit 204 can include wired or wireless connection capabilities. The network unit 204 can include a radio that communicates using standards such as IEEE 802.11a, 802.11b, 802.11g, or 802.1 In. The network unit 204 can be used by the server 200 to communicate with other devices or computers.

[0110] Network unit 204 may communicate with a network, such as network 104 (see FIG.1).

[0111] The display 206 may be an LED or LCD based display and may be a touch sensitive user input device that supports gestures.

[0112] The processor unit 208 controls the operation of the server 200. The processor unit 208 can be any suitable processor, controller or digital signal processor that can provide sufficient processing power depending on the configuration, purposes and requirements of the server 200 as is known by those skilled in the art. For example, the processor unit 208 may be a high-performance general processor. In alternative embodiments, the processor unit 208 can include more than one processor with each processor being configured to perform different dedicated tasks. The processor unit 208 may include a standard processor, such as an Intel® processor or an AMD® processor. The processor unit 208 can be the processor 112 of FIG. 1.

[0113] The memory unit 210 comprises software code for implementing an operating system 220, programs 222, databases 224, web scraper 226, LLM unit 228, user interface engine 230 and web / API unit 232.

[0114] The memory unit 210 can include RAM, ROM, one or more hard drives, one or more flash drives or some other suitable data storage elements such as disk drives, and the like. The memory unit 210 is used to store an operating system 220 and programs 222 as is commonly known by those skilled in the art.

[0115] The I / O unit 212 can include at least one of a mouse, a keyboard, a touch screen, a thumbwheel, a trackpad, a trackball, a card-reader, an audio source, a microphone, voice recognition software and the like again depending on the particular implementation of the server 200. In some cases, some of these components can be integrated with one another.

[0116] The power unit 216 can be any suitable power source that provides power to the server 200 such as a power adaptor or a rechargeable battery pack depending on the implementation of the server 200 as is known by those skilled in the art.

[0117] The operating system 220 may provide various basic operational processes for the server 200. For example, the operating system 220 may be a server operating system such as Ubuntu® Linux, Microsoft® Windows Server® operating system, or another operating system.

[0118] The programs 222 include various user programs. They may include several hosted applications delivering services to users over the network, for example, a patent portfolio management system, or patent analysis system.

[0119] In one or more embodiments, the programs 222 may provide a patent analytics and infringement monitoring platform that is web-based, or client-server-based application via Web / API Unit 232 that provides for the patent analysis and infringement monitoring in conjunction with an application at a user device.

[0120] The databases 224 may be at least one of a relational database or a vector database. The databases 224 can be stored in the storage component 110 of FIG. 1. There may be multiple different databases 224. The database 224 may run on the same server as the other components of the patent searching system, or alternatively, may run on a different server in networkcommunication with the other components of the patent searching system. The relational database may include a litigation dataset or / and a feedback dataset. The litigation dataset (for example, see FIG. 9) may include product-patent or service-patent associations, including content associated with a product or service subject to patent litigation. The content in relational database 224 may include content collected by web scraper 226. The content in the vector database 224 may include document embeddings associated with the content in the litigation dataset or the feedback dataset.

[0121] The databases 224 may index or store content ingested from remote system(s) 120 or retrieved from the external database 102, which may include a plurality of user content including documents associated with a products or services.

[0122] While the databases 224 is shown as operating on a single server 200, it is understood that the database may be located separately on another server or system in network communication with the server 200. The databases 224 may store the user content from the remote system(s) 120 in an indexed format.

[0123] The product / service content items associated with litigation records in the litigation dataset in database 224 may be stored using a collection timestamp by the web scraper 226. The web scraper 226 may, on user request or as a function of a periodic polling of remote systems may make a request to a remote system 120 to capture the content (including both text content and other media content) from the remote system and store it in the database 224.

[0124] The product / service content items in database 224 may include text data, media data such as image data, and screenshot data associated with a browser rendering of a website at a remote system. The content entries may include language translations of text data performed by the LLM subsystem.

[0125] The product / service content items in database 224 may include content created by the LLM Unit 228 when an LLM response is received from an LLM subsystem of the patent analytics and infringement monitoring system 108. For example, where the product / service content item includes an image content item, the image may be transmitted to the LLM subsystem and a text-based caption may be received and stored. Additionally, where the product / service content item includes a text content item, a text translation may be produced by the LLM subsystem based on the language of the claims or based upon a user language preference.

[0126] The web scraper 226 may be used to populate candidate litigation records in the litigation dataset, in order to capture content for the litigation dataset in association with patent litigation. The collected content may be described in FIG. 9. The collected data may be used to generate a vector database, a relational database, a non-relational database, and the like, asdescribed herein, and for measuring the performance of the patent search results.

[0127] The web scraper 226 may be a software system for connecting to remote system 120. The web scraper 226 may be triggered on a regular schedule, or on request of a user using a user interface described herein. The web scraper 226 may further be triggered automatically by a user submitting updated product / service content items to the fine-tuning subsystem based on a candidate patent litigation record in the litigation dataset. The web scraper 226 may be a headless browser-based scraper, such as a scraper based on Chromium.

[0128] The web scraper 226 may further be used to scrape content when the user submits a search query. This may include accessing the search result items identified at a remote device or by a search engine (see e.g., 802 in FIG. 8), collecting the associated content items, and then storing them.

[0129] The web scraper 226 may be used to transmit a content request to a remote system 120, and store the results in the database 224. The content request may be configured based on a user submitted configuration information database 224, including for example, one or more URLs associated with the candidate litigation record. The one or more URLs may include a URL that links to a website, and one or more third-party URLs that are associated with other publications associated with a product or service that is associated with the candidate litigation record. The web scraper 226 may generate a hash value and store it in association with product / service content items. Subsequently, the web scraper 226 may compare a subsequent hash value of the content response with the prior acquired hash to identify if the content has subsequently been updated. The web scraper 226 may capture static content, image or other media content, JavaScript content, cascading style sheet content, or other web assets. The web scraper 226 may further store a screenshot of one or more documents.

[0130] The web scraper 226 may perform post-scraping processing. Upon scraping, processing may include categorizing the content data (e.g., text, images, videos), filtering irrelevant content, translating content, and captioning image or video content, and the like. The web scraper 226 may similarly break the content down into chunks. For example, a large content item may be broken down into a plurality of smaller content items.

[0131] Images and other media collected by the web scraper 226 are stored in the database 224 may include captioning by the LLM system, enabling the media to be searchable and used when generating the vector database for fine-tuning. Similarly, images may be processed using an OCR program to store the text used within them. For example, PDFs may be scraped for their content and may have embedding data created in association with their content.

[0132] The LLM Unit 228 provides integration with the LLM subsystem. This may include a variety of LLMs such as those described above, with reference to FIG.l

[0133] This submitted search query information may be received by the LLM Unit 228 and may be used to produce a prompt for the LLM subsystem. There may be a static portion of the prompt, and a dynamic portion of the prompt.

[0134] The patent document identified by the user in their search query submission may be requested, including claims, drawings, descriptions, abstracts, forward and backward citations, charts, formulae, inventor information, assignee information, classification information, prosecution file wrapper, maintenance payment information, litigation history and the like. The drawings may be submitted using the LLM Unit 228 for captioning by the LLM subsystem so that there is text-based information that summarizes the drawings. This information may be stored in a structured format in the database 224 in association with a search query record for the user’s search.

[0135] The LLM Unit 228 may generate a prompt based on the search query submitted by the user as described in FIG. 8. The prompt may include a static and a dynamic portion. The dynamic portion may be generated using the vector database, and used to fine-tune the LLM subsystem (e.g. based on the vector database generated based on FIG. 6).

[0136] The LLM Unit 228 may connect to different LLM subsystems as described above with reference to FIG. 1.

[0137] The user interface engine 230 is configured to generate interfaces for users to perform patent searches (including, but not limited to patentability, invalidity, freedom-to-operate (FTO, infringement, and others), to review search results including providing search result feedback, to manage configuration of the patent analytics and infringement monitoring system 108, among other possibilities. The various interfaces generated by the user interface engine 230 may be transmitted to a user device by virtue of the Web / API Unit 232 and the network unit 204.

[0138] The processor unit 208 can also execute a user interface (UI) creation engine 230 that is used to generate various UIs for delivery via a web application provided by the Web / API Unit 232, some examples of which are shown and described herein, such as interfaces shown in FIGS. 10 and 11.

[0139] The Web / API Unit 232 may be a web-based application or Application Programming Interface (API) such as a REST (Representational State Transfer) API. The API may communicate in a format such as XML, JSON, or other interchange format.

[0140] The Web / API Unit 232 may receive a user request for a patent search or other functions of the system. The Web / API Unit 232 may apply methods herein to provide the search results in a user interface as provided herein (for example, FIG. 11).

[0141] Referring next to FIG. 3, there is shown a high-level method diagram 300 for servicing a user search query in accordance with one or more embodiments. The details of the inputs, search universe, and outputs of each of the different types of searches is summarized in FIG. 4.

[0142] At 302, a user provides the patent document identifier (e.g., patent number, patent application publication number) for searching to the patent analytics and infringement monitoring system 108. For example, the patent document identifier can be provided by the user via the computing device 106. This could include the user entering the patent document identifier of a patent of interest into a graphical user interface such as FIGS. 10-10B. In some embodiments, the user provides more than one patent document identifier. For example, the user may wish to identify products or services relevant to multiple patent documents. In such cases, the patent analytics and infringement monitoring system 108 can receive multiple patent document identifiers and conduct separate searches for each patent document and / or patent claim. In some embodiments, patent analytics and infringement monitoring system 108 can receive a selection of one or more claims of the patent document associated with the patent document identifier. For example, the user may wish to only assess specific claims of a patent document.

[0143] In some embodiments, in addition to the patent document identifier, the user can input the text of a claim of the patent of interest, or the text of a claim of an application of interest. In some embodiments, the user can input the text of a mock claim. For example, a patent attorney or other patent professional may wish to use the patent analytics and infringement monitoring system 108 as a claim drafting and enforcement strategy tool, in order to verify the content and scope of a proposed claim as against target products or services identified by the system 108. In this manner, the language of a claim may be adjusted prior to the filing of an application or during its prosecution in order to take into account potential infringement targets. The submission of the patent document identifier triggers data collection surrounding the patent document identified. This information can include additional information associated with the patent document, images, prosecution file history, and the like. The submitted images in the search query may be captioned as described herein, and the captioned text may be used as additional information searched in conjunction with the patent document identifier. In some embodiments, the patent analytics and infringement monitoring system 108 can provide an assignee search tool that enables a user to search for patent documents assigned to a particular assignee. In such embodiments, the patentanalytics and infringement monitoring system 108 can display a list of patent documents assigned to a particular assignee and receive a selection of one or more patent documents.

[0144] At 304, search results are provided to the user based on the query. For example, if the user requests an infringement search, then the search results will include web pages or other publications identified as featuring or describing products or services that potentially infringe the patent of interest. This may include references available on the public internet, or optionally at a remote system 120 at a client venue.

[0145] In the case that a validity search is requested, the search results output may include a listing of publications that are known to have been published prior to the priority date of the user submitted patent document. For example, when a validity search is requested, the patent analytics and infringement monitoring system 108 can determine the priority date of the patent of interest and analyze search results published prior to the priority date of the patent of interest. When an infringement search is requested, the patent analytics and infringement monitoring system 108 can determine the application date or the issue date, depending on the implementation of the patent analytics and infringement monitoring system 108 and analyze search results published after the relevant date.

[0146] The search results provided at 304 may be explained to the user using a request to the LLM subsystem via the LLM unit, as will be explained in further detail with reference to FIGS.11 and 16.

[0147] At 306, a user reviewing the search results may provide user feedback in association with a particular search result in the list. This may be accomplished for example using a thumbs-up and thumbs-down feedback button as described herein.

[0148] At 308, the user may set a periodic trigger that may monitor an infringement search. This may trigger email notifications or application notifications for newly identified search results that may infringe.

[0149] Referring next to FIG. 5, there is shown another high-level method diagram 500 of a method of operating the patent analytics and infringement monitoring system 108, in accordance with one or more embodiments.

[0150] At 502, litigation dataset construction, and vector database creation occurs. This is described in further detail in FIG. 6.

[0151] At 504, the vector database is used to fine-tune the operation of the LLM subsystem by generating dynamic portions of prompt requests to the LLM subsystem. The dynamic portionsof the prompt may provide examples for the LLM subsystem that narrow or focus the LLM subsystem appropriately to provide improved output. This may improve the operation of the LLM subsystem in order to service a user search query 508 for a patent search. This is described in further detail in FIG. 7.

[0152] At 506, the search results are generated and provided to the user. The user may provide feedback to the LLM subsystem, which may in turn be stored in the database. This is described further in FIG. 8.

[0153] Referring next to FIG. 6, there is shown another method diagram 600 for finetuning a machine learning model in accordance with one or more embodiments. At least some of the steps of method 600 can be performed by a processor of the patent analytics and infringement monitoring system 108, for example, processor 112 or a processor of processor unit 208.

[0154] At 602, a litigation dataset is provided in a memory for example the memory unit 210, the dataset comprising a plurality of historical patent litigation records, each patent litigation record comprising at least one patent claim, and a litigation outcome corresponding to each of the at least one patent claim.

[0155] At 604, a product / service content item associated with a candidate patent litigation record in the plurality of historical patent litigation records is received at the processor in communication with the memory.

[0156] At 606, the candidate patent litigation record is updated at the processor based on the product / service content item.

[0157] At 608, a vector database based on the litigation dataset is generated at the processor, the vector database used to fine-tune a machine learning model.

[0158] Optionally, in some embodiments, at 610 a prediction prompt is generated at the processor for an LLM-subsystem based on the vector database.

[0159] Optionally, in some embodiments, at 612, the prediction prompt is sent using a network device in communication with the processor to the LLM subsystem.

[0160] Optionally, in some embodiments, at 614, a prompt response that includes at least one search element is received using the network device.

[0161] Optionally, in some embodiments, a prompt response including at least two search elements is received using the network device.

[0162] In some embodiments, the method 600 involves receiving, at a network device in communication with the processor, a user submitted product / service content item associated witha product or service in the candidate patent litigation record.

[0163] In some embodiments, the content item includes a user-submitted URL content item corresponding to the product or service in the candidate patent litigation record and method 600 further involves sending, using a network device in communication with the processor, a content request, receiving, using the network device, a content response based on the content request, performing web scraping on the content response to identify at least one image content item and at least one text content item, and updating the candidate patent litigation record by storing the at least one image content item and the at least one text content item in association with the candidate patent litigation record. The content request can be based on a URL content item.

[0164] In some embodiments, the content item includes a user-submitted text content item corresponding to the product or service in the candidate patent litigation record, and method 600 includes storing the user-submitted text content item in association with the candidate patent litigation record.

[0165] In some embodiments, method 600 includes sending, using the network device, a captioning request to an LLM-subsystem, receiving, using the network device, a captioning response from the LLM-subsystem based on the captioning request and updating, at the processor, the candidate patent litigation record based on the at least one text-based caption. The captioning request can include the at least one image content item and the captioning response includes at least one text-based caption based on the at least one image content item.

[0166] In some embodiments, method 600 further includes determining, at the processor, a hash value corresponding to the content response, updating, at the processor, the candidate patent litigation record with the hash value; and subsequently sending, using a network device in communication with the processor, a second content request based on the URL content item; receiving, using the network device, a second content response based on the second content request; determining, at the processor, that a second hash value corresponding to the second content response is different from the hash value corresponding to the content response; and updating, at the processor, the candidate patent litigation record based on the second content response.

[0167] In some embodiments, the vector database is generated by generating an embedding for the product / service content item of the candidate patent litigation record.

[0168] In some embodiments, generating the vector database involves generating a plurality of vector embeddings for a corresponding plurality of chunks of the product / service content item of the candidate patent litigation record.

[0169] In some embodiments, the vector embedding is one text-embedding-3-small / large, text-embedding- ada-002, BAAI / llm-embedder and / or BAAI / bge-large-en.

[0170] In some embodiments, each patent litigation record in the dataset includes a corporate identification, an estimated damages quantum or any combination thereof.

[0171] In some embodiments, method 600 involves generating, at the processor, at least two machine learning models based on the vector database and updating, at the processor, each of the at least two search elements based on the at least two machine learning models. The at least two machine learning models can include a ranking model and a classification model, and each updated search element can include a rank and a classification.

[0172] In some embodiments, the vector database may be Pinecone™.

[0173] Referring next to FIG. 7, there is shown another method diagram 700 in accordance with one or more embodiments.

[0174] At 702, the user of the patent analytics and infringement monitoring system 108 may enter a patent document identifier into a user interface such as the one in FIGS. 10A-10B. The identifier may be a patent publication number, an issued patent number, a design patent registration number, a patent application number, among other possibilities. The patent document identifier may be for the US, Canada, Europe, or other patent jurisdictions as known.

[0175] The user submission of the patent document identifier in FIGS. 10A-10B may correspond to a search query submission in the system 108, i.e. the user wants to identify patent analytics, provide an invalidity search, or search for possible infringing products or services of a patent document or a mock claim. Once submitted, the user may select one or more patent claims associated with the patent document identified using the patent document identifier. The user may also separately provide specific information associated with the search query, such as a company name (such as a competitor), additional keywords, additional descriptive terms, geographical information, among other possibilities.

[0176] Alternatively, rather than selecting one or more patent claims, in some embodiments, the user can input the text of one or more mock claims (e.g., draft claims, draft amended claims) as free-form text. The one or more mock claims may be assigned names by the user.

[0177] The user may select what type of search, e.g., validity or infringement searching.

[0178] At 704, analysis of the user-submitted patent document may be performed. This can include using the patent document including the drawings, the specifications, the one or moreclaims. As noted, other information may be used to build an analysis prompt for the LLM Unit 228 to send to the LLM subsystem of the patent analytics and infringement monitoring system 108. This could include incorporating patent data related to the abstract, forward and backward citations, charts, formulae, inventor information, classification information, prosecution file wrapper, assignment, citations, classes, maintenance fee information, remaining patent term, potential users of the system, potential industries, keywords, among other possibilities. The analysis prompt may include a static portion and a dynamic portion. The dynamic portion may include patent data including the claim text of the patent. The static portion may be for example “you are a patent attorney, summarize the above text in 10 words, provide the answer in 4 programmatic JSON file in following structure {"summary": "summarized example"}”. As noted above, the drawings may have a textual caption identified by the LLM Unit 228 using the LLM subsystem so that there is text-based information that summarizes the drawings. This information may be stored in a structured format in the database 224 in associate with a search query record for the user’s search.

[0179] The fine-tuning of the LLM-subsystem may involve submitting matching contents of the vector database during the generation of the analysis prompt. The matching contents identified in the vector database may include records from the litigation dataset. The matching may rely on similarity matching within the litigation dataset, and the results may include litigation records that are similar to the user submitted query. These matching records may be included in the analysis prompt that is sent to the LLM subsystem as examples.

[0180] Citation data may be further evaluated and incorporated into the analysis prompt. This could include separate patent data associated with the cited documents.

[0181] The prompt may provide for keywords to be extracted by the LLM subsystem.

[0182] At 706, a prompt is generated by the LLM Unit 228 for the LLM subsystem. This includes sending a request to the LLM subsystem using the analysis prompt from 704 to generate queries for searching. Two approaches may be used. The first may be a heuristic approach that builds search queries using specific rules including extracted keywords. The search queries may be created for use with Google Search, Bing Search, or may be created to run against the remote devices at the user venue.

[0183] The second approach includes LLM subsystem-generated search queries for searching based on the instructions in the analysis prompt.

[0184] Referring next to FIG. 8, there is shown another method diagram 800 of using the patent analytics and infringement monitoring system 108, in accordance with one or moreembodiments. At least some of the steps of method 800 may be performed by a processor of the patent analytics and infringement monitoring system 108, for example, processor 112 or a processor of the processing unit 208.

[0185] At 802, the search queries generated at 706 are received and executed against a search engine (either a public one like Google™, or alternatively, against a private search index accessible on a remote device at a client venue).

[0186] The results from the search may be accessed by the web scraper 226 (e.g., FIG. 2), and the contents of each search result item may be collected and stored in the database. The image content of each URL associated with the search items may be captioned by the LLM Unit 228 as described herein, or translated as described herein.

[0187] The results from the search may be fdtered to remove advertising, non-commercial links, blacklisted domains, user-chosen sites or categories of links.

[0188] At 804, ranking of the search results may be performed using a machine learning model. The ranking can include, for each search result item, using a machine learning model with input from the patent claim and supporting information for product / service content items, identifying and ranking within the set of search results to enable a ranked list to be generated.

[0189] At 806, classification of the search results may be performed using a machine learning model. The classifying step may classify each search result into different entities or categories.

[0190] At 808, the search results from classifying step 806 and the ranking step 804 are collected and combined.

[0191] At 810, different actions may be offered to the user on the search results page, including feedback buttons.

[0192] At 812, users may select or click the feedback buttons, for example, thumbs-up and thumbs-down buttons in the search results or a classification of the search results (e.g., high relevance, medium relevance, low relevance). This selection sends a request back to the server with the associated feedback and a search result identified. The request, including the feedback, the analysis prompt, the patent claims, and the particular selected search result may be stored in the feedback dataset in the database

[0193] Referring next to FIG. 9, there is shown a data diagram 900 in accordance with one or more embodiments. FIG. 9 shows the litigation dataset. FIG. 9 may be a ground-truth dataset for patent litigation outcomes.

[0194] The litigation dataset may include both positive (where infringement was found by a decision maker such as a judge or jury) and negative examples (where infringement was not found by a decision maker such as a judge or jury).

[0195] Column 902 may be provided in the litigation dataset, including the year of a patent litigation record.

[0196] Column 904 may be a docket citation or case title of the court of the patent litigation record.

[0197] Column 906 may be a case title or style of cause including the party names of the patent litigation record.

[0198] Column 908 may be the plaintiff party of the patent litigation record.

[0199] Column 910 may be the defendant party of the of the patent litigation record.

[0200] Column 912 may identify whether the patent litigation record includes a finding of contributory infringement.

[0201] Column 914 may associate the patent litigation record with a patent document identifier.

[0202] Columns 916, 920 and 924 may be the first, second, and third product / service content items (in this case, three URL content items).

[0203] Column 918, 922 and 926 may be corresponding sources of the product / service content items 916, 920 and 924 respectively.

[0204] Other fields in the litigation dataset can include a quantum of damages, patent claims, and individual decision result for infringement of each of the corresponding claims. Separately, data and content from Markman hearings may be provided as well.

[0205] Referring next to FIG. 10A, there is shown a user interface 1000A in accordance with one or more embodiments.

[0206] A user may access the user interface 1000A either using a browser to connect to the server 106, or through an application running on a user device.

[0207] The user enters a patent document identifier in the text input 1002 (for example, US patent US1234567B2), and then may submit the search query by hitting enter or by clicking the search button. Alternatively, referring to FIG. 10B, which shows a user interface 1000B in accordance with one or more embodiments, the patent document identifier may correspond to a patent application publication number and the user may enter the text of a mock claim.

[0208] Referring next to FIG. 11, there is shown another screenshot of a graphical user interface diagram 1100 of a system for monitoring patent infringement, in accordance with one or more embodiments.

[0209] Upon submitting the search query in FIG. S 10A, 10B, the patent search system may return search results 1102a- 1102f. For example, the search query US1234567B2 may return search results 1102a through 1102f as well as others. The search results may be websites, including public websites, or websites hosted on a remote device at a client venue.

[0210] Each search result 1102a-l 102f may be presented with feedback buttons or controls that can allow a user to rate the results. For example, an upvote (e.g., thumbs-up) 1106 and a downvote (e.g., thumbs-down) 1104 button may be displayed. The user may click on, or select these buttons in order to generate a request back to the server 106. The feedback request may be stored in the feedback dataset in the database. The ratings can be used to train one or more of the ranking refinement models, as will be explained further below. FIG. 21 shows an alternative embodiment of a graphical user interface of the patent analytics and infringement monitoring system 108, showing feedback buttons.

[0211] Referring next to FIG. 12, there is shown another a flowchart of a method 1200 for providing updated search results by fine-tuning of machine learning models using a feedback dataset, in accordance with one or more embodiments. The method 1200 may be performed by a processor of the patent analytics and infringement monitoring system 108, for example, processor 112 or a processor of the processing unit 208.

[0212] At 1202, a search query comprising at least one of a patent document identifier or a product or service identifier is received at a network device from a user device.

[0213] At 1204, a search response user interface comprising a search identifier and at least one search result item is transmitted from the network device to the user device.

[0214] At 1206, a feedback item corresponding to a candidate search item in the at least one search result item is received at the network device from the user device.

[0215] At 1208, a feedback item record in a feedback dataset is stored in a memory, the feedback item record stored in association with the search identifier, the feedback item record comprising the feedback item, at least one of a patent claim associated with the patent document identifier or a product or service description, and the search result item.

[0216] At 1210, at least one update search result item is transmitted from the network device to the user device.

[0217] In some embodiments, method 1200 can involve extracting, at the processor patent information associated with the patent document identifier; and generating, at the processor, an initial analysis prompt based on the patent information.

[0218] In some embodiments, method 1200 can involve extracting, at the processor, product or service information associated with the product or service description; and generating, at the processor, an initial analysis prompt based on the product or service information.

[0219] In some embodiments, the patent information associated with the patent document identifier includes: claim text, drawing images, caption text for the drawing images, description text, abstract text, forward or backward citations, each citation including a patent document identifier, chart images, caption text for the chart images, formulae text, inventor information; assignee information, classification information, prosecution file wrapper text, maintenance payment information, or litigation history text or any combination thereof.

[0220] In some embodiments, the product or service information associated with the product or service identifier may include a text snippet, a URL, an icon, an image, a citation or any combination thereof.

[0221] In some embodiments, method 1200 involves transmitting, from the network device to an LLM subsystem, an initial analysis request comprising the initial analysis prompt; receiving, from the network device from the LLM subsystem, an initial analysis response based on the initial analysis request, transmitting, from the network device to a remote device, the at least one remote device search query; receiving, from the network device to the remote device, at least one remote device search response; and transmitting, from the network device to the user device, the at least one search result item based on the at least one remote device search response. The initial analysis response includes at least one remote device search query.

[0222] In some embodiments, the method 1200 may further include: generating, at the processor, a subsequent analysis prompt based on the initial analysis prompt and the feedback item record, transmitting, from the network device to an LLM system, a subsequent analysis request comprising the subsequent analysis prompt, receiving, from the network device from the LLM subsystem, a subsequent analysis response based on the subsequent analysis request, transmitting, from the network device to the remote device search provider, the at least one subsequent remote device search query, receiving, from the network device to the remote device search provider, at least one subsequent remote device search response, and transmitting, from the network device to the user device, at least one subsequent search result item based on the at least one subsequent remote device search response. The subsequent analysis response can include at least onesubsequent remote device search query.

[0223] In some embodiments, the feedback includes a rating corresponding to a feedback control selected by the user.

[0224] In some embodiments, the feedback control includes two or more button controls, a radio button control, a slider control or a star selector control.

[0225] In some embodiments, method 1200 includes: providing, in a memory such as memory unit 210, the feedback item dataset comprising a plurality of historical feedback records. Each feedback item record can be stored association with a corresponding search identifier and the feedback item record can include a historical feedback item, at least one of a patent claim associated with a corresponding patent document identifier or a product or service description, and a corresponding search result item.

[0226] In some embodiments, the method 1200 may further include: generating, at the processor, at least two machine learning models based on the feedback dataset and updating, at the processor, each of the at least one search result item based on the at least two machine learning models. The at least two machine learning models can include a ranking model and a classification model and each of the at least one search result item can include a rank and a classification.

[0227] Referring next to FIG. 13, there is shown another data diagram 1300 in accordance with one or more embodiments. The feedback requests may be stored in the feedback dataset that is stored in the database. Each row or record may identify a single feedback item, including a search identifier field 1302 (which may be a hash or other identifier associated with a particular search submitted by a user), claim index 1304 (which may correspond to a particular claim of a patent document identified by a user in a search query), a feedback identifier 1306 (which may uniquely identify the feedback that was submitted), and the rating provided by the user in the search (for example, 1 may correspond to a thumbs up, and -1 may correspond to a thumbs down feedback control submission). Various ratings may be stored in association with the different types of controls, for example, 0 to 5 may be stored for a star rating feedback control showing between 0 and 5 star buttons.

[0228] Referring next to FIG. 14, shown therein is a flowchart of a method 1400 for identifying products or services that may infringe a patent claim. As explained above, the disclosed systems and methods can identify products or services that may infringe patent claims, including mock patent claims, and determine a relevancy score that is indicative of the similarity between patent claims and a product or service, as described by the product or service description. The method 1400 can be performed by a processor of the patent analytics and infringement monitoringsystem 108, such as processor 112 or a processor of the processing unit 208. In at least one embodiment, method 1400 can be performed to identify products or services that may invalidate a patent. As explained above, the patent analytics and infringement monitoring system 108 can be used to assess whether a patent may be challenged for invalidity.

[0229] At 1410, the processor 112 receives a patent document identifier. The patent document identifier can be a patent number. The patent number can for example, be inputted by a user via a graphical user interface of the computing device 106 and transmitted to the processor 112 via the network 104. In at least one embodiment, the processor 112 receives an application number or an application publication number instead of a patent number. The processor 112 can retrieve the patent document from a database such as the external data storage 102.

[0230] The processor 112 can obtain a target patent claim. In at least one embodiment, as explained above, the processor 112 receives the text of a claim in free-form text. For example, the processor 112 can be configured to receive a natural language input. The text of the claim can be the text of a claim being drafted by a patent practitioner and that the patent practitioner seeks to evaluate, or the text of a claim of an issued patent or pending patent application. Alternatively, the processor 112 can retrieve one or more claims from the patent document identified by the patent document identifier.

[0231] In at least one embodiment, the processor 112 is configured to receive one or more search parameters from the computing device 106. The search parameters can include but are not limited to, a domain, a jurisdiction, a type of file, keywords, competitor names, a date range, an assignee and / or a selected patent claim. For example, the processor 112 can receive a selection of a patent claim of a patent document. For example, the user can specify a specific claim when inputting the patent identifier or provide the text of a claim.

[0232] At 1420, the processor 112 parses the contents of the patent document associated with the patent document identifier to extract information about the patent document. The processor 112 can extract the text, the formulas and the drawings of the patent document and extract bibliographic information about the patent document (e.g., classification, citations, assignee). Based on the contents of the patent document, the processor 112 can determine potential applications of a product or service targeted by the patent claim, product / service industries and / or potential users of a product / service targeted by the patent and / or the processor 112 can extract keywords and entities from the claims and / or the description, extract a textual representation of the unique features and / or concepts of the patent. The processor 112 can generate a claim summary of the claims of a patent document when the patent document contains more than one selected claim.

[0233] At 1430, the processor 112 generates a set of search queries for locating products or services that may infringe the patent claim. The processor 112 can generate a set of search queries for each claim of the patent or for each independent claim of the patent. The processor 112 can generate the set of search queries based on the information extracted at 1420 and in some cases, the search parameters received at 210. The processor 112 can employ a trained machinelearning model such as a large language model (LLM) to generate search queries. In at least one embodiment, the processor generates a guidance prompt for the LLM.

[0234] Alternatively, or in addition thereto, the processor 112 can generate the search queries according to predefined search query rules, using the information extracted at 1420 and in some cases, the search parameters received at 1410. For example, predefined search query rules can be defined for various keywords, industries, competitors, product or service or method names if applicable, and / or images. The storage component 110 and / or the external data storage 102 can store predefined search query rules relating to the information extracted.

[0235] At 1440, the processor 112 searches the Web for websites containing product or service descriptions, using the set of search queries generated at 1430. In at least one embodiment, prior to searching the Web, the processor 112 can modify the format of the search queries according to the search engine used. The processor 112 can execute the searches in parallel, using the set of search queries.

[0236] In at least one embodiment, when searching the Web, the processor 112 can apply one or more filters. For example, the processor 112 can apply the search parameters received at 1410 and / or exclude web domains that have been determined to be unlikely to yield useful results. Alternatively, the processor 112 can filter the search results to exclude search results that have been determined to be unlikely to pose a risk of infringement. For example, the processor 112 can exclude non-commercial, academic websites and / or predetermined domains.

[0237] At 1450, the processor 112 scores and ranks the search results located at 1440, as will be described in further detail with reference to FIG. 16.

[0238] Referring next to FIG.15, shown therein is a flowchart of a method 1500 for identifying one or more patents that may be infringed by a product or service. The method 1500 can be performed by the processor 112 of the patent analytics and infringement monitoring system 108. The method 1500 can mirror the method 1400.

[0239] At 1510, the processor 112 receives a product or service description. Similar to 1410, the product or service description can be inputted by a user via a graphical user interface of the computing device 106 and transmitted to the processor 112 via the network 104. The processor112 can parse the product or service description and extract information from the product or service description, such as keywords, industries, competitors, product / service names and / or images.

[0240] At 1520, the processor 112 searches a database of patents, such as the external data storage 102 using the product or service description received at 1510. For example, the processor 112 can search the database using the extracted information. In at least one embodiment, the database of patents is a database of vector embeddings of claims, as will be explained in further detail with reference to FIG. 16.

[0241] At 1530, the processor 112 scores and rank the results located, as will be described in further detail with reference to FIG. 16.

[0242] Referring next to FIG. 16, shown therein is a flowchart of a method 1600 for ranking search results obtained by a product / service-patent evaluation subsystem of the patent analytics and infringement monitoring system 108. The method 1600 can be implemented by a processor, such as the processor 112 of the patent analytics and infringement monitoring system 108 or by a processor of a separate system for ranking search results that receives search results located by the patent analytics and infringement monitoring system 108. Method 1600 can be implemented at step 1450 of method 1400 or step 1530 of method 1500.

[0243] At 1610, the processor receives the search results. The search results can include patents, patent claims, URLs of product or service websites, or contents of product or service websites, depending on the implementation of the patent analytics and infringement monitoring system 108. In embodiments where URLs of product or service websites are received, the processor can scrape the contents of the product or service websites. Similarly, in embodiments where patents are received, the processor can parse the patents to extract the claims. Alternatively, in some embodiments, the processor can search a database of vector embeddings of claims, for example storage component 110 or databases 224. In such embodiments, prior to method 1600, the processor can parse patent documents stored in a database such as the external data storage 120 and transform each patent claim of the patent documents stored in the database into its vector embedding.

[0244] At 1620, the processor compares each located product / service-patent pair. In at least one embodiment, the processor determines a relevancy score for each located product / service-patent pair. For example, in embodiments where the patent analytics and infringement monitoring system 108 is used to locate products or services that may infringe a patent, the processor determines a relevancy score for each of the located products or services. The relevancy score can correspond to a measure of similarity between the located products or servicesand a claim of the patent of interest or a mock claim, and can be indicative of a risk of infringement. As another example, in embodiments where the patent analytics and infringement monitoring system 108 is used to locate patents that may be infringed by a product or service, the processor determines a relevancy score for each claim of the patents located. The relevancy scores can be used by the processor to rank the search results. For example, a relevancy score approaching 1 can correspond to a highly relevant search result while a relevancy score approaching 0 can correspond to a search result that is unlikely to be relevant. The processor can rank the search results according to the relevancy scores. In embodiments where the patent analytics and infringement monitoring system 108 is used to locate patents that may be infringed by a product or service and the patents have multiple claims, the processor can rank each patent according to the relevancy score of the claim having the highest relevancy score.

[0245] The processor can determine the relevancy score by comparing vector embeddings. The processor can transform the claims of each of the patents and the product / service information derived from the product or service websites into vectors that represent the semantics of the claims and the contents of the product or service websites. The processor can use any vector embedding model known to those skilled in the art, including, but not limited to text-embedding-3 -small / large, text-embedding-ada-002, BAAI / llm-embedder, and BAAI / bge-large-en.

[0246] The processor can embed each independent claim of a patent document into a vector representation. Alternatively, the processor can embed each independent claim and its associated dependent claims into a vector representation. In at least one embodiment, the processor embeds each independent claim, its dependent claims and portions of the description and optionally, equations, chemical formulations, sequence listings, tables and / or figures corresponding to the claims as a vector. For example, the processor can process the patent, including the description, equations, chemical formulations, sequence listing tables (if applicable), and figures of the patent to identify portions of the patent that correspond to the claims so that the vector representation of the claims can include information for interpreting the language of the claims. The processor can employ one or more models for converting non-textual information into textual data, for example visual transformer models and other large language models and for identifying portions of the patent that correspond to the claims.

[0247] The vector embeddings are then stored in a database such as the storage component 110 or databases 224, where each index represents a different product or service or patent claim, depending on the implementation of the patent analytics and infringement monitoring system 108. In embodiments where the patent analytics and infringement monitoring system 108 identifies products or services that may infringe a claim of a patent or that may be relevant to a mock claim,the processor compares the vector embedding corresponding to the claim with the vector embeddings of each of the products or services, stored in the database. In embodiments where the patent analytics and infringement monitoring system 108 identifies patent claims that may be infringed by a product or service, the processor compares the vector embedding of the product or service description with the vector embeddings of each of the patent claims, stored in the database. To compare the vectors, the processor can compute the distance between the vectors. For example, the processor can compute the cosine distance, the Euclidian distance, the Hamming distance, or any other measure of distance between vectors.

[0248] In at least one embodiment, the processor identifies the K most relevant results, based on the relevancy scores, where K is a predetermined number.

[0249] At 1630, the processor applies one or more ranking refinement models to obtain refined relevancy scores. In embodiments, where the processor identifies the K best results, the processor can apply the one or more ranking refinement to the K best results identified at 1620. By only applying the ranking refinement models to a subset of the search results, the processor can avoid using processing resources to evaluate search results that are unlikely to be relevant. At least some of the ranking refinement models can be models that require more computing resources than vector comparison and accordingly, by applying the ranking refinement models to search results that have been pre-ranked through vector comparison, the processor can save processing resources when compared to applying the ranking refinement models without first pre-ranking the search results. By refining the relevancy scores of the search results, the accuracy of the ranking of the search results can be improved.

[0250] In at least one embodiment, the one or more ranking refinement models are machine-learning models that are trained to evaluate similarities between product or service information and patent claims. For example, one or more ranking refinement model can be a text classification machine-learning model.

[0251] In at least one embodiment, at least one of the ranking refinement models is a transformer model, such as, but not limited to, a BERT classifier, which are particularly effective for tasks that require understanding the relationship between two pieces of text, such as text entailment or paraphrase identification. The BERT classifier can be fine-tuned on a dataset containing text-summary pairs or claim-description pairs and trained to score the pairs based on their similarity. Alternatively, or in addition thereto, at least one of the ranking refinement models is an embedding model, such as a BGE model, which are well-suited for retrieval tasks and text matching. An embedding model can generate embeddings for product or service descriptions and patent claims and then rank the descriptions based on their similarity to the claim embeddings. Theranking refinement models can determine a qualitative or quantitative measure of similarity between the text of a product or service description (e.g., an inputted product or service description, a product or service description extracted from a website) and the text of a claim. In at least one embodiment, the ranking refinement models can determine the measure of similarity based additionally on the information extracted from the patent, for example, drawings, portions of the description, formulas, equations and / or tables where applicable. In at least one embodiment, the drawings can be converted to a textual representation and the ranking refinement models can compare the textual representation of the drawings with the text of the product or service description. Alternatively, the ranking refinement models can be models that can compare drawings with images of a product or service (e.g., an image on a website).

[0252] In at least one embodiment, at least one of the ranking refinement models is a LLM that determines a measure of similarity between product or service information and patent claims based on the patent claims, the product or service information and guidance prompts generated by the processor, that provide guidelines for the ranking refinement model. For example, the guidance prompts can configure the ranking refinement model to consider particular keywords, based on the subject matter of the patent, as determined by the patent analytics and infringement monitoring system 108. As another example, the guidance prompts can configure the ranking refinement model to favor search results associated with a particular industry or associated with particular companies. In at least one embodiment, the processor can generate the guidance prompts based in part on the search parameters received at 1410. In at least one embodiment, the guidance prompts can include feedback received from the user, as will be explained in further detail below.

[0253] In at least one embodiment, the processor selects the one or more ranking refinement models to be applied based on one or more factors, including but not limited to, the subject matter of the patent or of the mock claim as determined by the patent analytics and infringement monitoring system 108, the patent classifications of the patent and the confidence level of the ranking refinement models.

[0254] In at least one embodiment, the processor applies two or more ranking refinement models to the search results, according to weights determined by pre-defined rules, for example based on the subject matter of the patent as determined by the patent analytics and infringement monitoring system 108 and the patent classifications of the patent. For example, it may be determined that a first ranking refinement model is more accurate than a second ranking refinement model for determining similarities between product or service information and patent claims when the subject matter relates to a software. In such a case, the first ranking refinement model may be assigned a higher weight than the second ranking refinement model. In at least one embodiment,the weights are predetermined. For example, prior to applying the one or more refinement models, the processor can assign weights to the ranking refinement models and compare results obtained by applying the ranking refinement models to a test set against a ground truth and iteratively modifying the weights.

[0255] In at least one embodiment, the processor determines a refined relevancy score. The refined relevancy score can correspond to a score determined by the one or more ranking refinement models. In embodiments where more than one ranking refinement model is applied, the refined relevancy score can correspond to an average or a weighted average of the scores determined by the ranking refinement models.

[0256] At 1640, the processor displays the ranked results on a graphical user interface. For example, as shown in FIG. 11, which shows an example graphical user interface 1100, the located products 1102a- 1102f that may infringe the patent can be displayed, along with the relevancy score 1140. The score 1140 can correspond to the score determined at 1620 or the refined relevancy score determined at 1630. The ranked results can be stored in a database, such as the storage component 110, databases 224 or the external data storage 102. Although FIG. 11 shows located products 1102a-l 102f, it will be understood that a similar graphical user interface may be used to display located services that may infringe a patent. When more than one patent claim is analyzed by the patent analytics and infringement monitoring system 108, the results associated with each patent claim can be displayed as separate tabs (not shown) and the graphical user interface can include an interactable graphical element that enables a user to toggle between the tabs.

[0257] In at least one embodiment, the patent analytics and infringement monitoring system 108 can display a reason for the ranking and / or for the search results. For example, the GUI can include an interactable element that can enable the user to obtain an explanation of the patent-product / service evaluation subsystem’s ranking of the results and / or an explanation explaining the reasons for the patent analytics and infringement monitoring system 108 identification of the results as relevant. The explanation can include the similarities between the patent claims and the product or service, references to specific claim elements, a website reference of the product or service, an image on the website, a description on the website, a specific webpage of a website, among other possibilities. In at least one embodiment, the patent analytics and infringement monitoring system 108 can generate an intelligence report that can be viewed and optionally, downloaded by the end-user of the patent analytics and infringement monitoring system 108. In some embodiments, for example, the intelligence report may include a claim chart for summarizing results for a target claim (e.g., as described with respect to FIGS. 33-34), among other possibilities.

[0258] The intelligence report can include information about a potentially infringing product or service (e.g., a description of the product or service, technical specifications of the product or technical details about the service, images of the product or images illustrating the service), information about the manufacturer and / or retailer (e.g., address, individuals associated with the business, links to Linkedln® profiles of individuals associated with the business, Securities Exchange Commission filings, business size, financial information, public relations extracts and summaries, links to products or service offerings, litigation data if applicable and available indicating whether a company appeared as a defendant or plaintiff, company attorneys, competitors, information about competitors, among other possibilities.). In at least one embodiment, the intelligence report can be compiled and prepared by an external system, for example, an artificial intelligence (Al) system that can search publicly available sources (e.g., the Web) and retrieve information about the product / service and / or the manufacturer / service provider and that can prepare a report based on the information retrieved. For example, the external system can be GPT Researcher. The patent analytics and infringement monitoring system 108 can transmit, to the external system, the name of the product / service and / or the name of the manufacturer or retailer selling the product and the external system can retrieve information about the product / service and / or the manufacturer / retailer / service provider.

[0259] Although in the example shown in FIG. 11, the relevancy score is shown as a score between 0 and 1, it will be understood that the relevancy score can be presented in other manners. For example, the relevancy score can be expressed as a percentage. As other non-limiting examples, the relevancy score can be presented as a level of relevancy on a scale or as a color gradient.

[0260] In at least one embodiment, as shown in FIG. 11, the results displayed can be rated by a user via feedback buttons 1104-1106. The ratings can be used to improve the search results. For example, the ratings can be used to train one or more of the ranking refinement models. For example, the processor can generate the guidance prompts based in part on the ratings. As another example, the feedback can be used to generate the feedback dataset used for fine-tuning the LLM subsystem, as described above.

[0261] Reference is next made to FIGS. 17-18, which illustrate another method of identifying products or services that may infringe a target patent claim. The target patent claim can be an issued claim, or a patent application claim, including a claim being drafted or amended by a patent practitioner and that could be infringed if the patent claim was to issue. The method involves receiving, from a computing device associated with a user, patent document identifier (a patent number, a patent application number, patent publication number, or utility model number);obtaining the target patent claim, fetching (retrieving) predetermined patent document details associated with the target patent claim, using the patent document identifier; generating a set of queries based on the fetched (retrieved) patent document details; using the set of queries to search commercially relevant resources for identifying products or services each associated with the query set; scoring the search results to provide a relevancy score; and if the relevancy score is over a predetermined value, generating a ranked list of the products, methods or services for the user; else providing feedback to the computing device on the lack of relevant scores.

[0262] The patent document details fetched can be, for example, those illustrated in FIG.19B namely, but not limited to, inventor(s) name(s), class, industry-specific terms, cited documents, and the like. In an exemplary implementation, the user can input a patent number using an input device in communication with the computing device 106. These can be, for example any computer input devices that allow a user to input data into a computer. For example, a keyboard having a set of keys used to input text and commands into a computer, a mouse used to point, click, and select items on the screen, a touchpad used to point, click, and select items on the screen. In general, use of the term “input device” is intended to include all possible types of devices and mechanisms for inputting information to computer system. In some embodiments, the patent document identifier can be received by the patent analytics and infringement monitoring system 108, as a digital image of the patent document and the patent analytics and infringement monitoring system can use techniques such as optical character recognition (OCR) to extract the patent document identifier from the digital image of the document.

[0263] Fetching the data (see e.g., FIG. 19A) whether related to the patents, patent applications, patent publications or utility models, or related to products or services involves executing a fetch request. The term “fetch request” comprises any selection received from the computing device 106 and can be associated with any action taken by the user that results in the computing device 106 performing a selection.

[0264] An example of a pseudo code for such a fetch request is provided below:# import necessary librariesimport requestsimport j son# function to fetch patent detailsdef I etch paten t detai I s(paten t number) :# construct the URL for the USPTO APIurl = "https: / / patent-api.uspto.gov / patent / application?patent_number={}".format(patent_number)# send a GET request to the USPTO APIresponse = requests, get(url)# check if the request was successfulif response. status_code == 200:# parse the JSON responsedata =json.loads(response. text)# extract the patent detailstitle = data["title"]abstract = data[" abstract"]assignee = data["assignee"]inventor = data [“first name inventor”]patent type = data["patent_type"]# print the patent detailsprint("Title:", title)print(" Abstract:", abstract)print(" Assignee:", assignee)print(“first name inventor:”, first name inventor)print("Type:", patent type)else:# print error messageprintf'Error:", response. status_code)# example usage of the functionfetch_patent_details(" US 1234567 ")

[0265] The structure of the Json response may vary depending on the API endpoint used, and the token (or other authentication methods) used. Upon receiving the patent document identifier, the patent analytics and infringement monitoring system 108 retrieves the corresponding patent document (patent application publication, patent publication) details from databases 1702, e.g., by opening a connection to the databases 1702 and using the patent document identifier toretrieve the information, following which the connection is terminated and the predetermined patent document details are retrieved. The databases may be remote databases such as databases of the external data storage 102. In an exemplary embodiment, depending on the input data, namely patent number, or application number or a publication number, the retrieved data may be adjusted based on a menu displayed on a graphical user interface of the patent analytics and infringement monitoring system 108 and provided to the user.

[0266] Generating the set of queries (see e.g., FIG. 19B), can include sorting the retrieved data for common phrases whether related to the patents, patent applications, patent publications or utility models, or related to products or services, and assembling the relevant search terms or expressions or formulae.

[0267] An exemplary implementation of a pseudo code for generating the search query(s) is provided below:# Input: patent details# Output: set of queriesdef generate_queries(patent_details):# Extract key terms and phrases from patent detailskey terms = extract_key_terms(patent_details)# Generate queries based on key termsqueries = generate_queries_from_terms(key_terms)return queries

[0268] The above pseudo code is also an exemplary implementation of a code within a query-generation module. The set of executable instructions takes in the patent document identifier details as input, extracts key terms and phrases from the patent document details, and uses those key terms to generate queries. The extract_key_terms function can use different techniques such as natural language processing, text mining or machine learning to extract the key terms from the patent (or application / publication or utility model) details. The generate queries from terms function can use the extracted key terms and phrases to generate the queries. Depending on the specific requirements of the system, the code may need to be adjusted to match the specific requirements of the module.

[0269] Following the generation of the set of parameters and creating the query set (see e.g., FIG. 19B), the method further involves searching free and / or commercial resources online. For example, the method can involve searching remote systems 120 and / or databases stored in the external data storage 102. An example of a pseudocode used to affect the search based on the query terms generated is provided below:# import necessary librariesimport requestsfrom bs4 import BeautifulSoup# function to search online sourcesdef search_online_sources(title, abstract, assignee):# create a list to store the search resultssearch_results = []# create a list of online sources to searchsources = ["websitel.com", "website2.com", "website3.com"]# loop through each online sourcefor source in sources:# construct the URL for the searchurl = "https: / / {} / search?q={}+{}+{}".format(source, title, abstract, assignee)# send a GET request to the sourceresponse = requests. get(url)# parse the HTML responsesoup = BeautifulSoup(response.text, "html. parser")# extract the search resultsresults = soup.find_all("div", class_=" search-result")# loop through each resultfor result in results:# extract the product name, description, and linkproduct_name = result.find("h3").textproduct_description = result.find("p").textproduct_link = result.find("a")["href']# add the result to the search results listsearch_results . append( {"name": product_name,"description" : product_description,"link": product link})# return the search resultsreturn search_results# example usage of the functionpatent_details = fetch_patent_details("US1234567")search_results = search_online_sources(patent_details ["title"], patent_details["abstract"], patent_details [" assignee" ])print(search_results)

[0270] In an exemplary implementation, the pathway to search, the structure of the HTML, and the parsing may vary from one website to another requiring adapting the code. Also, specific API calls may be necessary for some websites, and this pseudo-code is focused on web scraping of free web pages. Entering commercial databases may require stored authentication protocols that can be stored in the system’s modules. In an exemplary implementation, the step of searching commercially relevant resources comprises searching online marketplaces, product catalogs, and patent databases.

[0271] In an exemplary implementation, the method involves ranking the retrieved results based on relevancy score assigned to the search results (see e.g., FIG. 19C). An example of a pseudocode to generate the relevancy score is provided (see also, FIGS. 19C-19D), hereinbelow:# Input: search results, patent details# Output: relevancy scoredef score_results(search_results, patent_details):# Initialize relevancy scorerelevancy _score = 0# Iterate through search resultsfor result in search_results:# Compare result to patent detailsscore = compare_result_to_patent(result, patent_details)# Add score to relevancy scorerelevancy _score += score# Normalize relevancy scorerelevancy _score = relevancy _score / len(search_results)return relevancy _score

[0272] The above pseudocode provides an example of how the code associated with the scoring module used in the systems disclosed could work. It takes in the search results and the patent document details as input and iterates through each search result, comparing it to the patent document details using the compare_result_to_patent (etc.) function. The function can use different techniques such as natural language processing, text mining, or machine learning to determine the similarity or relevancy between the search result and the patent details. In certain exemplary implementations, the score is then added to the total relevancy score, and divided by the number of search results to get a normalized score. Depending on the specific requirements of the patent analytics and infringement monitoring system 108, in certain other exemplary implementations, the code is adjusted to match the specific requirements.

[0273] The method also involves providing feedback on the search results, by for example, generating a ranked list of products or services and the like that potentially would infringe the input patent document (e.g., patent, patent application, patent publication, or utility patent) number(s). The ranked list can be provided on a graphical user interface of the patent analytics and infringement monitoring system. For example, as shown in FIGS. 11 and 20, the results canbe shown as a ranked list with relevancy scores. The ranked list can be provided in other formats, such as, but not limited to, a table or graph, and can include interactable graphical user interface elements that can enable the user to fdter or sort the list by various user defined criteria. As shown in FIG. 19E, the feedback may also be used to improve model performance by incorporating feedback into model training, for example, for improving result sets, data extraction processes and data queries, among other possibilities.

[0274] Reference is next made to FIG. 22, which illustrates another method of operating the patent analytics and infringement monitoring system 108 to identify patents that may be infringed by products or services. The method involves: receiving, from a computing device 106 associated with a user, a description of a product or service; extracting predetermined details from the inputted description; generating a set of queries based on the extracted details; searching at least one database for patents or patent applications associated with the query set; scoring the search results to provide a relevancy score; and if the relevancy score is over a predetermined value, generating a ranked list of the patents or patent applications to the user. In an exemplary implementation, the step of detail extraction step includes extracting information such as product or service features, technical specifications, or method parameters / steps.

[0275] The patent analytics and infringement monitoring system 108 can include modules that can carry out the method steps. For example, the system can include an input module for receiving a patent document identifier from a computing device 106, a patent detail fetching module for fetching predetermined patent details; a query generation module for generating a set of queries based on the fetched patent details; a search module for using the set of queries to search commercially relevant resources for products, methods or services associated with the query set; a scoring module for scoring the search results to provide a relevancy score; a ranking module for generating a ranked list of the products, methods or services for the user if the relevancy score is over a predetermined value; and a feedback module.

[0276] As another example, the system can include an input module receiving a description of a product or service; a detail extraction module for extracting predetermined details from the inputted description; a query generation module for generating a set of queries based on the extracted details; a search module for searching at least one database for patents or patent applications associated with the query set; a scoring module for scoring the search results to provide a relevancy score; and a list generation module for generating a ranked list of the patents or patent applications to the user if the relevancy score is over a predetermined value. The computing device can receive an input (e.g., description of a product or service) from a user, via a graphical user interface of the computing device.

[0277] In at least some embodiments, the patent analytics and infringement monitoring system 108 can use at least some of the same modules and databases for identifying patents that may infringe a product or service and for identifying products or services that may infringe claims of a patent.

[0278] In at least one embodiment, the patent detail-fetching module includes a communication interface for connecting to a remote database of patent information, for example, databases of the external data storage 102, the query-generation module includes a natural language processing component for analyzing the fetched patent document details to identify key terms and phrases relevant to the patent document associated with the patent document identifier, the search module includes a web crawling component for online searching of at least one of: marketplaces, product catalogs, service offering catalogs, social media content, litigation history of the patent and patent databases, the scoring module includes a machine learning component that uses a pre-trained model as described herein to determine the relevancy of the search results to the patent document identifier inputted, and the ranking module includes a visualization component that generates a user-friendly interface to present the list of products, methods or services and that includes elements with which a user can interact to filter or sort the list by various selectable criteria, as well as a feedback module that includes an output device (e.g., printer, screen, removable memory device and the like). These components can be used to identify products or services that may infringe a patent and / or to identify patents that may be infringed by a product or service.

[0279] FIG. 23 is a schematic diagram showing components of the patent analytics and infringement monitoring system 108, which may be used to carry out the method of FIG. 22. In examples, a system processor of the patent analytics and infringement monitoring system 108 may receive and compile information from various sources to identify patents that may be infringed by products or services.

[0280] As illustrated in FIG. 24, the patent analytics and infringement monitoring system 108 can include a module that enables the patent analytics and infringement monitoring system 108 to receive feedback. For example, the patent analytics and infringement monitoring system 108 can prompt the user to provide feedback to the patent analytics and infringement monitoring system 108 on the relevancy score, which can enable the patent analytics and infringement monitoring system 108 to modify, for example, the patent terms extracted, the type of product / service data inputted, the weighting and the calculation of the relevancy score. These can be done using machine learning models trained on the search results. The models can be used with a plurality of users and implemented regardless of the inputted search parameters.

[0281] Referring next to FIG. 25, shown therein is a flowchart of a method 2500 for patent portfolio ranking. Reference is simultaneously made to FIG. 26, which shows a schematic representation of method 2500. Patent portfolio ranking involves ranking a collection of patent documents to evaluate their strength, quality, potential and litigation value and can assist a patent owner or patent portfolio manager in identifying patents to maintain, and candidates for licensing or sale. The method 2500 involves ranking patent portfolios based on patent relevance, as described with reference to FIGS. 14, 16, 18-20. The method 2500 can be performed by aprocessor of the patent analytics and infringement monitoring system 108, such as processor 112 or a processor of the processing unit 208.

[0282] At 2510, the method 2500 involves receiving a patent portfolio 2610 that includes two or more patents. The patent portfolio 2610 may be a custom patent portfolio (i.e., a patent portfolio created by a user). For example, the processor 112 can receive patent numbers from a computing device 106. The patent numbers can for example, be inputted by a user via a graphical user interface of the computing device 106 and transmitted to the processor 112 via the network 104.

[0283] Alternatively, or in addition thereto, in at least one embodiment, the processor 112 can receive one or more search terms (e.g., patent number, assignee, inventor, keywords, publication date, priority date) from a computing device 106 and search a database, for example, external data storage 102, to retrieve patents matching the search term(s). In some cases, when multiple patents match the search terms, the processor 112 can then receive a selection from the computing device 106. For example, the processor 112 can display a list of patents and the user associated with computing device 106 can select patents to add to the patent portfolio. The patent analytics and infringement monitoring system 108 can accordingly enable a user to create a custom patent portfolio. FIG. 30 shows a screenshot of an example user interface displaying a ranked patent portfolio. In the example of FIG. 30, a patent can be added to a custom portfolio by clicking on the “Add patent” button 3010.

[0284] Alternatively, in at least one embodiment, the processor 112 can receive a patent portfolio as a file (e.g., pdf, xlsx, etc.) and extract information from the file. For example, a user may upload a patent portfolio to the patent analytics and infringement monitoring system 108.

[0285] At 2520, for each claim of interest of each patent in the patent portfolio received at 2510, the processor 112 identifies products or services that may infringe the patent claim. The claims of interest can be all claims of a patent, the independent claims, or a subset of the claims (e.g., the first independent claim), depending on the implementation of the method 2500. In some embodiments, the processor 112 can receive a selection of the claims of interest from computingdevice 106. For example, the user may be presented with options (e.g., all claims of a patent, the independent claims, a subset of the claims) and select the desired option. The processor 112 can first retrieve the claims of each patent in the patent portfolio. For example, the processor 112 can retrieve the text of the patents identified by the patent numbers received or determined at 2510 from a database and extract the claims of interest.

[0286] The processor 112 can identify products or services that may infringe the patent claim using the methods described with reference to FIGS. 14, 16, 18-20 and 31. As described above, for a given claim, search results corresponding to products or services that may infringe the patent claim can be identified and the processor 112 can score the search results located so that each search result can have a corresponding score (a relevancy score and / or a refined relevancy score). In some embodiments, search results having relevancy scores (or refined relevancy scores) below a predetermined threshold may be removed.

[0287] In some cases, when searching for products or services that may infringe a patent claim, the processor 112 may identify multiple search results (e.g., URLs) describing the same product or service. In some embodiments, the processor 112 can combine search results that correspond to the same products or services to obtain combined search results. Each combined search result can correspond to a unique product or service. In such embodiments, the processor 112 can combine the relevancy scores of duplicate products or services to obtain global relevancy scores so that each product or service is associated with a single relevancy score.

[0288] In some cases, when multiple search results (e.g., URLs) describing the same unique product or service are combined, the combined search results may be provided to an LLM for generating a summary of the unique product or service. Further details about the combination of search results and summary generation are provided below with respect to FIG. 31. For example, a prompt may be generated to instruct the LLM to generate a textual summary or description of the unique product or service, using information contained in the combined search results. In some cases, the combined search results may include textual content, image content, video content, audio content or other information media corresponding to the unique product or service, and the LLM may be instructed to output a LLM-generated textual summary that identifies and / or describes features of the product or service, based on the combined search results. Additionally, or alternatively, the LLM may be instructed to include information about the owner, manufacturer, distributor, importer, exporter, provider or other such market participants in respect of the unique product or service, in the LLM-generated textual summary. Illustratively, the LLM-generated textual summary may effectively consolidate information across disparate search results to generate a more complete and comprehensive understanding of the product or service.

[0289] At 2530, the processor 112 determines a patent score for each patent in the patent portfolio. To determine the patent score of a patent, the processor 112 can first determine the claim score of each claim of interest. For example, in embodiments where only the independent claims are of interest, the processor 112 can determine a claim score for each independent claim.

[0290] The claim score can be an indicator of the number of products or services that may infringe the patent claim and / or the relevance of the products or services (i.e., how similar a product or service is to a claim). The claim score of a given claim can be calculated according to the following equation: c = fi([n, n, ...rn]), where c is the claim score, and n, n, ...rnare the relevancy scores or refined relevancy scores of the search results associated with the claim. The function can be a summation function. For example, the claim score can correspond to a sum of the relevancy scores or refined relevancy scores of the search results (e.g., Claim;Score= Resulti score+ Result2 score +...+ ResultNscore). For example, if a given claim is associated with three search results having relevancy scores or refined relevancy scores 0.9, 0.88, 0.7, respectively, the claim score of the claim can be 0.9+0.88+0.7= 2.48. Alternatively, the function can be an average function, a maximum function, a median function, a geometric / harmonic means function, a top-k averages function, a SoftMax function, a weighted means function, a length-normalized sums function, a coverage ratios function, or any other function that enables a claim score representative of the search results located to be computed.

[0291] In another embodiment, the relevancy scores or refined relevancy scores n, n, ... rnof the individual search results associated with the claim may be replaced by summary relevancy scores rsi, rs2, ...rsn, and the claim score of a given claim can be calculated according to the following equation: c = fi([rsi, rs2, ...rsn]), where c is the claim score, and rsi, rs2, ...rsn are the relevancy scores associated with the LLM-generated summaries of unique products or services associated with the claim. In some embodiments, for example, the summary relevancy scores rsi, rs2, ... rsn may be determined using the method as described with respect to step 1620 of FIG. 16, or another method may be used. For example, the processor can transform the claim language and the LLM-generated summaries into corresponding vectors that represent the semantics of the claim(s) and the LLM-generated summaries, for example, using a vector embedding model, and a similarity score may be determined by comparing the vectors (e.g., by computing a distance between the vectors in a corresponding embedding space, among other possibilities). Illustratively, determining a claim score based on a similarity to the LLM-generated summaries may more accurately detect potential infringing products and / or services because the LLM-generated summaries represent a more comprehensive description of the unique products or services. In situations where multiple features of an infringing product or service (e.g., that map to multipleelements of one claim) are separately described in unique search results, consolidating these search results into a single LLM-generated summary may more effectively capture a relevance of the infringing product or service to a particular claim.

[0292] The patent score of each patent can be determined based on the claim scores of the claims being assessed (i.e., the claims of interest). The patent score of a given patent can be calculated according to the following equation: Patentscore = f2([ci,C2,.., CM]), where ci,C2,.., CM are the claim scores of the claims assessed. In some embodiments, the function is an average function (e.g., Patent Score = Average (Claimi score, C lai rmscore. ... , ClaimN score). For example, a given patent with three assessed claims having claim scores 2.4, 1.7, 2.65 can have a patent score of 2.25. Alternatively, the function can be a summation function, a maximum function, a median function, a geometric / harmonic means function, a top-k averages function, a SoftMax function, a weighted means function, a length-normalized sums function, a coverage ratios function, or any other function that enables a patent score representative of the claim scores to be computed. In some embodiments, each claim can be assigned a weight and the patent score can be determined by weighing the claim scores (e.g., weighted sum, weighted average, among other possibilities.). For example, the patent analytics and infringement monitoring system 108 can determine the breadth of each claim and assign a higher weight to broader claims.

[0293] FIG. 27 shows example claim scores computed for claims of a patent and a patent score for the patent. FIG. 28 shows example combinations of example functions that may be used to compute claim scores and patent scores. It will be appreciated that other functions may be used, depending on the implementation of the patent analytics and infringement monitoring system 108.

[0294] Each patent in the patent portfolio can be assigned a patent score.

[0295] At 2540, the processor 112 ranks the patents of the patent portfolio based in part on the patent scores determined at 2530. To rank the patents, the processor 112 can assign a weight to each patent and compute a weighted patent score for each patent. The weight can be a value between 0 and 1 and can be determined based on various factors including but not limited to, potential financial reward (e.g., license, sale, damages, financial state of the potential infringer), existence of similar defense patents (i.e., patents that may be used to defend against an infringement action), litigation history of the patent, whether the patent is associated with a core product of the patent owner or a secondary product. The weights can be predefined, can vary, or can be defined in part by the use, depending on the implementation of the patent analytics and infringement monitoring system 108. For example, in some embodiments, the user can rank the factors and the processor 112 can assign weights to the patents according to the ranked list of factors.

[0296] The weighted patent score of each patent can be computed by multiplying the weight of the patent with the patent score (i.e., Si= pi x wi, where si is the weighted patent score of patent i, pi is the patent score for patent i, and wi is the weight of patent i). Since 0 < wi< 1, the score Si is 0 < si< pi.

[0297] In some embodiments, the patent scores of the patents of a patent portfolio and the weights can be represented by vectors and a vector of the weight patent scores can be determined by computing the Hadamard product of the vector of patent scores and the vector of weights.

[0298] The patents can then be ranked (e.g., in descending order) according to the weighted patent scores Si, where the highest weighted patent score si represents the patent having the highest litigation value, to obtain a ranked patent portfolio 2620. FIG. 30 shows another exampled ranked patent portfolio 3020.

[0299] FIG. 29 shows a table of ranked patents. As shown, since patent A has a patent score of 1.22, and a weight of 0.9; patent B has a patent score of 0.85 and a weight of 0.3; and patent C has a patent score of 0.71 and a weight of 1.00, the weighted scores of patents A, B and C are 1.1, 0.26 and 0.71, respectively. The patents are accordingly ranked in the following order: Patent A, Patent C, Patent B.

[0300] In some embodiments, the weighted patent scores can be normalized. For example, in some cases, it may be advantageous to normalize the scores to a common scale (e.g., [0,1]) to further analyze the patent scores and / or to improve explainability when the patent scores are displayed.

[0301] In some embodiments, the weighted patent scores are normalized using a max normalization method where the normalized weighted patent scores are computed according to the following equation si' = Si / maxj sj, where Si is the weighted patent score of patent i before normalization and maxj Sj is the highest (i.e., maximum) weighted patent score in the patent portfolio.

[0302] In other embodiments, the weighted patent scores are normalized using a min-max normalization method. The normalized weighted patent score using min-max normalization can be computed according to the following equation Si' = (si - minj Sj) / (maxj Sj - minj Sj) where minj Sj is the lowest (i.e., minimum) weighted patent score in the patent portfolio and and maxj sj is the highest (i.e., maximum) weighted patent score in the patent portfolio.

[0303] In other embodiments, the weighted patent scores are normalized using any of Z-score normalization (i.e., zi= (si - p) / o), percentile rank normalization (i.e., n = rank; ZN), softmax normalization (i.e., pi = exp(siZr) / X exp(sjZr)) and log or root scaling normalization (i.e., Iog(l+s0or si). The normalization method used can vary, depending on the implementation of method 2500.

[0304] At 2550, the processor 112 displays the ranking on a graphical user interface. For example, the graphical user interface can display the ranked list of patents and in some cases, the patent scores of each patent, as shown in FIG. 30.

[0305] In some embodiments, the processor 112 can apply one or more filters to the ranked list. For example, patents or patent scores can be color coded according to the patent scores (e.g., high patent scores can be shown in green, low patent scores can be shown in red).

[0306] Reference is next made to FIGS. 31-32, which illustrate another method of operating the patent analytics and infringement monitoring system 108 to identify products or services that may infringe a target patent claim, in accordance with one or more embodiments. The target patent claim can be an issued claim, or a patent application claim, including a claim being drafted or amended by a patent practitioner and that could be infringed if the patent claim was to issue. At least some of the steps of the method may be performed by a processor of the patent analytics and infringement monitoring system 108, for example, processor 112 or a processor of the processing unit 208. The method involves receiving, from a computing device associated with a user, a patent document identifier (a patent number, a patent application number, patent publication number, or utility model number); obtaining the target patent claim, fetching (retrieving) predetermined patent document details associated with the target patent claim, using the patent document identifier; generating a first set of queries based on the fetched (retrieved) patent document details; using the first set of queries to search commercially relevant resources for identifying products or services each associated with the first query set, thereby producing a first set of returned search results; generating a second set of queries based on the first set of returned search results; using the second set of queries to further search commercially relevant resources for identifying products or services each associated with the first query set, thereby producing a second set of search results; scoring the second set of search results to provide a relevancy score; and if the relevancy score is over a predetermined value, generating a ranked list of the products, methods or services for the user; else providing feedback to the computing device on the lack of relevant scores.

[0307] The patent document details may be fetched and a first query set may be generated according to the process described with respect to FIG. 19B. In examples, the first query set may be executed against a search engine to generate a set of preliminary search results (see e.g., FIG.19C). In some embodiments, the fetching and querying processes may be performed by the LLM subsystem(s), for example, to generate one or more search queries (for example, Google searches,requests to web servers, or requests to user remote devices). See also FIGS. 17-18 and its associated description above regarding the fetching and querying processes.

[0308] In examples, the set of preliminary search results may be processed, for example, the processor 112 can parse the set of preliminary search results and extract information such as keywords, industries, competitors, product / service names and / or images, among other possibilities. In examples, the set of preliminary search results may be analyzed using a probabilistic approach to determine features (e.g., companies, manufacturers, products, services and the like) of the preliminary search results that appear most frequently. In some embodiments, a list of target or high-priority features may be generated, for example, based on the frequency in which the features appear in the preliminary search results. In examples, the processor 112 may also process the initial search results for a purpose of cleaning the data. For example, the identified key targets may be validated to ensure that the features identified as frequently occurring in the preliminary search results represent genuine entities (e.g., companies, products, services and the like). In examples, the processor 112 may further consolidate and / or deduplicate data included in the preliminary search results, for example, to identify search results representing the same key targets and / or the same products or services, among other possibilities. For example, company names may change in the event of an acquisition or merger, or product names may be misspelled, among other possibilities. In examples, the LLM subsystem(s) may perform the data validation and data cleaning processes, or in other examples, regular expressions (REGEX) or string comparisons may be used for cleaning the data, for example, depending on the use case. In examples, the identified key targets may be ranked, for example, to generate a prioritized list of key targets.

[0309] In examples, the prioritized list of key targets may be used to generate a second query set (e.g., according to the process described with respect to FIG. 19B), where the second query set may represent a more focused or refined query set corresponding to the key targets. In examples, the second query set may be executed against a search engine to generate a set of secondary search results (see e.g., FIG. 19C) as previously described. In examples, the set of secondary search results may be processed, for example, the processor 112 can parse the set of secondary search results.

[0310] In examples, the LLM subsystem(s) may receive the set of secondary search results and may generate textual summaries associated with each key target in the prioritized list of key targets (e.g., including potentially infringing products or services). For example, a prompt may be generated by the LLM Unit 228 for the LLM subsystem(s) that includes the set of secondary search results, for instructing the LLM subsystem to output a detailed textual summary that identifiesand / or describes features of a target product or service. Additionally, or alternatively, the LLM may be instructed to include information about the owner, manufacturer, distributor, importer, exporter, provider or other such market participants in respect of the unique product or service, in the LLM-generated textual summary. Illustratively, the LLM-generated textual summary may effectively aggregate information across disparate search results to generate a more complete and comprehensive understanding of identified products or services.

[0311] In examples, the processor 112 may rank and / or categorize data, for example, to identify potentially infringing products or services (see e.g., FIG 19D). In some embodiments, for example, relevancy scores may be generated for each potential infringing product or service, based on the LLM-generated textual summaries. For example, the processor can transform the claim language and each of the LLM-generated summaries into corresponding vectors that represent the semantics of the claim(s) and the LLM-generated summaries, for example, using a vector embedding model. In examples, a relevancy score may be determined by comparing the vectors (e.g., by computing a distance between the vectors in a corresponding embedding space, among other possibilities) for each claim-summary pair. In examples, where the relevancy score is over a predetermined value, a ranked list of the potentially infringing products or services may be provided to the user, among other possibilities. In some examples, the ranked list of potentially infringing products or services may be output in graphical user interface, for example, in a summary view, among other possibilities. In examples, a user may interact with each identified product or service in the ranked list to view the corresponding LLM-generated summary, for example, clicking on an identified product or service in the ranked list may expand a graphical object in the graphical user interface to show the LLM-generated summary in the summary view, among other possibilities.

[0312] In examples, the disclosed solution may provide a technical benefit of more accurately identifying and ranking a list of potential infringers in a more computationally efficient manner. For example, in determining a relevancy score for each identified product or service, generating a single embedding vector for the LLM-generated summary, for the purpose of for performing a similarity comparison between the LLM-generated summary and the target claim, may be more computationally efficient than generating a separate embedding vector for each respective search result and evaluating a relevancy of each search result individually. In this regard, performing only one vector comparison between the LLM-generated summary and the target claim consumes fewer computing resources (e.g., memory, processing power, computing time), particularly when the number of identified products or services in the returned search results is large.

[0313] Reference is next made to FIG. 33 which illustrates another method of operating the patent analytics and infringement monitoring system 108 to generate a claim chart 3400, in accordance with one or more embodiments. In some embodiments, for example, responsive to generating a ranked list of potentially infringing products or services, a claim chart 3400 may be generated for each identified product or service in the ranked list, for example, to further evaluate a relevance of the identified product or service to each element of the target claim. However, unlike a traditional claim chart, which simply maps features of a potentially infringing product or service to elements of a claim, the claim chart 3400 may provide additional analysis, context and reasoning for why certain features of the identified product or service are considered relevant (or not relevant) to each claim element. In this regard, the claim chart 3400 may provide further context to the ranking of identified products or services in the ranked list. At least some of the steps of the method may be performed by a processor of the patent analytics and infringement monitoring system 108, for example, processor 112 or a processor of the processing unit 208. In some embodiments, for example, the claim chart 3400 may be output as part of a report (e.g., an intelligence report, as described with respect to FIG. 16) or as a table within a graphical user interface, among other possibilities.

[0314] In examples, at step 3310, a patent document identifier (e.g., a patent number, a patent application number, patent publication number, or utility model number) including one or more potential target claims may be provided to a user in a graphical user interface. At step 3315, a user may select one of the potential claims as the target claim. In the example provided, there may be n potential target claims, and claim 5 may be selected by the user as the target claim. In other embodiments, the processor of the patent analytics and infringement monitoring system 108 may automatically select the target claim from the pool of available claims, for automatically generating the claim chart 3400. For example, the processor may execute instructions to automatically generate a respective claim chart for each target claim from the pool of available claims.

[0315] In examples, at step 3320, a processor of the patent analytics and infringement monitoring system 108 may extract one or more claim elements from the target claim. For example, claim elements may represent the specific components, features, or steps that define the invention in a patent claim. In the example provided, the selected target claim (e.g., claim 5) includes three claim elements, for example, identified and extracted by the processor of the patent analytics and infringement monitoring system 108 and labeled as claim elements 5.1, 5.2 and 5.3, respectively. In examples, the LLM subsystem(s) may receive the target claim and may extract the claim elements, among other possibilities. In examples, at step 325, the extracted claim elementsmay be stored in a database, such as the storage component 110, databases 224 or the external data storage 102.

[0316] In examples, at step 3340, a summary view of infringement search results may be provided to a user in a graphical user interface. In examples, the summary view may represent a ranked list of identified products or services (e.g., potentially infringing products or services, as described with respect to FIGS. 31-32). At step 3345, a user may select one of the identified product or services in the ranked list as the selected product or service to be evaluated in the claim chart 3400. In the example provided, there may be multiple identified products or services in the ranked list, where each identified product or service represents a unique product or service within a collection of returned search results and includes a corresponding owner, manufacturer, distributor, importer, exporter, provider or other such market participants. In the example provided, product “A” by manufacturer “X” may be selected from the ranked list by the user as the selected product or service to be evaluated in the claim chart 3400. In other embodiments, the processor of the patent analytics and infringement monitoring system 108 may automatically select the product or service from the ranked list, for automatically generating a corresponding claim chart 3400. For example, the processor may execute instructions to automatically generate a respective claim chart for each identified product or service in the ranked list.

[0317] At step 3350, the extracted claim elements and the selected product or service are provided to a processor of the of the patent analytics and infringement monitoring system 108, for example, for performing an Al-assisted infringement analysis of the claim elements and the selected product or service. For example, the LLM subsystem(s) may receive the extracted claim elements and the selected product or service from the ranked list and may execute instructions to compare each claim element to an LLM-generated textual summary for the selected product or service. In some examples, the LLM-generated textual summary may have been previously generated as described with respect to FIGS. 31-32 and stored in a database (such as the storage component 110, databases 224 or the external data storage 102), such that the stored LLM-generated textual summary can be retrieved by the LLM subsy stem(s), among other possibilities. In examples, a prompt may be generated by a processor of the of the patent analytics and infringement monitoring system 108 for instructing the LLM subsy stem(s) to identify a match for the relevant features or characteristics of the selected product or service with each claim element of the target claim. The LLM subsystem(s) may also be instructed to provide a reasoning for the identified match, along with any supporting evidence for the reasoning (e.g., as citations, or links to sources), among other possibilities. In some embodiments, an indication of a closeness of the match (e.g., a level of similarity) between each claim element and the selected product or servicemay also be generated, for example, on a numerical scale or using a label to describe the match, such as “high”, “medium” or “low”, among other possibilities. In examples, the processor 112 may transform the language of each claim element and the LLM-generated summary of the selected product or service into corresponding vectors that represent the semantics of the claim(s) and the LLM-generated summaries, for example, using a vector embedding model, or the processor 112 may retrieve a previously stored vector corresponding to the LLM-generated summary of the selected product or service, among other possibilities. In examples, a similarity score may be determined by comparing the vectors (e.g., by computing a distance between the vectors in a corresponding embedding space, among other possibilities) for each claim element-summary pair and generating a corresponding similarity score, among other possibilities. In examples, the indicator of the closeness of the match (e.g., “high”, “medium”, “low”) may be mapped to a similarity score threshold for the claim element-summary pair.

[0318] At step 3360 a claim chart 3400 may be generated and output to a display via a graphical user interface, for example, for providing the results of the Al-assisted infringement analysis of the claim elements and the selected product or service. FIG. 34 provides an example of a claim chart configuration, however it should be understood that this is only exemplary and is not intended to be limiting. In examples, the claim chart 3400 may include, for each claim element, a corresponding element text, such as a language of the claim element in the target claim (e.g., verbatim from the target claim), among other possibilities. In examples, the claim chart 3400 may also include an indication of the match between each claim element of the target claim and the selected product or service (e.g., indicated as a label such as “high”, “medium”, “low”, or as a numerical score, or another visual indicator may be used, for example, incorporating color as an indicator of the closeness of the match, among other possibilities). In examples, the claim chart 3400 may also include a reason for the closeness of the match for each claim element as construed. In examples, the reason may represent an LLM-generated text that references features of the selected product or service that the LLM subsystem(s) has identified as being relevant to aspects of the claim element, or differences between features of the selected product or service to from the claim element, or missing features, among other possibilities. In examples, the LLM subsystem(s) may include in the reasoning, a description of why certain features of the selected product or service are considered relevant and / or indicate a match to the claim element, or why certain features of the selected product or service are not considered to represent a match with the claim element, among other possibilities. In examples, the claim chart 3400 may further include a listing of sources provided as supporting evidence for the reasoning, for example, as clickable links, among other possibilities.

[0319] In at least one embodiment, the computing device can include a user interface module for providing a user interface for inputting the description of the product, service, apparatus, composition, or method. The patent analytics and infringement monitoring system can include a detail extraction module that includes means for extracting information such as, for example, product or service features, technical specifications, or method parameters can be used in the generation of a ranked list of patent(s), patent application(s) patent publication(s) utility model(s), or design patent(s) relevant to products or services and the like that are potentially infringing the patent.

[0320] In the context of the disclosure, the term “module”, as used herein, means, but is not limited to, a software or hardware component, such as a Field Programmable Gate-Array (FPGA) or Application-Specific Integrated Circuit (ASIC), which performs certain tasks. A module may advantageously be configured to reside on the addressable storage medium and configured to execute on one or more processors. Thus, a module may include, by way of example, components, such as software components, object-oriented software components, class components and task components, processes, functions, attributes, procedures, subroutines, segments of program code, drivers, firmware, microcode, circuitry, data, databases, data structures, tables, arrays, and variables. The functionality provided for in the components and modules may be combined into fewer components and modules or further separated into additional components and modules.

[0321] For example, the module is a self-contained piece of code that performs a specific task or set of tasks. Typically, the code is a separate file that can be imported into other code to be used as a reusable component. The code module can have its own variables, functions, and classes, in certain exemplary implementations, “module” can be a "building block" that can be combined with other modules to create a larger program. In certain exemplary implementations, the modules are also used to encapsulate functionality and data, so that it can be used without exposing its internal details or implementation.

[0322] In an exemplary implementation, provided herein is an article of manufacture for generating a list of products or services potentially infringing a target patent comprising a non-transitory memory device storing thereon a computer-readable media with a set of executable instructions embodied thereon configured, when executed by the at least one computer to: receive a patent number; fetch predetermined patent details; generate a set of queries based on the fetched patent details; using the set of queries to search commercially relevant resources for product, service, apparatus, composition, or method each associated with the query set score the search results to provide a relevancy score; and, if the relevancy score is over a predetermined value,generate a ranked list of the products, methods or services for the user.

[0323] In an exemplary implementation, the set of executable instructions further comprise program code for performing optical character recognition (OCR) on a digital image of the patent document identifier received, as well as program code for connecting to a remote database of patent information and retrieving information specific to the input patent document identifier, program code for analyzing the fetched patent document details to identify key terms and phrases relevant to the patent and generating a set of queries, program code for using a pre-trained machine learning model to determine the relevancy of the search results to the input patent number and generating a ranked list of product, service, apparatus, composition, or method for the user, and program code for presenting the list of product, service, apparatus, composition, or method in a user-friendly format, such as a table or graph, and allowing the user to filter or sort the list by various selectable criteria, such as date, price, or popularity.

[0324] In another exemplary implementation, provided herein is an article of manufacture for generating a list of patents potentially reading on a product, a service, an apparatus, a composition, or a method, comprising a non-transitory memory device, storing thereon a computer-readable media (CRM) with a set of executable instructions that, when executed cause the processor to: receive a description of a product, service, apparatus, composition, or method from a computing device; extract predetermined details from the inputted description; generate a set of queries based on the extracted details; search at least one database for patents or patent applications associated with the query set; score the search results to provide a relevancy score; and if the relevancy score is over a predetermined value, generate a ranked list of the patents or patent applications to the user.

[0325] In yet another exemplary implementation, the CRM further comprises executable instructions for providing a user interface for inputting the description of the product, service, apparatus, composition or method, as well as extracting information such as product or service features, technical specifications, formulae, or method parameters.

[0326] In accordance with a first aspect of this disclosure, there is provided a method for identifying products or services relevant to a target patent claim. The method involves operating at least one processor to: receive a patent document identifier associated with the target patent claim from a computing device in communication with the at least one processor via a network; obtain the target patent claim; retrieve, from a database in communication with the at least one processor via the network, patent document details associated with the target patent claim, using the patent document identifier; generate a set of queries, based on the retrieved patent document details and the target patent claim; search remote databases using the set of queries to identifyproducts or services satisfying the query set in a set of returned search results; generate, by a large language model (LLM), textual summaries respectively corresponding to each of the identified products or services in the set of returned search results; score each of the identified products or services in the set of returned search results, based on the corresponding LLM-generated textual summaries, to provide a respective relevancy score for each identified product or service; generate a ranked list of the products or services having a relevancy score greater than a predetermined threshold; and display the ranked list.

[0327] In an example of the first example aspect, wherein generating a set of queries comprises operating the at least one processor to: generate a first set of queries, based on the retrieved patent document details; search remote databases using the first set of queries to identify products or services satisfying the first set of queries in a first set of returned search results; and generate a second set of queries based on the first set of returned search results.

[0328] In an example of the first example aspect, wherein the method further comprises operating the at least one processor to: analyze the first set of returned search results using a probabilistic approach to determine features of the preliminary search results that appear frequently in the first set of returned search results; validate one or more frequently appearing features in the first set of search results to generate a list of validated key features; and generate the second set of queries, based on the list of validated key features.

[0329] In an example of the first example aspect, wherein validating the frequently appearing features in the first set of search results comprises operating the at least one processor to: refine the first set of search results to include only the one or more frequently appearing features representing a genuine entity; and process the refined first set of search results to consolidate the one or more frequently appearing features which represent the same genuine entity.

[0330] In an example of the first example aspect, wherein searching remote databases using the set of queries to identify products or services satisfying the query set in a set of returned search results comprises operating the at least one processor to: search the remote databases using the second set of queries to identify products or services satisfying the second set of queries in a second set of returned search results.

[0331] In an example of the first example aspect, wherein generating the textual summaries respectively corresponding to each of the identified products or services in the set of returned search results comprises operating the at least one processor to: provide a prompt to the LLM including the second set of search results, to instruct the LLM to generate a respective detaileddescription for each of the identified products or services in the second set of returned search results.

[0332] In an example of the first example aspect, wherein scoring each of the identified products and services in the set of returned search results comprises operating the at least one processor to: generate a relevancy score for each identified product or service based on a similarity of the corresponding LLM-generated textual summary to the target claim.

[0333] In an example of the first example aspect, wherein generating a relevancy score for each identified product or service comprises operating the at least one processor to: transform, using a vector embedding model, the target claim into a corresponding target claim vector; transform, using the vector embedding model, the LLM-generated summary for each identified product or service into a corresponding summary vector; and compare the target claim vector with the summary vector for each identified product or service to generate the relevancy score for each identified product or service.

[0334] In an example of the first example aspect, further comprising operating the at least one processor to: extract one or more claim elements from the target claim; generate a claim chart for one of the identified products and services in the ranked list, the claim chart identifying a relevance of the one of the identified products and services to each of the one or more claim elements; and display the claim chart.

[0335] In an example of the first example aspect, wherein generating the claim chart comprises operating the at least one processor to: transform, using a vector embedding model, the one or more claim elements into corresponding one or more claim element vectors; transform, using the vector embedding model, the LLM-generated summary for the one of the identified products and services in the ranked list into a corresponding summary vector; and compare the summary vector with each of the one or more claim element vectors to identify the relevance of the one of the identified products and services to each of the one or more claim elements.

[0336] In accordance with a second aspect of this disclosure, there is provided a system for identifying products or services relevant to a target patent claim, the system comprising: one or more processors; and a memory storing machine-executable instructions, which, when executed by the one or more processors, cause the system to: receive a patent document identifier associated with the target patent claim from a computing device in communication with the at least one processor via a network; obtain the target patent claim; retrieve, from a database in communication with the at least one processor via the network, patent document details associated with the target patent claim, using the patent document identifier; generate a set of queries, based on the retrievedpatent document details and the target patent claim; search remote databases using the set of queries to identify products or services satisfying the query set in a set of returned search results; generate, by a large language model (LLM), textual summaries respectively corresponding to each of the identified products or services in the set of returned search results; score each of the identified products or services in the set of returned search results, based on the corresponding LLM-generated textual summaries, to provide a respective relevancy score for each identified product or service; generate a ranked list of the identified products or services having a relevancy score greater than a predetermined threshold; and display the ranked list.

[0337] In an example of the second example aspect, wherein in generating a set of queries, the machine-executable instructions, when executed by the one or more processors cause the system to: generate a first set of queries, based on the retrieved patent document details; search remote databases using the first set of queries to identify products or services satisfying the first set of queries in a first set of returned search results; and generate a second set of queries based on the first set of returned search results.

[0338] In an example of the second example aspect, wherein the machine-executable instructions, when executed by the one or more processors, further cause the system to: analyze the first set of returned search results using a probabilistic approach to determine features of the preliminary search results that appear frequently in the first set of returned search results; validate one or more frequently appearing features in the first set of search results to generate a list of validated key features; and generate the second set of queries, based on the list of validated key features.

[0339] In an example of the second example aspect, wherein in validating the frequently appearing features in the first set of search results, the machine-executable instructions, when executed by the one or more processors cause the system to: refine the first set of search results to include only the one or more frequently appearing features representing a genuine entity; and process the refined first set of search results to consolidate the one or more frequently appearing features which represent the same genuine entity.

[0340] In an example of the second example aspect, wherein in searching remote databases using the set of queries to identify products or services satisfying the query set in a set of returned search results, the machine-executable instructions, when executed by the one or more processors cause the system to: search the remote databases using the second set of queries to identify products or services satisfying the second set of queries in a second set of returned search results.

[0341] In an example of the second example aspect, wherein in generating the textual summaries respectively corresponding to each of the identified products or services in the set of returned search results, the machine-executable instructions, when executed by the one or more processors cause the system to: provide a prompt to the LLM including the second set of search results, to instruct the LLM to generate a respective detailed description for each of the identified products or services in the second set of returned search results.

[0342] In an example of the second example aspect, wherein in scoring each of the identified products and services in the set of returned search results, the machine-executable instructions, when executed by the one or more processors cause the system to: generate a relevancy score for each identified product or service based on a similarity of the corresponding LLM-generated textual summary to the target claim.

[0343] In an example of the second example aspect, wherein in generating a relevancy score for each identified product or service, the machine-executable instructions, when executed by the one or more processors cause the system to: transform, using a vector embedding model, the target claim into a corresponding target claim vector; transform, using the vector embedding model, the LLM-generated summary for each identified product or service into a corresponding summary vector; and compare the target claim vector with the summary vector for each identified product or service to generate the relevancy score for each identified product or service.

[0344] In an example of the second example aspect, wherein the machine-executable instructions, when executed by the one or more processors further cause the system to: extract one or more claim elements from the target claim; generate a claim chart for one of the identified products and services in the ranked list, the claim chart identifying a relevance of the one of the identified products and services to each of the one or more claim elements; and display the claim chart.

[0345] In an example of the second example aspect, wherein in generating the claim chart, the machine-executable instructions, when executed by the one or more processors further cause the system to: transform, using a vector embedding model, the one or more claim elements into corresponding one or more claim element vectors; transform, using the vector embedding model, the LLM-generated summary for the one of the identified products and services in the ranked list into a corresponding summary vector; and compare the summary vector with each of the one or more claim element vectors to identify the relevance of the one of the identified products and services to each of the one or more claim elements.

[0346] In accordance with a third aspect of this disclosure, there is provided a method for patent portfolio ranking. The method involves operating at least one processor to: receive a plurality of patent numbers corresponding to a plurality of patents; for each claim of interest of each patent of the plurality of patents, identify at least one search result corresponding to at least one product or service relevant to the claim of interest; for each identified search result, determine a relevancy score that includes an indication of a similarity between the product or service corresponding to the search result and the claim of interest; determine a claim score based on the relevancy score of at least one identified search result; for each patent of the plurality of patents, determine a patent score based on the claim score of at least one claim of interest; rank the plurality of patents based at least in part on the patent scores of the plurality of patents; and display a ranked list of the plurality of patents.

[0347] In an example of the third example aspect, identifying the at least one product or services relevant to the claim of interest comprises operating the at least one processor to: obtain a text of the claim of interest; retrieve, from a database in communication with the at least one processor via a network, patent document details associated with the claim of interest, using the patent number; generate a set of queries based on the retrieved patent document details; and search remote databases using the set of queries to identify search results corresponding to the products or services satisfying the query set.

[0348] In an example of the third example aspect, the claim of interest is an independent claim.

[0349] In an example of the third example aspect, the method further involves operating the at least one processor to: for each patent of the plurality of patents, determine a weighted patent score; and rank the plurality of patents based on the weighted patent scores of the plurality of patents.

[0350] In an example of the third example aspect, determining the weighted patent score comprises assigning a weight to each patent of the plurality of patents based on one or more of: a financial reward associated with the patent, defense patents associated a potential infringer of the patent, a litigation history of the patent, an importance of a product or service associated with the patent.

[0351] In an example of the third example aspect, determining the claim score based on the relevancy score of the at least one identified search result comprises operating the at least one processor to compute a sum of the relevancy score of the at least one identified search result.

[0352] In an example of the third example aspect, determining the patent score comprises operating the at least one processor to compute an average of the claim score of the at least one claim of interest.

[0353] In an example of the third example aspect, the method further involves operating the at least one processor to: for each patent of the plurality of patents, determine a normalized patent score; and display the normalized patent scores in association with the ranked list of the plurality of patents.

[0354] In an example of the third example aspect, determining the relevancy score for the at least one identified search result comprises operating the at least one processor to: calculate a vector embedding of the claim of interest; and calculate a vector embedding of the search result and comparing the vector embedding of the claim and the vector embedding of the search result to obtain the relevancy score for the search result.

[0355] In an example of the third example aspect, receiving the plurality of patent numbers corresponding to the plurality of patents comprises operating the at least one processor to: receive one or more search terms from a computing device in communication with the at least one processor via a network; search a database using the one or more search terms to retrieve one or more patents matching the search terms; display the one or more patents matching the search terms; and receive a selection from the computing device, the selection corresponding to at least a subset of the one or more patents matching the search terms, wherein the plurality of patent numbers correspond to patent numbers of the at least subset of the one or more patents matching the search terms.

[0356] In an example of the third example aspect, each search result corresponds to a unique product or service.

[0357] In an example of the third example aspect, the method further involves operating the at least one processor to: combine search results corresponding to duplicate products or services and corresponding relevancy scores to obtain combined search results and corresponding global relevancy scores, each combined search result corresponding to a unique product or service; and for each claim of interest, determine the claim score based on the global relevancy score of each combined search result.

[0358] In accordance with a fourth aspect of this disclosure, there is provided a system for patent portfolio ranking. The system includes at least one processor configured to: receive a plurality of patent numbers corresponding to a plurality of patents; for each claim of interest of each patent of the plurality of patents, identify at least one search result corresponding to at leastone product or service relevant to the claim of interest; for each identified search result, determine a relevancy score that includes an indication of a similarity between the product or service corresponding to the search result and the claim of interest; determine a claim score based on the relevancy score of at least one identified search result; for each patent of the plurality of patents, determine a patent score based on the claim score of at least one claim of interest; rank the plurality of patents based at least in part on the patent scores of the plurality of patents; and display a ranked list of the plurality of patents.

[0359] In an example of the fourth example aspect, the at least one processor is configured to: obtain a text of the claim of interest; retrieve, from a database in communication with the at least one processor via a network, patent document details associated with the claim of interest, using the patent number; generate a set of queries based on the retrieved patent document details; and search remote databases using the set of queries to identify search results corresponding to the products or services satisfying the query set.

[0360] In an example of the fourth example aspect, the claim of interest is an independent claim.

[0361] In an example of the fourth example aspect, the at least one processor is configured to: for each patent of the plurality of patents, determine a weighted patent score; and rank the plurality of patents based on the weighted patent scores of the plurality of patents.

[0362] In an example of the fourth example aspect, determining the weighted patent score comprises assigning a weight to each patent of the plurality of patents based on one or more of: a financial reward associated with the patent, defense patents associated a potential infringer of the patent, a litigation history of the patent, an importance of a product or service associated with the patent.

[0363] In an example of the fourth example aspect, the at least one processor is configured to compute a sum of the relevancy score of the at least one identified search result to determine the claim score of each claim of interest.

[0364] In an example of the fourth example aspect, the at least one processor is configured to compute an average of the claim score of the at least one claim of interest to determine the patent score of each patent.

[0365] In an example of the fourth example aspect, wherein the at least one processor is configured to: for each patent of the plurality of patents, determine a normalized patent score; and display the normalized patent scores in association with the ranked list of the plurality of patents.

[0366] In an example of the fourth example aspect, the at least one processor is configured to: calculate a vector embedding of the claim of interest; and calculate a vector embedding of the search result and comparing the vector embedding of the claim and the vector embedding of the search result to obtain the relevancy score for the search result.

[0367] In an example of the fourth example aspect, the at least one processor is configured to: receive one or more search terms from a computing device in communication with the at least one processor via a network; search a database using the one or more search terms to retrieve one or more patents matching the search terms; display the one or more patents matching the search terms; and receive a selection from the computing device, the selection corresponding to at least a subset of the one or more patents matching the search terms, wherein the plurality of patent numbers correspond to patent numbers of the at least subset of the one or more patents matching the search terms.

[0368] In an example of the fourth example aspect, each search result corresponds to a unique product or service.

[0369] In an example of the fourth example aspect, the at least one processor is configured to: combine search results corresponding to duplicate products or services and corresponding relevancy scores to obtain combined search results and corresponding global relevancy scores, each combined search result corresponding to a unique product or service; and for each claim of interest, determine the claim score based on the global relevancy score of each combined search result.

[0370] In accordance with a fifth aspect of this disclosure, there is provided a non-transitory computer readable medium having a set of executable instructions stored thereon that, when executed by at least one processor, cause the at least one processor to perform the method in accordance with the first or the third aspect of this disclosure, described herein.

[0371] The present invention has been described here by way of example only. Various modifications and variations may be made to these exemplary embodiments without departing from the scope of the invention, which is limited only by the appended claims.

Claims

CLAIMS:

1. A computer-implemented method for identifying products or services relevant to a target patent claim, the method comprising operating at least one processor to:receive a patent document identifier associated with the target patent claim from a computing device in communication with the at least one processor via a network;obtain the target patent claim;retrieve, from a database in communication with the at least one processor via the network, patent document details associated with the target patent claim, using the patent document identifier;generate a set of queries, based on the retrieved patent document details and the target patent claim;search remote databases using the set of queries to identify products or services satisfying the query set in a set of returned search results;generate, by a large language model (LLM), textual summaries respectively corresponding to each of the identified products or services in the set of returned search results;score each of the identified products or services in the set of returned search results, based on the corresponding LLM-generated textual summaries, to provide a respective relevancy score for each identified product or service;generate a ranked list of the identified products or services having a relevancy score greater than a predetermined threshold; anddisplay the ranked list.

2. The method of claim 1, wherein generating a set of queries comprises operating the at least one processor to:generate a first set of queries, based on the retrieved patent document details;search the remote databases using the first set of queries to identify products or services satisfying the first set of queries in a first set of returned search results;generate a second set of queries based on the first set of returned search results; and search the remote databases using the second set of queries to identify products or services satisfying the second set of queries in a second set of returned search results.

3. The method of claim 2, wherein the method further comprises operating the at least one processor to:analyze the first set of returned search results using a probabilistic approach to determinefeatures of the preliminary search results that appear frequently in the first set of returned search results;validate one or more frequently appearing features in the first set of search results to generate a list of validated key features by operating the at least one processor to:refine the first set of search results to include only the one or more frequently appearing features representing a genuine entity; andprocess the refined first set of search results to consolidate the one or more frequently appearing features which represent the same genuine entity; and generate the second set of queries, based on the list of validated key features.

4. The method of any one of claims 2 and 3, wherein generating the textual summaries corresponding respectively corresponding to each of the identified products or services in the set of returned search results comprises operating the at least one processor to:provide a prompt to the LLM including the second set of search results, to instruct the LLM to generate a respective detailed description for each of the identified products or services in the second set of returned search results.

5. The method of any one of claims 1 to 4, wherein scoring each of the identified products and services in the set of returned search results comprises operating the at least one processor to: transform, using a vector embedding model, the target claim into a corresponding target claim vector;transform, using the vector embedding model, the LLM-generated summary for each identified product or service into a corresponding summary vector; andcompare the target claim vector with the summary vector for each identified product or service to generate a relevancy score for each identified product or service based on a similarity of the corresponding LLM-generated textual summary to the target claim.

6. The method of any one of claims 1 to 5, further comprising operating the at least one processor to:extract one or more claim elements from the target claim;transform, using a vector embedding model, the one or more claim elements into corresponding one or more claim element vectors;transform, using the vector embedding model, the LLM-generated summary for the one of the identified products and services in the ranked list into a corresponding summary vector; compare the summary vector with each of the one or more claim element vectors toidentify a relevance of the one of the identified products and services to each of the one or more claim elements;generate a claim chart for the one of the identified products and services in the ranked list, the claim chart identifying the relevance of the one of the identified products and services to each of the one or more claim elements; anddisplay the claim chart.

7. A computer-implemented method for patent portfolio ranking, the method comprising operating at least one processor to:receive a plurality of patent numbers corresponding to a plurality of patents;for each claim of interest of each patent of the plurality of patents,identify at least one search result corresponding to at least one product or service relevant to the claim of interest;for each identified search result, determine a relevancy score that includes an indication of a similarity between the product or service corresponding to the search result and the claim of interest;determine a claim score based on the relevancy score of at least one identified search result;for each patent of the plurality of patents, determine a patent score based on the claim score of at least one claim of interest;rank the plurality of patents based at least in part on the patent scores of the plurality of patents; anddisplay a ranked list of the plurality of patents.

8. The method of claim 7, wherein the claim of interest is an independent claim and identifying the at least one product or services relevant to the claim of interest comprises operating the at least one processor to:obtain a text of the claim of interest;retrieve, from a database in communication with the at least one processor via a network, patent document details associated with the claim of interest, using the patent number;generate a set of queries based on the retrieved patent document details; and search remote databases using the set of queries to identify search results corresponding to the products or services satisfying the query set.

9. The method of any one of claims 7 and 8, further comprising operating the at least oneprocessor to:for each patent of the plurality of patents, determine a weighted patent score by assigning a weight to each patent of the plurality of patents based on one or more of: a financial reward associated with the patent, defense patents associated a potential infringer of the patent, a litigation history of the patent, an importance of a product or service associated with the patent; and rank the plurality of patents based on the weighted patent scores of the plurality of patents.

10. The method of any one of claims 7 to 9, wherein determining the claim score based on the relevancy score of the at least one identified search result comprises operating the at least one processor to compute a sum of the relevancy score of the at least one identified search result and / or determining the patent score comprises operating the at least one processor to compute an average of the claim score of the at least one claim of interest.

11. The method of any one of claims 7 to 10, further comprising operating the at least one processor to:for each patent of the plurality of patents,determine a normalized patent score; anddisplay the normalized patent scores in association with the ranked list of the plurality of patents.

12. The method of any one of claims 7 to 11, wherein determining the relevancy score for the at least one identified search result comprises operating the at least one processor to:calculate a vector embedding of the claim of interest; andcalculate a vector embedding of the search result and comparing the vector embedding of the claim and the vector embedding of the search result to obtain the relevancy score for the search result.

13. A system for identifying products or services relevant to a target patent claim, the system comprising:one or more processors; anda memory storing machine-executable instructions, which, when executed by the one or more processors, cause the system to:receive a patent document identifier associated with the target patent claim from a computing device in communication with the at least one processor via a network;obtain the target patent claim;retrieve, from a database in communication with the at least one processor via the network, patent document details associated with the target patent claim, using the patent document identifier;generate a set of queries, based on the retrieved patent document details and the target patent claim;search remote databases using the set of queries to identify products or services satisfying the query set in a set of returned search results;generate, by a large language model (LLM), textual summaries respectively corresponding to each of the identified products or services in the set of returned search results;score each of the identified products or services in the set of returned search results, based on the corresponding LLM-generated textual summaries, to provide a respective relevancy score for each identified product or service;generate a ranked list of the identified products or services having a relevancy score greater than a predetermined threshold; anddisplay the ranked list.

14. A system for patent portfolio ranking, the system comprising:one or more processors; anda memory storing machine-executable instructions, which, when executed by the one or more processors, cause the system to:receive a plurality of patent numbers corresponding to a plurality of patents; for each claim of interest of each patent of the plurality of patents, identify at least one search result corresponding to at least one product or service relevant to the claim of interest;for each identified search result, determine a relevancy score that includes an indication of a similarity between the product or service corresponding to the search result and the claim of interest;determine a claim score based on the relevancy score of at least one identified search result;for each patent of the plurality of patents, determine a patent score based on the claim score of at least one claim of interest;rank the plurality of patents based at least in part on the patent scores of the plurality of patents; anddisplay a ranked list of the plurality of patents.

15. A non-transitory computer readable medium having a set of executable instructions stored thereon that, when executed by at least one processor, cause the at least one processor to perform the method of any one of claims 1 to 12.