System and method for assessing and mitigating risks in digital content items

Abstract

A method comprising: extracting a search term from a first user input; converting the search term into a vector representation; performing a vector search to detect contextually relevant digital content items on the Internet; extracting from a second user input a filtering question which facilitates detecting at least one of inaccuracies, biased representations, and misleading information within the content items; generating for each filtering question synonymous filtering questions and corresponding synonymous code algorithms that validate each question; supplying each content item and the synonymous filtering questions to an LLM adapted to provide an output for each of the synonymous filtering questions; applying the code algorithms to the content items to produce validation results for the at least one question; generating a content risk score for each content item based on the LLM output and the validation results; and executing a mitigating action responsive to the content risk score.

Description

SYSTEM AND METHOD FOR ASSESSING AND MITIGATING RISKS IN DIGITAL CONTENT ITEMSCROSS REFERENCE TO RELATED APPLICATIONS

[0001] This application claims the benefit of U.S. Provisional Application No.63 / 753,753 filed on February 4, 2025, the contents of which are incorporated herein by reference.TECHNICAL FIELD

[0002] The disclosure generally relates to online asset protection, and more particularly, to systems and methods for assessing and mitigating identified risks in digital content items.BACKGROUND

[0003] In today’s digital landscape, online platforms play a critical role in disseminating information to the public. However, as digital content creation and publication become increasingly accessible, the risk of inaccurate or misleading information spreading on such platforms has grown significantly. This issue is particularly concerning for high-visibility platforms, such as news outlets, social media channels, and public information sites, where the spread of misinformation can erode public trust and significantly influence public perception.

[0004] The proliferation of misinformation on these platforms can have far-reaching consequences, influencing public opinion, swaying public discourse, and, in some cases, endangering public health and safety. This issue is exacerbated by the rapid pace at which content is published and shared online, making it difficult for existing verification processes to keep up. Even minor inaccuracies or misleading representations on high-visibility platforms can quickly gain traction, leading to widespread dissemination before corrections or retractions can be issued.

[0005] Traditional methods for content verification, including manual review and basic automated tools, are often insufficient for large-scale, real-time monitoring. Existing solutions lack the robustness needed to proactively detect inaccuracies and prevent theirdissemination. Therefore, it would be advantageous to provide a solution that overcomes the shortcomings of prior art solutions noted above.SUMMARY OF THE DISCLOSURE

[0006] A summary of several example embodiments of the disclosure follows. This summary is provided for the convenience of the reader to provide a basic understanding of such embodiments and does not wholly define the breadth of the disclosure. This summary is not an extensive overview of all contemplated embodiments and is intended to neither identify key or critical elements of all embodiments nor to delineate the scope of any or all aspects. Its sole purpose is to present some concepts of one or more embodiments in a simplified form as a prelude to the more detailed description that is presented later. For convenience, the term “certain embodiments” may be used herein to refer to a single embodiment or multiple embodiments of the disclosure.

[0007] Certain embodiments disclosed herein include a method for assessing and mitigating identified risks in digital content items by a computing system. The method comprises: extracting, by the computing system, a search term from a first user input received at the computing system; converting, by the computing system, the search term into a vector representation; performing, by the computing system, a vector search to detect contextually relevant digital content items in at least one web source, based on the vector representing the search term; extracting, by the computing system, at least one filtering question from a second user input received at the computing system, wherein the at least one filtering question facilitates detecting at least one of inaccuracies, biased representations, and misleading information within the relevant digital content items; generating, by the computing system, for each of the at least one filtering question a plurality of synonymous filtering questions and a plurality of synonymous code algorithms that validate each of the at least one question; supplying, by the computing system, each relevant digital content item and the plurality of synonymous filtering questions to a large language model (LLM), wherein the LLM is adapted to provide an output for each respective one of the plurality of synonymous filtering questions; applying, by the computing system, the plurality of synonymous code algorithms to the relevant digital content items to so as to produce validation results for the at least one question;generating, by the computing system, a content risk score for each relevant digital content item based on the output of the first LLM for each of the plurality of synonymous filtering questions and the validation results of the at least one filtering question; and executing a mitigating action for each relevant digital content item having a content risk score that is above a threshold value.

[0008] Certain embodiments disclosed herein also include a non-transitory computer readable medium having stored thereon instructions for causing a processing circuitry to execute a process for assessing and mitigating identified risks in digital content items by a computing system, the process comprising: extracting, by the computing system, a search term from a first user input received at the computing system; converting, by the computing system, the search term into a vector representation; performing, by the computing system, a vector search to detect contextually relevant digital content items in at least one web source, based on the vector representing the search term; extracting, by the computing system, at least one filtering question from a second user input received at the computing system, wherein the at least one filtering question facilitates detecting at least one of inaccuracies, biased representations, and misleading information within the relevant digital content items; generating, by the computing system, for each of the at least one filtering question a plurality of synonymous filtering questions and a plurality of synonymous code algorithms that validate each of the at least one question; supplying, by the computing system, each relevant digital content item and the plurality of synonymous filtering questions to a large language model (LLM), wherein the LLM is adapted to provide an output for each respective one of the plurality of synonymous filtering questions; applying, by the computing system, the plurality of synonymous code algorithms to the relevant digital content items to so as to produce validation results for the at least one question; generating, by the computing system, a content risk score for each relevant digital content item based on the output of the first LLM for each of the plurality of synonymous filtering questions and the validation results of the at least one filtering question; and executing a mitigating action for each relevant digital content item having a content risk score that is above a threshold value.

[0009] Certain embodiments disclosed herein also include a system for assessing and mitigating identified risks in digital content items. The system comprises: aprocessing circuitry; and a memory, the memory containing instructions that, when executed by the processing circuitry, configure the system to: extract a search term from a first user input received at the computing system; convert the search term into a vector representation; perform a vector search to detect contextually relevant digital content items in at least one web source, based on the vector representing the search term; extract at least one filtering question from a second user input received at the computing system, wherein the at least one filtering question facilitates detecting at least one of inaccuracies, biased representations, and misleading information within the relevant digital content items; generate for each of the at least one filtering question a plurality of synonymous filtering questions and a plurality of synonymous code algorithms that validate each of the at least one question; supply each relevant digital content item and the plurality of synonymous filtering questions to a large language model (LLM), wherein the LLM is adapted to provide an output for each respective one of the plurality of synonymous filtering questions; apply the plurality of synonymous code algorithms to the relevant digital content items to so as to produce validation results for the at least one question; generate a content risk score for each relevant digital content item based on the output of the first LLM for each of the plurality of synonymous filtering questions and the validation results of the at least one filtering question; and execute a mitigating action for each relevant digital content item having a content risk score that is above a threshold value.BRIEF DESCRIPTION OF THE DRAWING

[0010] In the drawing:

[0011] FIG. 1 shows an illustrative network diagram utilized to describe various embodiments;

[0012] FIG. 2 is a block diagram of an illustrative management server according to an embodiment;

[0013] FIG. 3 is a flowchart showing an illustrative method for generating a content risk score for digital content items, according to an embodiment; and

[0014] FIG. 4 is a flowchart showing an illustrative method for mitigating identified risks in digital content items, according to an embodiment.DETAILED DESCRIPTION

[0015] It is important to note that the embodiments disclosed herein are only examples of the many advantageous uses of the innovative teachings herein. In general, statements made in the specification of the present application do not necessarily limit any of the various claimed embodiments. Moreover, some statements may apply to some inventive features but not to others. In general, unless otherwise indicated, singular elements may be in plural and vice versa with no loss of generality. In the drawings, like numerals refer to like parts through several views.

[0016] A method for assessing and mitigating risks in digital content items is disclosed. The method includes extracting a search term from a first user input, converting it into a vector, and performing a vector search to detect relevant digital content items from web sources which are then retrieved. At least one filtering question, extracted from a second user input, is used to detect inaccuracies, biased representations, or misleading information within the detected content items in conjunction with the LLM. For each filtering question, synonymous questions and synonymous code algorithms are generated. The filtering questions and the relevant digital content items are supplied to a large language model (LLM) for analysis. By using the filtering questions and their synonymous variants as analytical prompts or evaluation criteria, the LLM performs an analysis of the relevant digital content items, including, for example, assessing at least one of their factual accuracy, bias, and potential misleading representations so as to provide output indicative of the reliability of the digital content, e.g., with respect to its accuracy, bias, and potentially misleading representations. Each relevant digital content item is applied to the synonymous code algorithms to validate the filtering question. A content risk score is generated for each digital content item based on the LLM's output and the validation results. The type of risk in each content item is determined, and for content with a risk score above a threshold, a mitigating action is executed to address the identified risk.

[0017] FIG. 1 shows an illustrative network diagram 100 utilized to describe various embodiments. In the example diagram 100, a management server 120, a web source 130, a user device 140, and a database 150 are communicatively connected to a network 110. The network 110 may be, a wireless network, a wired network, a wide areanetwork (WAN), local area network (LAN), or any other kind of applicable network, as well as any combination thereof.

[0018] The management server 120 may include hardware and software layers that enable the management server 120 to communicate with the different components connected to the network 110, collect data and / or digital content items, apply models to the collected data and / or digital content items, generate content risk score, and so on, as further described herein.

[0019] The web source 130 may include, a website, a database, or similar online resources. The web source 130 may be for example, a news website in which articles are published, social media pages, and the like.

[0020] The user device 140 may include, a smartphone, a tablet, a personal computer (PC), a laptop, a wearable device, or any other electronic device capable of supporting the disclosed embodiments. The user device 140 serves as the primary interface for users, enabling them to interact with the system by entering inputs and receiving generated outputs.

[0021] The database 150 serves as a digital warehouse, designed to store and manage extensive data sets, digital content items, and the like. This data may include user inputs, search terms, filtering questions, synonymous filtering questions, code algorithms, and so on. The database 150 facilitates efficient retrieval and organization of data, allowing the management server 120 to access the necessary information swiftly to generate new content or refine existing content based on user inputs.

[0022] One or more end-point devices (EPD), e.g., the EPD 160, may be communicatively connected to the network 110. The EPD 160 may be, for example, a mobile phone, tablet, personal computer, server, or any other computing device capable of sending and receiving data over the network. The EPD 160 may be associated with an entity such as a business organization, governmental agency, educational institution, individual user, and the like.

[0023] FIG. 2 is a block diagram of an illustrative management server 120 according to an embodiment. The management server 120 includes a processing circuitry 121 coupled to a memory 122, a storage 123, a network interface 124 and a machine learning (ML) processor 125. In the embodiment, the components of themanagement server 120 may be communicatively connected via a bus 126.

[0024] The processing circuitry 121 may be realized as one or more hardware logic components and circuits. For example, and without limitation, illustrative types of hardware logic components that can be used, include field programmable gate arrays (FPGAs), application-specific integrated circuits (ASICs), Application-specific standard products (ASSPs), system-on-a-chip systems (SOCs), general-purpose microprocessors, microcontrollers, digital signal processors (DSPs), and the like, or any other hardware logic components that can perform calculations or other manipulations of information.

[0025] The memory 122 may be volatile, e.g., RAM, etc., non-volatile, e.g., ROM, flash memory, etc., or a combination thereof. In one configuration, computer readable instructions to implement one or more embodiments disclosed herein may be stored in the storage 123.

[0026] In another embodiment, the memory 122 is configured to store software. Software shall be construed broadly to mean any type of instructions, whether referred to as software, firmware, middleware, microcode, or hardware description language. Instructions may include code in formats such as source code, binary code, executable code, or any other suitable format of code. The instructions, when executed by the processing circuitry 121, cause the processing circuitry 121 to perform the various processes described herein.

[0027] The storage 123 may be magnetic storage, optical storage, and the like, and may be realized, for example, as flash memory or other memory technology, or any other medium which can be used to store the desired information.

[0028] The network interface 124 is configured to connect to a network. The network interface 124 may include, but is not limited to a wireless port, e.g., an 802.11 compliant Wi-Fi circuitry, configured to connect to a network. The network interface 124 allows the management server 120 to communicate with databases, other servers, and the like, such that the management server 120 can execute the embodiments discussed herein.

[0029] The machine learning processor 125 is configured to perform machine learning based on data received via the network interface 124 as described further herein.In an embodiment, the machine learning processor 125 is further configured to generate a content risk score for digital content items based on one or more large language models (LLMs). LLMs, which are trained on vast amounts of text data, enable the machine learning processor 125 to analyze the contextual relevance, accuracy, coherence, and so on, of digital content items. The machine learning processor 125 utilizes these models to assess whether the content aligns with factual information and to detect potential biases or misleading information. By leveraging the LLMs' advanced natural language understanding, the machine learning processor 125 can evaluate nuances in language, detect inaccuracies, biases, and other content deficiencies, thereby providing a robust mechanism for determining the overall trustworthiness of the content. This assessment is used to generate a content risk score that reflects the content’s integrity, which can then be used to filter or flag questionable content.

[0030] In an embodiment, the management server 120 receives, from a user device, e.g., the user device 140, a first user input. Typically, the user is a platform that is trying to get content evaluated. Alternatively, the user may be a person who is looking for accurate content or may be looking to otherwise evaluate content, e.g. , prior to posting the content. The first user input may include various forms of text, such as a word, a keyword, a term, a sentence, or even a longer phrase.

[0031] In an embodiment, the management server 120 extracts a search term from the first user input. The search term may represent the entire user input or only a portion thereof. For example, while the first user input may consist of multiple words or sentences, the extracted search term may include only a subset of the words deemed relevant to the subsequent search process. To that end, the management server 120 may utilize natural language processing (NLP) techniques to identify and isolate key terms that are most relevant to the user's intent. In an embodiment, the management server 120 converts the search term into a vector representation. This vector representation allows for efficient processing and comparison of the search term with other digital content items. The conversion process may involve embedding techniques, such as word embeddings or sentence embeddings, to transform the search term into a high-dimensional vector that captures its semantic meaning. The vectorized search term can then be used in downstream processes, such as vector searches, to detect contentfrom one or more web sources that is contextually relevant with respect to the search term.

[0032] In an embodiment, the management server 120 performs a vector search to detect relevant digital content items in one or more web sources, based on the vector representing the search term. The vector search utilizes the vectorized representation of the search term to efficiently match it against content indexed in a high-dimensional space. By leveraging vector-based searching techniques, the management server 120 can detect semantically related content items, even if they do not contain an exact match to the original search term. For example, if the user inputs the search term "climate change impact," the management server 120 can retrieve content items that discuss related concepts such as "global warming effects", "environmental changes", or "carbon footprint reduction", even if these terms are not explicitly present in the original query. This capability is made possible through the use of semantic embeddings, where words and phrases with similar meanings are mapped to nearby vectors in a high-dimensional space. The management server 120 may access various online platforms, databases, and web sources to retrieve content that aligns with the meaning and context of the search term.

[0033] The vector search allows the system to extend beyond keyword-based searches, enabling it to capture nuances in language and detect content that would otherwise be overlooked by traditional methods. By using embeddings that capture semantic relationships, the management server 120 can retrieve content that is contextually relevant, even if the specific words used differ from those in the original user input.

[0034] In an embodiment, the management server 120 extracts, from a second user input, at least one filtering question. The filtering question may be designed to assist in evaluating digital content items. More specifically, the second user input may include questions specifically designed to evaluate whether the content is accurate, whether there are any biases, and whether the content is complete and reliable. For example, a filtering question might be: "Does the article present sugary beverages as a healthy option?". This question aims to identify whether the content might include potentially misleading statements related to health benefits.

[0035] In an embodiment, the management server 120 generates for each filtering question a plurality of synonymous filtering questions. This process involves creating alternative formulations of the original question to account for variations in how the same inquiry might be phrased or interpreted. For instance, a filtering question like "Does the article present sugary beverages as a healthy option?" might be expanded into synonymous questions such as, "Is the content suggesting that sugary drinks have health benefits?" or "Does the article imply that sweetened beverages are a good choice for wellbeing?". By varying the phrasing, the system can capture a wider range of linguistic expressions, enhancing its ability to detect relevant content discrepancies.

[0036] The generation of synonymous filtering questions may be achieved by leveraging natural language processing (NLP) techniques, such as paraphrasing models and semantic similarity algorithms. The management server 120 may be configured to utilize these techniques to ensure that the generated questions remain semantically aligned with the intent of the original question. This process allows the system to recognize equivalent questions that may differ in wording but are aimed at assessing the same aspect of the content’s reliability, accuracy, or objectivity.

[0037] In addition to generating synonymous questions, the management server 120 also generates a plurality of synonymous code algorithms that are designed to validate each filtering question each of which independently analyzes the digital content items to detect indicators in the digital content items corresponding to the filtering question which the algorithm is designed to validate and to generate validation results that corroborate or qualify the LLM output. The validation results may include one or more machine-readable indicators, such as numerical scores, confidence values, classifications, flags, or structured outputs, and in some embodiments may further include explanatory information. These validation results are suitable for aggregation and combination with the LLM-derived outputs when generating the content risk score. These algorithms may vary in their approach to analyzing content, utilizing different techniques such as pattern recognition, sentiment analysis, or contextual keyword matching. For instance, to validate a question like "Is the content promoting sugary beverages as healthy?" the system may generate algorithms that analyze the frequency of health-related terms in conjunction with brand names or product mentions. Another algorithmmight focus on detecting positive sentiment associated with unhealthy products, thereby uncovering promotional bias.

[0038] These synonymous code algorithms are designed to operate in parallel, enabling the management server 120 to cross-validate the content against multiple criteria at once, e.g., substantially simultaneously. By utilizing a range of algorithms, the system can address the different ways in which misleading or biased information might appear. For instance, while one algorithm may focus on detecting overt promotional language, another might detect more nuanced indicators, such as suggestive phrasing or implied endorsements. This multi-layered strategy ensures a thorough assessment of the content from various perspectives.

[0039] The combination of synonymous filtering questions and corresponding validation algorithms enhances the system's flexibility and resilience. By expanding the range of questions and algorithms, the management server 120 can adapt to different contexts and content types, thereby enhancing its ability to detect inaccuracies, half-truths, biases, and misleading information. This redundancy not only increases the accuracy of the evaluation but also reduces the risk of false negatives, where potentially misleading or biased content might otherwise go undetected.

[0040] In an embodiment, the management server 120 supplies each detected relevant digital content item along with the plurality of synonymous filtering questions as input to a large language model (LLM) for content-level analysis of the digital content items. The content-level analysis may include, for example, evaluative analysis, classification, or inference-based assessment of at least one of factual accuracy, bias, and potential misleading representations in each detected relevant digital content item. The LLM is adapted to process both the content and the generated synonymous questions to assess the content’s alignment with the criteria established by the filtering questions and to supply an output indicative of the level of alignment. The output may be an alignment score, confidence value, or other machine-readable indicator.

[0041] This approach ensures a more thorough assessment of the content’s accuracy, impartiality, and reliability using multiple variations of the original filtering question.

[0042] The application of the LLM allows the system to perform nuanced contentanalysis beyond simple keyword matching. For example, if a filtering question is designed to detect biased language, the LLM can analyze the context and tone of the content to determine whether it subtly implies favoritism or prejudice, even if such bias is not explicitly stated. The LLM's advanced natural language understanding capabilities enable it to interpret the intent behind the content, thereby detecting subtle forms of misinformation or partiality.

[0043] The LLM generates responses for each of the synonymous filtering questions, effectively cross-referencing the content against various formulations of the original filtering question. This redundancy increases the likelihood of detecting issues that may be overlooked when using a single question. For instance, if one filtering question asks, "Are terrorists presented as freedom fighters?" a synonymous question might be, "Does the content portray acts of violence as justified resistance?", other synonymous questions could include, "Is the use of force depicted as a legitimate struggle?", or "Does the article frame violent actions as acts of heroism?". Thus, by evaluating the content against several synonymous questions, the LLM can deliver a more thorough assessment.

[0044] By utilizing an LLM in this manner, the system can generate specific outputs for each variation, thereby capturing a more comprehensive assessment of the content. The LLM is adapted to provide responses to each synonymous filtering question, allowing the system to cross-check the content against multiple perspectives. This approach enhances the accuracy of content evaluation, as the varied outputs help detect potential issues such as biases, inaccuracies, or misleading information that might not be detected using a single filtering question. The ability to handle multiple filtering questions simultaneously enables the management server 120 to efficiently analyze large volumes of content, flagging content that contains inaccuracies, biases, or misleading elements as problematic.

[0045] In an embodiment, the management server 120 applies the plurality of synonymous code algorithms to the relevant digital content items, using a runtime engine, in order to validate the filtering question. These algorithms are designed to analyze the content from different angles, ensuring a robust assessment of the content's accuracy, objectivity, and overall reliability. By using a set of synonymous algorithms, the systemcan cross-validate the content against multiple criteria, thereby reducing the likelihood of missing critical issues due to variations in content presentation.

[0046] Each synonymous algorithm may utilize different techniques to evaluate the content. For example, one algorithm might focus on detecting discrepancies between stated facts and verified sources, while another may look for patterns in language that indicate biased or misleading information. This multi-algorithm approach allows the system to capture both explicit and subtle inconsistencies within the content.

[0047] For example, if the filtering question is "Does the article present sugary beverages as a healthy option?", one algorithm may focus on detecting phrases that directly suggest health benefits, such as "drinking soda boosts your energy" or "soft drinks are part of a balanced diet." Another algorithm might examine the overall tone of the article to see if it implies positive associations between sugary beverages and well-being without explicitly stating it.

[0048] As another example, if the filtering question is "Are terrorists portrayed as freedom fighters?", one algorithm could analyze specific terminology used, such as detecting words like "heroic," "liberation," or "sacrifice" when describing violent actions. A different algorithm might focus on detecting the context in which these terms are used, such as detecting whether violent acts are framed as justified responses to oppression.

[0049] The algorithms operate in parallel, enabling the management server 120 to efficiently process large volumes of content in real-time. This parallel processing not only speeds up the validation process but also improves its reliability by allowing the system to aggregate results from different algorithms. If one algorithm flags content for potential inaccuracies while another detects biased language, the system can prioritize that content for further review or flag it as potentially problematic.

[0050] By applying multiple algorithms, each tailored to at least one specific aspect of content analysis, the system increases its resilience against false positives and false negatives. For instance, while a single algorithm might miss nuanced language that subtly promotes a biased viewpoint, the combination of different algorithms helps ensure a more comprehensive validation process. This holistic approach enhances the system’s ability to detect content that may not meet the required standards for accuracy and impartiality.

[0051] In an embodiment, the management server 120 generates a content risk score for each of the detected relevant digital content items, based on the output of the LLM for each of the plurality of synonymous filtering questions, as well as the validation results of the filtering question. The content risk score provides an indication of potential issues within the content, such as inaccuracies, biases, or misleading information, reflecting the content’s overall reliability and objectivity.

[0052] For example, if the LLM's output indicates that the content contains phrases suggesting that sugary beverages are beneficial to health, and if the validation algorithms confirm that the content lacks scientific references or uses promotional language, the system may assign a high content risk score to that content item. This high score serves as an indication that the content may be misleading, biased, and so on.

[0053] In an embodiment, the content risk score is a weighted combination of the severity of the issues detected by the LLM and the validation algorithms. The LLM may generate multiple outputs, e.g., one for each of the synonymous filtering questions, which may then be aggregated into an LLM-derived score. Also, the validation using the synonymous algorithms may generate multiple validation results, e.g., one per algorithm, which can likewise be aggregated into a validation-derived score. The content risk score is then generated as a weighted combination of the aggregated LLM-derived score and the aggregated validation-derived score, where the weighting reflects the severity and / or confidence associated with the detected issues. In an embodiment, content that is flagged for factual inaccuracies may receive a higher score compared to content with minor bias, ensuring that the score reflects the severity of the content discrepancies. This enables the system to prioritize which content items may require immediate attention.

[0054] The advantages of this approach are multifold. By combining the insights from both the LLM and the validation algorithms, the system achieves a more comprehensive evaluation of digital content. This multi-layered assessment reduces the likelihood of false positives and false negatives, ensuring that content flagged as unreliable truly requires scrutiny. The use of synonymous filtering questions further enhances the system’s robustness, allowing it to detect issues that might be missed by a single, narrowly defined filtering question.

[0055] To exemplify the process of generating the content risk score, thefollowing scenario is presented. The management server 120 is tasked with analyzing an article discussing the purported health benefits of sugary beverages. The management server 120 employs the LLM to evaluate the content in relation to a filtering question, such as "Does the article present sugary beverages as a healthy option?".

[0056] In this context, the system generates a set of, for example, ten synonymous filtering questions, including, for example: "Are sugary drinks portrayed as beneficial to health?", "Is there language suggesting that soft drinks are part of a balanced diet?", "Does the content imply that consuming sugary beverages improves well-being?", and the like.

[0057] The LLM processes the content with respect to each of these synonymous questions. For instance, if the LLM indicates a positive response for 8 out of the 10 questions, this represents a probability of 80% that the article portrays sugary beverages as a healthy option. This high probability may suggest that the content is biased in favor of sugary beverages.

[0058] In addition, the management server 120 applies a set of synonymous algorithms designed to detect various content issues, such as factual inaccuracies, biased language, half-truths, misleading information, and so on. For example, these algorithms detect content issues in 7 out of 10 evaluations, this translates to a score of 70% in detecting problematic content.

[0059] To generate an aggregated content risk score, the system considers both the LLM's output and the algorithms’ results and may apply a respective weight to each. In this example, the score would be calculated as a weighted average, considering the detection rates from both sources.

[0060] Therefore, the system assigns a content risk score of 75% to the content item. This score reflects the combined probability that the content contains issues such as, biases, inaccuracies, etc. Additionally, the system can classify the types of issues detected, such as "positive bias", "factual inaccuracy", "half-truths", and the like, providing more detailed insights into the nature of the content’s deficiencies.

[0061] In an embodiment, the management server 120 is configured to store the content risk scores generated for each digital content item in a database. These content risk scores are saved alongside metadata related to the content items. The metadatamay include various attributes of the content, such as the source of publication, the author or organization responsible for the content, the date and time of publication, the content’s thematic categories, e.g., health, politics, technology, and so on. By associating the content risk scores with metadata, the system enables efficient retrieval, filtering, and further analysis of digital content items.

[0062] Storing the content risk scores with corresponding metadata allows the system to perform more advanced analytics and reporting. For instance, the system can analyze patterns in digital content from different sources, time periods, or topics, thereby detecting trends in misinformation or biased content. Additionally, the metadata can include information about the specific filtering questions and algorithms used to generate the content risk score. This contextual information helps provide transparency in how the score was derived, which can be useful for audits, compliance, or manual reviews.

[0063] In an embodiment, the management server 120 determines the type of risk present in each relevant digital content item based on the output of the large language model (LLM) and the validation results. The LLM processes the digital content item and analyzes it against a set of synonymous filtering questions, which may include detecting inaccuracies, biased representations, misleading information, and the like. Thus, the LLM generates output that highlights a potential risk, i.e. , issue, within the content and classify the risk respectively.

[0064] As noted, the validation results are obtained by applying a plurality of synonymous code algorithms to the relevant digital content items. These algorithms analyze the content from multiple perspectives, ensuring that the identified types of risks are accurate. Thus, the management server 120 combines the output of the LLM with the validation results to classify the type of risk associated with each digital content item. Examples of risk types include, but are not limited to, factual inaccuracies, legal violations, regulatory non-compliance, biased representations, and the like. By accurately determining the type of risk, the management server 120 ensures that the system operates with precision, directing appropriate mitigating actions to address the specific nature of the identified risk of each digital content item.

[0065] In an embodiment, the management server 120 executes a mitigating action for each relevant digital content item having a content risk score that is above athreshold value. The mitigating action may include, generating a report for the platform hosting the content, submitting a formal complaint, generating cease-and-desist letter, flagging the content for further review, and the like. In one embodiment, the mitigating action may be, or may include, deleting the content, e.g., at the source at which the content was detected or some other location where it exists.

[0066] In an embodiment, the management server 120 generates a vector database. The vector database includes a vector representation of at least one digital content item having a content risk score that is above a threshold value and includes a vector representation of at least one of 1) a pre-existing content response document, 2) terms and conditions for at least one entity, and 3) rules and regulations for at least one jurisdiction.

[0067] The pre-existing content response documents are documents that have been successfully used to address similar risks in digital content items. These preexisting content response documents are converted into vector representations and stored in the vector database. Examples of such documents include complaint letters, cease-and-desist notices, and the like.

[0068] The terms and conditions are platform-specific guidelines, such as terms of service or community standards. These guidelines are vectorized to allow the management server 120 to evaluate whether the digital content item violates platform policies.

[0069] The rules and regulations are legal and regulatory requirements relevant to the digital content items. This includes local, regional, or international standards applicable to the digital content item.

[0070] The management server 120 uses one or more machine learning models to convert each of these items into respective vector representations. By leveraging the contextual and semantic relationships encoded within these vectors, the management server 120 ensures accurate retrieval and analysis during subsequent steps of the process. For example, the vectorized terms and conditions may help identify whether the content conflicts with specific clauses of a platform’s rules, while vectorized regulatory documents may aid in identifying legal violations.

[0071] The creation of the vector database allows the management server 120to maintain a repository of structured knowledge, which can be dynamically updated as new content is detected and obtained or when new content response documents, new terms and conditions, or new regulations become available. The management server 120 may store the vector database locally or in a distributed environment, ensuring accessibility and scalability.

[0072] In an embodiment, the management server 120 feeds a large language model (LLM), which may, but need not, be the LLM mentioned hereinabove, or it may be, a second LLM, and which, for sake of convenience and to distinguish over the initial use of the LLM described hereinabove will be referred to herein as a second LLM, with the vector representations stored in the vector database. The vector representations include the digital content item having a content risk score that is above a threshold value, the plurality of pre-existing content response documents, the terms and conditions for at least one entity, and the rules and regulations for at least one jurisdiction.

[0073] The second LLM is adapted to analyze the vector representations and retrieve, from the vector database, at least one of a pre-existing content response document relevant to the digital content item, terms and conditions of at least one entity that is relevant to the digital content item, and rules and regulations of a jurisdiction that are relevant to the digital content item.

[0074] The pre-existing content response documents relevant to the digital content item are selected based on their semantic and contextual similarity to the first relevant digital content item, allowing the system to leverage previously used actions for addressing similar issues.

[0075] With regard to the terms and conditions that are relevant to the digital content item, the second LLM identifies guidelines applicable to the digital content, such as terms of service or community standards, to determine potential violations. These terms and conditions are typically specific to the platform, such as news outlets, social media channels, etc. that are hosting the digital content item.

[0076] As for the rules and regulations that are relevant to the digital content item, the second LLM retrieves regulatory and legal requirements that may be applicable to the digital content item in at least one jurisdiction, which may also include at least one of local, regional, or international standards.

[0077] The second LLM performs semantic matching and contextual analysis of the vector representations. For example, the second LLM may compare the vectorized representation of the digital content item to the vectorized terms and conditions to detect clauses that the digital content item potentially violates. Similarly, the second LLM may match the digital content item to relevant pre-existing content response documents to retrieve actions previously taken against similar risks.

[0078] By leveraging the capabilities of the second LLM, the management server 120 ensures precise and efficient retrieval of relevant documents from the vector database. This process enables the system to tailor the subsequent mitigating actions to address the specific risk associated with the digital content item.

[0079] In an embodiment, the management server 120 generates at least one new content response document based on at least one of the retrieved pre-existing content response documents, the retrieved terms and conditions, and the retrieved rules and regulations. The new content response document is tailored to address the specific issues identified in the digital content item. For example, the new content response document may include specific references to the platform's terms and conditions or applicable legal standards that the digital content item potentially violates. It should be noted that, by leveraging the retrieved pre-existing content response documents, terms and conditions, and the rules and regulations, the management server 120 ensures that the new document incorporates effective language and structure previously used in successful mitigating actions.

[0080] The management server 120 uses machine learning models and semantic analysis techniques to generate the new content response document. This involves, for example, analyzing the retrieved pre-existing content response documents, extracting relevant portions therefrom, and combining such portions with specific clauses from a relevant platform's terms and conditions and / or regulatory requirements. The resulting new content response document is customized to reflect the type and severity of the identified risk in the digital content item. By generating a new content response document based on the retrieved elements, the management server 120 ensures that the mitigating action is precise, relevant, and aligned with established guidelines and regulations.

[0081] For instance, if the digital content item is identified as containingmisleading information, the new content response document may include a proposed correction, a citation of the relevant platform policies, and a recommendation for action. Additionally, the document may be formatted as a formal complaint, a cease-and-desist letter, or another appropriate form of communication, depending on the context.

[0082] In an embodiment, the new content response document further includes a proposed statement presenting at least one fact the management server 120 believes to be accurate and that contradicts the inaccuracies identified in the digital content item.

[0083] According to one embodiment, the proposed statement is generated based on the analysis of sources across the web that management server 120 believes to be reliable and the retrieved pre-existing content response documents. To that end, the management server 120 utilizes machine learning models and semantic processing techniques to identify facts that it takes as being accurate from sources that it understand to be authoritative and credible. These facts are selected such that they directly contradict the misleading or inaccurate claims present in the digital content item.

[0084] For example, if the digital content item contains a claim suggesting that a certain product is healthy despite scientific evidence to the contrary, the generated statement may present factual data supported by authoritative sources to refute the inaccurate claim. The statement is tailored to be precise, contextually relevant, and aligned with the specific nature of the identified inaccuracies.

[0085] Adding a statement with accurate facts strengthens the effectiveness of the new content response document. By presenting clear and factual counterarguments, the document increases the chances of resolving the issue and ensures adherence to platform guidelines and regulatory standards.

[0086] In addition to presenting accurate facts, the new content response document may include supporting references, such as links to authoritative sources or citations to published research. By including these references, the new content response document ensures that its proposed statement is backed by verifiable and authoritative evidence. This strengthens the document’s ability to address the identified issues effectively and increases the likelihood of its acceptance by the platform or other relevant entity.

[0087] In an embodiment, the management server 120 sends the generated newcontent response document to at least one relevant end-point device. The end-point device may include a PC, laptop, server, or other computing devices that are associated with platform moderation systems, human moderators, third-party regulatory and compliance systems, or content creators. For instance, the content response document can be transmitted to a social media platform’s automated content review system or moderation team for further evaluation. In cases involving violations of legal or regulatory requirements, the document may be sent to an external authority for action.

[0088] The management server 120 determines the appropriate end-point device based on at least one of the following criteria: the platform associated with the detection of the digital content item, e.g., which may be a platform where the digital content item already exists or may be a platform where the digital content item is being posted, the type of risk identified in the digital content item, the platform’s guidelines, or any applicable regulatory requirements. For example, a first content response document addressing biased content may be routed to the platform's monitoring system, whereas a second content response document addressing legal violations may be directed to a regulatory agency.

[0089] It should be noted that by sending the content response document to the relevant end-point device, the management server 120 ensures timely and effective resolution of the identified risks, enhancing the system's ability to mitigate issues within digital content.

[0090] FIG. 3 is a flowchart showing an illustrative method for generating a content risk score for digital content items, according to an embodiment. The method may be executed by the management server 120.

[0091] At S310, a search term is extracted from user input. Typically, the user is a platform that is trying to get content evaluated. Alternatively, the user may be a person who is looking for accurate content or may be looking to otherwise evaluate content, e.g., prior to posting the content. The user input may include keywords, phrases, or terms provided by the user through a user interface. The extracted search term serves as the basis for detecting contextually relevant digital content items.

[0092] At S320, the search term is converted into a vector. The vector representation enables semantic searching by capturing the contextual meaning of theterm, allowing the system to perform more accurate content retrieval.

[0093] At S330, a vector search is performed to detect relevant digital content items from one or more web sources. The management server uses the vector to search for content that is semantically aligned with the original search term.

[0094] At S340, a filtering question is extracted from the user input. The filtering question is utilized to detect potential issues in the content, such as inaccuracies, biased representations, misleading information, and the like. For example, a question might be "Does the content portray sugar-sweetened beverages as healthy for children?". This question helps assess whether the content includes biased or misleading claims that promote unhealthy products.

[0095] At S350, the system generates synonymous filtering questions and synonymous code algorithms. The generated synonymous filtering questions help broaden the evaluation criteria to ensure comprehensive content analysis. The synonymous code algorithms are designed to validate the filtering question, as further described herein.

[0096] At S360, each relevant digital content item, as well as to the generated synonymous filtering questions are applied to a large language model (LLM). The LLM processes the content and provides output for each question, where the output identifies at least one potential issue such as bias, inaccuracies, and the like, if any, for each content.

[0097] At S370, the synonymous code algorithms are applied to the relevant digital content items to validate the filtering question. These algorithms analyze the content from various perspectives to detect discrepancies, inconsistencies, misleading information, and so on.

[0098] At S380, a content risk score is generated for each digital content item based on the outputs from the LLM and the validation results from the synonymous code algorithms. The content risk score reflects the likelihood of inaccuracies, bias, or other issues within the content.

[0099] At S390, a mitigating action is executed for each relevant digital content item having a content risk score that is above a threshold value. The mitigating action may include generating a formal complaint, a cease-and-desist letter, flagging the contentfor further review, submitting a report to the hosting platform, and the like. In one embodiment, the mitigating action may be, or may include, deleting the content, e.g., at the source at which the content was detected or some other location where it exists. Deleting the content may include at least instigating the deletion of the content by transmitting a message that causes the deletion in response thereto. The specific mitigating action is determined based on the type of risk associated with the content, its severity, platform-specific guidelines, and applicable rules and regulations.

[0100] FIG. 4 is a flowchart showing an illustrative method for mitigating identified risks in digital content items, according to an embodiment. The method may be executed by the management server 120.

[0101] At S390-10, a vector database is generated. The vector database includes vector representations of a digital content item having a content risk score that is above a threshold value, pre-existing content response documents, terms and conditions (T&C) of one or more entities, and rules and regulations.

[0102] At S390-20, a second large language model (LLM) is fed with the vector representations of the digital content item having a content risk score that is above a threshold value, the pre-existing content response documents, the T&C of one or more entities, and the rules and regulations. The second LLM processes these inputs to detect semantic relationships and contextual similarities.

[0103] At S390-30, a vector search is performed. The search is based on the vectorized documents, allowing the system to locate vectorized documents that are most relevant to the identified risk in the digital content item. The search process may include applying similarity metrics, such as cosine similarity, to measure the degree of alignment between the vector representation of the digital content item and the vector representations of the documents stored in the database. This ensures that the retrieved documents are not only syntactically similar but also semantically aligned with the context of the identified risk.

[0104] At S390-40, relevant documents are retrieved from the vector database. These documents may include pre-existing content response documents addressing similar risks, specific clauses from the T&C applicable to the digital content item, and rules or regulations that may be violated by the digital content item.

[0105] At S390-50, a new content response document is generated based on at least one of the retrieved vectorized documents. The generated document is tailored to address the identified risk in the digital content item.

[0106] At S390-60, the new content response document is sent to a predetermined end-point device. The end-point device may be associated with a platform’s content moderation system, a regulatory agency, or a human moderator, depending on the nature of the identified risk.

[0107] The various embodiments disclosed herein can be implemented as hardware, firmware executing on hardware, software executing on hardware, or any combination thereof. Moreover, the software is implemented tangibly embodied on a program storage unit or computer readable medium consisting of parts, or of certain devices and / or a combination of devices. The application program may be uploaded to, and executed by, a machine comprising any suitable architecture. Preferably, the machine is implemented on a computer platform having hardware such as one or more central processing units (CPUs), a memory, and input / output interfaces. The computer platform may also include an operating system and microinstruction code. The various processes and functions described herein may be implemented as either part of the microinstruction code or part of the application program, or any combination thereof, which may be executed by a CPU, whether or not such a computer or processor is explicitly shown. In addition, various other peripheral units may be connected to the computer platform such as an additional data storage unit and a printing unit. Furthermore, a non-transitory computer readable medium is any computer readable medium except for a transitory propagating signal.

[0108] The principles of the disclosure are implemented as hardware, firmware, software, or any combination thereof. Moreover, the software is preferably implemented as an application program tangibly embodied on a program storage unit or computer readable medium. The application program may be uploaded to, and executed by, a machine comprising any suitable architecture. Preferably, the machine is implemented on a computer platform having hardware such as one or more central processing units ("CPUs"), a memory, and input / output interfaces. The computer platform may also include an operating system and microinstruction code. The various processes andfunctions described herein may be either part of the microinstruction code or part of the application program, or any combination thereof, which may be executed by a CPU, whether or not such computer or processor is explicitly shown. In addition, various other peripheral units may be connected to the computer platform such as an additional data storage unit and a printing unit.

[0109] All examples and conditional language recited herein are intended for pedagogical purposes to aid the reader in understanding the principles of the disclosure and the concepts contributed by the inventor to furthering the art and are to be construed as being without limitation to such specifically recited examples and conditions. Moreover, all statements herein reciting principles, aspects, and embodiments of the disclosure, as well as specific examples thereof, are intended to encompass both structural and functional equivalents thereof. Additionally, it is intended that such equivalents include both currently known equivalents as well as equivalents developed in the future, i.e., any elements developed that perform the same function, regardless of structure.

[0110] A person skilled-in-the-art will readily note that other embodiments of the disclosure may be achieved without departing from the scope of the disclosed disclosure. All such embodiments are included herein. The scope of the disclosure should be limited solely by the claims thereto.

Claims

CLAIMSWhat is claimed is:

1. A method for assessing and mitigating identified risks in digital content items by a computing system, the method comprising:extracting, by the computing system, a search term from a first user input received at the computing system;converting, by the computing system, the search term into a vector representation; performing, by the computing system, a vector search to detect contextually relevant digital content items in at least one web source, based on the vector representing the search term;extracting, by the computing system, at least one filtering question from a second user input received at the computing system, wherein the at least one filtering question facilitates detecting at least one of inaccuracies, biased representations, and misleading information within the relevant digital content items;generating, by the computing system, for each of the at least one filtering question, a plurality of synonymous filtering questions and a plurality of synonymous code algorithms that validate each of the at least one question;supplying, by the computing system, each relevant digital content item and the plurality of synonymous filtering questions to a large language model (LLM), wherein the LLM is adapted to provide an output for each respective one of the plurality of synonymous filtering questions;applying, by the computing system, the plurality of synonymous code algorithms to the relevant digital content items to so as to produce validation results for the at least one question;generating, by the computing system, a content risk score for each relevant digital content item based on the output of the LLM for each of the plurality of synonymous filtering questions and the validation results of the at least one filtering question; and executing a mitigating action for each relevant digital content item having a content risk score that is above a threshold value.

2. The method of claim 1 , further comprising:determining a type of risk present in each relevant digital content item based on the output of the LLM and the validation results.

3. The method of claim 1 , further comprising:creating, by the computing system, a vector database, wherein the vector database comprises a vector representation of at least one of the detected digital content items that has a content risk score that is above a threshold value and a vector representation of at least one of a plurality of pre-existing content response documents, terms and conditions of at least one entity, and rules and regulations.

4. The method of claim 3, wherein each pre-existing content response document has been successfully used as a mitigating action against digital content items that required investigation.

5. The method of claim 3, further comprising:supplying, by the computing system, an LLM, which is one of the LLM and a second LLM, with the vector representation of the at least one of the detected digital content items that has a content risk score that is above a threshold value and the vector representation of the at least one of a plurality of pre-existing content response documents, terms and conditions of at least one entity, and rules and regulations, wherein the LLM supplied with the vector representation of the at least one of the detected digital content items that has a content risk score that is above a threshold value and the vector representation of the at least one of a plurality of pre-existing content response documents is adapted to retrieve from the vector database at least one of pre-existing content response documents relevant to the at least one of the detected digital content items that has a content risk score that is above a threshold value, terms and conditions relevant to the at least one of the detected digital content items that has a content risk score that is above a threshold value, and rules and regulations relevant to the at least one of the detected digital content items that has a content risk score that is above a threshold value.

6. The method of claim 5, further comprising:generating by the computing system, at least one new content response document based on at least one of the retrieved pre-existing content response documents relevant to the at least one of the detected digital content items that has a content risk score that is above a threshold value, terms and conditions relevant to the at least one of the detected digital content items that has a content risk score that is above a threshold value, and rules and regulations relevant to the at least one of the detected digital content items that has a content risk score that is above a threshold value.

7. The method of claim 6, wherein the at least one new content response document further comprises a proposed statement presenting at least one fact determined by the computing system to be accurate and that contradicts at least one fact in the at least one of the detected digital content items that has a content risk score that is above a threshold value that was determined by the computing system to be inaccurate.

8. The method of claim 6, wherein the at least one new content response document further comprises a link to a source that was determined by the computing system to be authoritative and that substantiates the at least one new content response document.

9. The method of claim 6, further comprising:transmitting, by the computing system, the at least one new content response document to at least one end-point device.

10. A non-transitory computer readable medium having stored thereon instructions for causing a processing circuitry to execute a process for assessing and mitigating identified risks in digital content items by a computing system, the process comprising:extracting, by the computing system, a search term from a first user input received at the computing system;converting, by the computing system, the search term into a vector representation;performing, by the computing system, a vector search to detect contextually relevant digital content items in at least one web source, based on the vector representing the search term;extracting, by the computing system, at least one filtering question from a second user input received at the computing system, wherein the at least one filtering question facilitates detecting at least one of inaccuracies, biased representations, and misleading information within the relevant digital content items;generating, by the computing system, for each of the at least one filtering question, a plurality of synonymous filtering questions and a plurality of synonymous code algorithms that validate each of the at least one question;supplying, by the computing system, each relevant digital content item and the plurality of synonymous filtering questions to a large language model (LLM), wherein the LLM is adapted to provide an output for each respective one of the plurality of synonymous filtering questions;applying, by the computing system, the plurality of synonymous code algorithms to the relevant digital content items to so as to produce validation results for the at least one question;generating, by the computing system, a content risk score for each relevant digital content item based on the output of the LLM for each of the plurality of synonymous filtering questions and the validation results of the at least one filtering question; and executing a mitigating action for each relevant digital content item having a content risk score that is above a threshold value.

11. A system for assessing and mitigating identified risks in digital content items by a computing system comprising:a processing circuitry; anda memory, the memory containing instructions that, when executed by the processing circuitry, configure the system to:extract a search term from a first user input received at the computing system; convert the search term into a vector representation;perform a vector search to detect contextually relevant digital content items in at least one web source, based on the vector representing the search term;extract at least one filtering question from a second user input received at the computing system, wherein the at least one filtering question facilitates detecting at least one of inaccuracies, biased representations, and misleading information within the relevant digital content items;generate for each of the at least one filtering question a plurality of synonymous filtering questions and a plurality of synonymous code algorithms that validate each of the at least one question;supply each relevant digital content item and the plurality of synonymous filtering questions to a large language model (LLM), wherein the LLM is adapted to provide an output for each respective one of the plurality of synonymous filtering questions;apply the plurality of synonymous code algorithms to the relevant digital content items to so as to produce validation results for the at least one question;generate a content risk score for each relevant digital content item based on the output of the LLM for each of the plurality of synonymous filtering questions and the validation results of the at least one filtering question; andexecute a mitigating action for each relevant digital content item having a content risk score that is above a threshold value.

12. The system of claim 11, wherein the processing circuitry is further configured to:determine a type of risk present in each relevant digital content item based on the output of the LLM and the validation results.

13. The system of claim 11, wherein the processing circuitry is further configured to:create a vector database, wherein the vector database comprises a vector representation of at least one of the detected digital content items that has a content risk score that is above a threshold value and a vector representation of at least one of aplurality of pre-existing content response documents, terms and conditions of at least one entity, and rules and regulations.

14. The system of claim 13, wherein each pre-existing content response document has been successfully used as a mitigating action against digital content items that required investigation.

15. The system of claim 13, wherein the processing circuitry is further configured to:supply an LLM, which is one of the LLM and a second LLM, with the vector representation of the at least one of the detected digital content items that has a content risk score that is above a threshold value and the vector representation of the at least one of a plurality of pre-existing content response documents, terms and conditions of at least one entity, and rules and regulations, wherein the LLM supplied with the vector representation of the at least one of the detected digital content items that has a content risk score that is above a threshold value and the vector representation of the at least one of a plurality of pre-existing content response documents is adapted to retrieve from the vector database at least one of pre-existing content response documents relevant to the at least one of the detected digital content items that has a content risk score that is above a threshold value, terms and conditions relevant to the at least one of the detected digital content items that has a content risk score that is above a threshold value, and rules and regulations relevant to the at least one of the detected digital content items that has a content risk score that is above a threshold value.

16. The system of claim 15, wherein the processing circuitry is further configured to:generate at least one new content response document based on at least one of the retrieved pre-existing content response documents relevant to the at least one of the detected digital content items that has a content risk score that is above a threshold value, terms and conditions relevant to the at least one of the detected digital content items that has a content risk score that is above a threshold value, and rules and regulations relevantto the at least one of the detected digital content items that has a content risk score that is above a threshold value.

17. The system of claim 16, wherein the at least one new content response document further comprises a proposed statement presenting at least one fact determined by the computing system to be accurate and that contradicts at least one fact in the at least one of the detected digital content items that has a content risk score that is above a threshold value that was determined by the computing system to be inaccurate.

18. The system of claim 16, wherein the at least one new content response document further comprises a link to a source that was determined by the computing system to be authoritative and that substantiates the at least one new content response document.

19. The system of claim 16, wherein the processing circuitry is further configured to:transmit the at least one new content response document to at least one end-point device.

Images