Systems and methods for generating a content risk score for digital content items

Abstract

A method comprising: extracting a search term from a first user input; converting the search term into a vector representation; performing a vector search to detect contextually relevant digital content items on the Internet; extracting from a second user input a filtering question which facilitates detecting at least one of inaccuracies, biased representations, and misleading information within the content items; generating for each filtering question synonymous filtering questions and corresponding synonymous code algorithms that validate each question; supplying each content item and the synonymous filtering questions to an LLM adapted to provide an output for each of the synonymous filtering questions; applying the code algorithms to the content items to produce validation results for the at least one question; and generating a content risk score for each content item based on the output of the LLM and the validation results of the at least one filtering question.

Description

SYSTEMS AND METHODS FOR GENERATING A CONTENT RISK SCORE FOR DIGITAL CONTENT ITEMSCROSS REFERENCE TO RELATED APPLICATIONS

[0001] This application claims the benefit of U.S. Provisional Application No.63 / 753,755 filed on February 4, 2025, the contents of which are incorporated herein by reference.TECHNICAL FIELD

[0002] The disclosure generally relates to online asset protection, and more particularly, to systems and methods for generating a content risk score for digital content items.BACKGROUND

[0003] In today’s digital landscape, online platforms play a critical role in disseminating information to the public. However, as digital content creation and publication become increasingly accessible, the risk of inaccurate or misleading information spreading on such platforms has grown significantly. This issue is particularly concerning for high-visibility platforms, such as news outlets, social media channels, and public information sites, where the spread of misinformation can erode public trust and significantly influence public perception.

[0004] The proliferation of misinformation on these platforms can have far-reaching consequences, influencing public opinion, swaying public discourse, and, in some cases, endangering public health and safety. This issue is exacerbated by the rapid pace at which content is published and shared online, making it difficult for existing verification processes to keep up. Even minor inaccuracies or misleading representations on high-visibility platforms can quickly gain traction, leading to widespread dissemination before corrections or retractions can be issued.

[0005] Traditional methods for content verification, including manual review and basic automated tools, are often insufficient for large-scale, real-time monitoring. Existing solutions lack the robustness needed to proactively detect inaccuracies and prevent their dissemination. Therefore, it would be advantageous to provide a solution that overcomesthe shortcomings of prior art solutions noted above.SUMMARY OF THE DISCLOSURE

[0006] A summary of several example embodiments of the disclosure follows. This summary is provided for the convenience of the reader to provide a basic understanding of such embodiments and does not wholly define the breadth of the disclosure. This summary is not an extensive overview of all contemplated embodiments and is intended to neither identify key or critical elements of all embodiments nor to delineate the scope of any or all aspects. Its sole purpose is to present some concepts of one or more embodiments in a simplified form as a prelude to the more detailed description that is presented later. For convenience, the term “certain embodiments” may be used herein to refer to a single embodiment or multiple embodiments of the disclosure.

[0007] Certain embodiments disclosed herein include a method for generating a content risk score for digital content items by a computing system, The method comprises: extracting, by the computing system, a search term from a first user input received at the computing system; converting, by the computing system, the search term into a vector representation; performing, by the computing system, a vector search to detect contextually relevant digital content items in at least one web source, based on the vector representing the search term; extracting, by the computing system, at least one filtering question from a second user input received at the computing system, wherein the at least one filtering question facilitates detecting at least one of inaccuracies, biased representations, and misleading information within the relevant digital content items; generating, by the computing system, for each of the at least one filtering question a plurality of synonymous filtering questions and a plurality of corresponding synonymous code algorithms that validate each of the at least one question; supplying, by the computing system, each relevant digital content item and the plurality of synonymous filtering questions to a large language model (LLM), wherein the LLM is adapted to provide an output for each respective one of the plurality of synonymous filtering questions; applying, by the computing system, the plurality of synonymous code algorithms to the relevant digital content items so as to produce validation results for the at least one question; and generating, by the computing system, a content risk score foreach relevant digital content item based on the output of the LLM for each of the plurality of synonymous filtering questions and the validation results of the at least one filtering question.

[0008] Certain embodiments disclosed herein also include a non-transitory computer readable medium having stored thereon instructions for causing a processing circuitry to execute a process for generating a content risk score for digital content items by a computing system, the process comprising: extracting, by the computing system, a search term from a first user input received at the computing system; converting, by the computing system, the search term into a vector representation; performing, by the computing system, a vector search to detect contextually relevant digital content items in at least one web source, based on the vector representing the search term; extracting, by the computing system, at least one filtering question from a second user input received at the computing system, wherein the at least one filtering question facilitates detecting at least one of inaccuracies, biased representations, and misleading information within the relevant digital content items; generating, by the computing system, for each of the at least one filtering question a plurality of synonymous filtering questions and a plurality of corresponding synonymous code algorithms that validate each of the at least one question; supplying, by the computing system, each relevant digital content item and the plurality of synonymous filtering questions to a large language model (LLM), wherein the LLM is adapted to provide an output for each respective one of the plurality of synonymous filtering questions; applying, by the computing system, the plurality of synonymous code algorithms to the relevant digital content items so as to produce validation results for the at least one question; and generating, by the computing system, a content risk score for each relevant digital content item based on the output of the LLM for each of the plurality of synonymous filtering questions and the validation results of the at least one filtering question.

[0009] Certain embodiments disclosed herein also include a system for generating a content risk score for digital content items. The system comprises: a processing circuitry; and a memory, the memory containing instructions that, when executed by the processing circuitry, configure the system to: extract a search term from a first user input received at the computing system; convert the search term into a vectorrepresentation; perform a vector search to detect contextually relevant digital content items in at least one web source, based on the vector representing the search term; extract at least one filtering question from a second user input received at the computing system, wherein the at least one filtering question facilitates detecting at least one of inaccuracies, biased representations, and misleading information within the relevant digital content items; generate for each of the at least one filtering question a plurality of synonymous filtering questions and a plurality of corresponding synonymous code algorithms that validate each of the at least one question; supply each relevant digital content item and the plurality of synonymous filtering questions to a large language model (LLM), wherein the LLM is adapted to provide an output for each respective one of the plurality of synonymous filtering questions; apply the plurality of synonymous code algorithms to the relevant digital content items so as to produce validation results for the at least one question; and generate a content risk score for each relevant digital content item based on the output of the LLM for each of the plurality of synonymous filtering questions and the validation results of the at least one filtering question.BRIEF DESCRIPTION OF THE DRAWING

[0010] In the drawing:

[0011] FIG. 1 shows an illustrative network diagram utilized to describe various embodiments;

[0012] FIG. 2 is a block diagram of an illustrative management server according to an embodiment; and

[0013] FIG. 3 is a flowchart showing an illustrative method for generating a content risk score for digital content items, according to an embodiment.DETAILED DESCRIPTION

[0014] It is important to note that the embodiments disclosed herein are only examples of the many advantageous uses of the innovative teachings herein. In general, statements made in the specification of the present application do not necessarily limit any of the various claimed embodiments. Moreover, some statements may apply to some inventive features but not to others. In general, unless otherwise indicated, singularelements may be in plural and vice versa with no loss of generality. In the drawings, like numerals refer to like parts through several views.

[0015] A method for generating a content risk score for digital content items. The method includes extracting a search term from a first user input, converting it into a vector, and performing a vector search to detect relevant digital content items from web sources which are then retrieved. At least one filtering question, extracted from a second user input, is used to detect inaccuracies, biased representations, or misleading information within the detected content items in conjunction with the LLM. For each filtering question, synonymous questions and synonymous code algorithms are generated. The filtering questions and the relevant digital content items are supplied to a large language model (LLM) for analysis. By using the filtering questions and their synonymous variants as analytical prompts or evaluation criteria, the LLM performs an analysis of the relevant digital content items, including, for example, assessing at least one of their factual accuracy, bias, and potential misleading representations so as to provide output indicative of the reliability of the digital content, e.g., with respect to its accuracy, bias, and potentially misleading representations. Each relevant digital content item is applied to the synonymous code algorithms to validate the filtering question. Additionally, a content risk score is generated for each digital content item based on the LLM's output and the validation results.

[0016] FIG. 1 shows an illustrative network diagram 100 utilized to describe various embodiments. In the example diagram 100, a management server 120, a web source 130, a user device 140, and a database 150 are communicatively connected to a network 110. The network 110 may be, but is not limited to, a wireless network, a wired network, a wide area network (WAN), local area network (LAN), or any other kind of applicable network, as well as any combination thereof.

[0017] The management server 120 may include hardware and software layers that enable the management server 120 to communicate with the different components connected to the network 110, collect data and / or digital content items, apply models to the collected data and / or digital content items, generate content risk score, and so on, as further described herein.

[0018] The web source 130 may include, but is not limited to, a website, adatabase, or similar online resources. The web source 130 may be for example, a news website in which articles are published, social media pages, and the like.

[0019] The user device 140 may include, but is not limited to, a smartphone, a tablet, a personal computer (PC), a laptop, a wearable device, or any other electronic device capable of supporting the disclosed embodiments. The user device 140 serves as the primary interface for users, enabling them to interact with the system by entering inputs and receiving generated outputs.

[0020] The database 150 serves as a digital warehouse, designed to store and manage extensive data sets, digital content items, and the like. This data may include user inputs, search terms, filtering questions, synonymous filtering questions, code algorithms, and so on. The database 150 facilitates efficient retrieval and organization of data, allowing the management server 120 to access the necessary information swiftly to generate new content or refine existing content based on user inputs.

[0021] FIG. 2 is a block diagram of an illustrative management server 120 according to an embodiment. The management server 120 includes a processing circuitry 121 coupled to a memory 122, a storage 123, a network interface 124 and a machine learning (ML) processor 125. In the embodiment, the components of the management server 120 may be communicatively connected via a bus 126.

[0022] The processing circuitry 121 may be realized as one or more hardware logic components and circuits. For example, and without limitation, illustrative types of hardware logic components that can be used, include field programmable gate arrays (FPGAs), application-specific integrated circuits (ASICs), Application-specific standard products (ASSPs), system-on-a-chip systems (SOCs), general-purpose microprocessors, microcontrollers, digital signal processors (DSPs), and the like, or any other hardware logic components that can perform calculations or other manipulations of information.

[0023] The memory 122 may be volatile, e.g., RAM, etc., non-volatile, e.g., ROM, flash memory, etc., or a combination thereof. In one configuration, computer readable instructions to implement one or more embodiments disclosed herein may be stored in the storage 123.

[0024] In another embodiment, the memory 122 is configured to store software.Software shall be construed broadly to mean any type of instructions, whether referred to as software, firmware, middleware, microcode, or hardware description language. Instructions may include code in formats such as source code, binary code, executable code, or any other suitable format of code. The instructions, when executed by the processing circuitry 121, cause the processing circuitry 121 to perform the various processes described herein.

[0025] The storage 123 may be magnetic storage, optical storage, and the like, and may be realized, for example, as flash memory or other memory technology, or any other medium which can be used to store the desired information.

[0026] The network interface 124 is configured to connect to a network. The network interface 124 may include, but is not limited to a wireless port, e.g., an 802.11 compliant Wi-Fi circuitry, configured to connect to a network. The network interface 124 allows the management server 120 to communicate with databases, other servers, and the like, such that the management server 120 can execute the embodiments discussed herein.

[0027] The machine learning processor 125 is configured to perform machine learning based on data received via the network interface 124 as described further herein. In an embodiment, the machine learning processor 125 is further configured to generate content risk score for digital content items based on one or more large language models (LLMs). LLMs, which are trained on vast amounts of text data, enable the machine learning processor 125 to analyze the contextual relevance, accuracy, coherence, and so on, of digital content items. The machine learning processor 125 utilizes these models to assess whether the content aligns with factual information and to detect potential biases or misleading information. By leveraging the LLMs' advanced natural language understanding, the machine learning processor 125 can evaluate nuances in language, detect inaccuracies, biases, and other content deficiencies, thereby providing a robust mechanism for determining the overall trustworthiness of the content. This assessment is used to generate a content risk score that reflects the content’s integrity, which can then be used to filter or flag questionable content.

[0028] In an embodiment, the management server 120 receives, from a user device, e.g., the user device 140, a first user input. Typically, the user is a platform thatis trying to get content evaluated. Alternatively, the user may be a person who is looking for accurate content or may be looking to otherwise evaluate content, e.g. , prior to posting the content. The first user input may include various forms of text, such as a word, a keyword, a term, a sentence, or even a longer phrase.

[0029] In an embodiment, the management server 120 extracts a search term from the first user input. The search term may represent the entire user input or only a portion thereof. For example, while the first user input may consist of multiple words or sentences, the extracted search term may include only a subset of the words deemed relevant to the subsequent search process. To that end, the management server 120 may utilize natural language processing (NLP) techniques to identify and isolate key terms that are most relevant to the user's intent. In an embodiment, the management server 120 converts the search term into a vector representation. This vector representation allows for efficient processing and comparison of the search term with other digital content items. The conversion process may involve embedding techniques, such as word embeddings or sentence embeddings, to transform the search term into a high-dimensional vector that captures its semantic meaning. The vectorized search term can then be used in downstream processes, such as vector searches, to detect content from one or more web sources that is contextually relevant with respect to the search term.

[0030] In an embodiment, the management server 120 performs a vector search to detect relevant digital content items in one or more web sources, based on the vector representing the search term. The vector search utilizes the vectorized representation of the search term to efficiently match it against content indexed in a high-dimensional space. By leveraging vector-based searching techniques, the management server 120 can detect semantically related content items, even if they do not contain an exact match to the original search term. For example, if the user inputs the search term "climate change impact," the management server 120 can retrieve content items that discuss related concepts such as "global warming effects", "environmental changes", or "carbon footprint reduction", even if these terms are not explicitly present in the original query. This capability is made possible through the use of semantic embeddings, where words and phrases with similar meanings are mapped to nearby vectors in a high-dimensionalspace. The management server 120 may access various online platforms, databases, and web sources to retrieve content that aligns with the meaning and context of the search term.

[0031] The vector search allows the system to extend beyond keyword-based searches, enabling it to capture nuances in language and detect content that would otherwise be overlooked by traditional methods. By using embeddings that capture semantic relationships, the management server 120 can retrieve content that is contextually relevant, even if the specific words used differ from those in the original user input.

[0032] In an embodiment, the management server 120 extracts, from a second user input, at least one filtering question. The filtering question may be designed to assist in evaluating digital content items. More specifically, the second user input may include questions specifically designed to evaluate whether the content is accurate, whether there are any biases, and whether the content is complete and reliable. For example, a filtering question might be: "Does the article present sugary beverages as a healthy option?". This question aims to identify whether the content might include potentially misleading statements related to health benefits.

[0033] In an embodiment, the management server 120 generates for each filtering question a plurality of synonymous filtering questions. This process involves creating alternative formulations of the original question to account for variations in how the same inquiry might be phrased or interpreted. For instance, a filtering question like "Does the article present sugary beverages as a healthy option?" might be expanded into synonymous questions such as, "Is the content suggesting that sugary drinks have health benefits?" or "Does the article imply that sweetened beverages are a good choice for wellbeing?". By varying the phrasing, the system can capture a wider range of linguistic expressions, enhancing its ability to detect relevant content discrepancies.

[0034] The generation of synonymous filtering questions may be achieved by leveraging natural language processing (NLP) techniques, such as paraphrasing models and semantic similarity algorithms. The management server 120 may be configured to utilize these techniques to ensure that the generated questions remain semantically aligned with the intent of the original question. This process allows the system torecognize equivalent questions that may differ in wording but are aimed at assessing the same aspect of the content’s reliability, accuracy, or objectivity.

[0035] In addition to generating synonymous questions, the management server 120 also generates a plurality of synonymous code algorithms that are designed to validate each filtering question each of which independently analyzes the digital content items to detect indicators in the digital content items corresponding to the filtering question which the algorithm is designed to validate and to generate validation results that corroborate or qualify the LLM output. The validation results may include one or more machine-readable indicators, such as numerical scores, confidence values, classifications, flags, or structured outputs, and in some embodiments may further include explanatory information. These validation results are suitable for aggregation and combination with the LLM-derived outputs when generating the content risk score. These algorithms may vary in their approach to analyzing content, utilizing different techniques such as pattern recognition, sentiment analysis, or contextual keyword matching. For instance, to validate a question like "Is the content promoting sugary beverages as healthy?" the system may generate algorithms that analyze the frequency of health-related terms in conjunction with brand names or product mentions. Another algorithm might focus on detecting positive sentiment associated with unhealthy products, thereby uncovering promotional bias.

[0036] These synonymous code algorithms are designed to operate in parallel, enabling the management server 120 to cross-validate the content against multiple criteria at once, e.g., substantially simultaneously. By utilizing a range of algorithms, the system can address the different ways in which misleading or biased information might appear. For instance, while one algorithm may focus on detecting overt promotional language, another might detect more nuanced indicators, such as suggestive phrasing or implied endorsements. This multi-layered strategy ensures a thorough assessment of the content from various perspectives.

[0037] The combination of synonymous filtering questions and corresponding validation algorithms enhances the system's flexibility and resilience. By expanding the range of questions and algorithms, the management server 120 can adapt to different contexts and content types, thereby enhancing its ability to detect inaccuracies, half-truths, biases, and misleading information. This redundancy not only increases the accuracy of the evaluation but also reduces the risk of false negatives, where potentially misleading or biased content might otherwise go undetected.

[0038] In an embodiment, the management server 120 supplies each detected relevant digital content item along with the plurality of synonymous filtering questions as input to a large language model (LLM) for content-level analysis of the digital content items. The content-level analysis may include, for example, evaluative analysis, classification, or inference-based assessment of at least one of factual accuracy, bias, and potential misleading representations in each detected relevant digital content item. The LLM is adapted to process both the content and the generated synonymous questions to assess the content’s alignment with the criteria established by the filtering questions and to supply an output indicative of the level of alignment. The output may be an alignment score, confidence value, or other machine-readable indicator.

[0039] This approach ensures a more thorough assessment of the content’s accuracy, impartiality, and reliability using multiple variations of the original filtering question.

[0040] The application of the LLM allows the system to perform nuanced content analysis beyond simple keyword matching. For example, if a filtering question is designed to detect biased language, the LLM can analyze the context and tone of the content to determine whether it subtly implies favoritism or prejudice, even if such bias is not explicitly stated. The LLM's advanced natural language understanding capabilities enable it to interpret the intent behind the content, thereby detecting subtle forms of misinformation or partiality.

[0041] The LLM generates responses for each of the synonymous filtering questions, effectively cross-referencing the content against various formulations of the original filtering question. This redundancy increases the likelihood of detecting issues that may be overlooked when using a single question. For instance, if one filtering question asks, "Are terrorists presented as freedom fighters?" a synonymous question might be, "Does the content portray acts of violence as justified resistance?", other synonymous questions could include, "Is the use of force depicted as a legitimate struggle?", or "Does the article frame violent actions as acts of heroism?". Thus, byevaluating the content against several synonymous questions, the LLM can deliver a more thorough assessment.

[0042] By utilizing an LLM in this manner, the system can generate specific outputs for each variation, thereby capturing a more comprehensive assessment of the content. The LLM is adapted to provide responses to each synonymous filtering question, allowing the system to cross-check the content against multiple perspectives. This approach enhances the accuracy of content evaluation, as the varied outputs help detect potential issues such as biases, inaccuracies, or misleading information that might not be detected using a single filtering question. The ability to handle multiple filtering questions simultaneously enables the management server 120 to efficiently analyze large volumes of content, flagging content that contains inaccuracies, biases, or misleading elements as problematic.

[0043] In an embodiment, the management server 120 applies the plurality of synonymous code algorithms to the relevant digital content items, using a runtime engine, in order to validate the filtering question. These algorithms are designed to analyze the content from different angles, ensuring a robust assessment of the content's accuracy, objectivity, and overall reliability. By using a set of synonymous algorithms, the system can cross-validate the content against multiple criteria, thereby reducing the likelihood of missing critical issues due to variations in content presentation.

[0044] Each synonymous algorithm may utilize different techniques to evaluate the content. For example, one algorithm might focus on detecting discrepancies between stated facts and verified sources, while another may look for patterns in language that indicate biased or misleading information. This multi-algorithm approach allows the system to capture both explicit and subtle inconsistencies within the content.

[0045] For example, if the filtering question is "Does the article present sugary beverages as a healthy option?", one algorithm may focus on detecting phrases that directly suggest health benefits, such as "drinking soda boosts your energy" or "soft drinks are part of a balanced diet." Another algorithm might examine the overall tone of the article to see if it implies positive associations between sugary beverages and well-being without explicitly stating it.

[0046] As another example, if the filtering question is "Are terrorists portrayedas freedom fighters?", one algorithm could analyze specific terminology used, such as detecting words like "heroic," "liberation," or "sacrifice" when describing violent actions. A different algorithm might focus on detecting the context in which these terms are used, such as detecting whether violent acts are framed as justified responses to oppression.

[0047] The algorithms operate in parallel, enabling the management server 120 to efficiently process large volumes of content in real-time. This parallel processing not only speeds up the validation process but also improves its reliability by allowing the system to aggregate results from different algorithms. If one algorithm flags content for potential inaccuracies while another detects biased language, the system can prioritize that content for further review or flag it as potentially problematic.

[0048] By applying multiple algorithms, each tailored to at least one specific aspect of content analysis, the system increases its resilience against false positives and false negatives. For instance, while a single algorithm might miss nuanced language that subtly promotes a biased viewpoint, the combination of different algorithms helps ensure a more comprehensive validation process. This holistic approach enhances the system’s ability to detect content that may not meet the required standards for accuracy and impartiality.

[0049] In an embodiment, the management server 120 generates a content risk score for each of the detected relevant digital content items, based on the output of the LLM for each of the plurality of synonymous filtering questions, as well as the validation results of the filtering question. The content risk score provides an indication of potential issues within the content, such as inaccuracies, biases, or misleading information, reflecting the content’s overall reliability and objectivity.

[0050] For example, if the LLM's output indicates that the content contains phrases suggesting that sugary beverages are beneficial to health, and if the validation algorithms confirm that the content lacks scientific references or uses promotional language, the system may assign a high content risk score to that content item. This high score serves as an indication that the content may be misleading, biased, and so on.

[0051] In an embodiment, the content risk score is a weighted combination of the severity of the issues detected by the LLM and the validation algorithms. The LLM may generate multiple outputs, e.g., one for each of the synonymous filtering questions,which may then be aggregated into an LLM-derived score Also, the validation using the synonymous algorithms may generate multiple validation results, e.g., one per algorithm, which can likewise be aggregated into a validation-derived score. The content risk score is then generated as a weighted combination of the aggregated LLM-derived score and the aggregated validation-derived score, where the weighting reflects the severity and / or confidence associated with the detected issues. In an embodiment, content that is flagged for factual inaccuracies may receive a higher score compared to content with minor bias, ensuring that the score reflects the severity of the content discrepancies. This enables the system to prioritize which content items may require immediate attention.

[0052] The advantages of this approach are multifold. By combining the insights from both the LLM and the validation algorithms, the system achieves a more comprehensive evaluation of digital content. This multi-layered assessment reduces the likelihood of false positives and false negatives, ensuring that content flagged as unreliable truly requires scrutiny. The use of synonymous filtering questions further enhances the system’s robustness, allowing it to detect issues that might be missed by a single, narrowly defined filtering question.

[0053] To exemplify the process of generating the content risk score, the following scenario is presented. The management server 120 is tasked with analyzing an article discussing the purported health benefits of sugary beverages. The management server 120 employs the LLM to evaluate the content in relation to a filtering question, such as "Does the article present sugary beverages as a healthy option?".

[0054] In this context, the system generates a set of, for example, ten synonymous filtering questions, including, for example: "Are sugary drinks portrayed as beneficial to health?", "Is there language suggesting that soft drinks are part of a balanced diet?", "Does the content imply that consuming sugary beverages improves well-being?", and the like.

[0055] The LLM processes the content with respect to each of these synonymous questions. For instance, if the LLM indicates a positive response for 8 out of the 10 questions, this represents a probability of 80% that the article portrays sugary beverages as a healthy option. This high probability may suggest that the content is biased in favor of sugary beverages.

[0056] In addition, the management server 120 applies a set of synonymous algorithms designed to detect various content issues, such as factual inaccuracies, biased language, half-truths, misleading information, and so on. For example, these algorithms detect content issues in 7 out of 10 evaluations, this translates to a score of 70% in detecting problematic content.

[0057] To generate an aggregated content risk score, the system considers both the LLM's output and the algorithms’ results and may apply a respective weight to each. In this example, the score would be calculated as a weighted average, considering the detection rates from both sources.

[0058] Therefore, the system assigns a content risk score of 75% to the content item. This score reflects the combined probability that the content contains issues such as, biases, inaccuracies, etc. Additionally, the system can classify the types of issues detected, such as "positive bias", "factual inaccuracy", "half-truths", and the like, providing more detailed insights into the nature of the content’s deficiencies.

[0059] In an embodiment, the management server 120 is configured to store the content risk scores generated for each digital content item in a database. These content risk scores are saved alongside metadata related to the content items. The metadata may include various attributes of the content, such as the source of publication, the author or organization responsible for the content, the date and time of publication, the content’s thematic categories, e.g., health, politics, technology, and so on. By associating the content risk scores with metadata, the system enables efficient retrieval, filtering, and further analysis of digital content items.

[0060] Storing the content risk scores with corresponding metadata allows the system to perform more advanced analytics and reporting. For instance, the system can analyze patterns in digital content from different sources, time periods, or topics, thereby detecting trends in misinformation or biased content. Additionally, the metadata can include information about the specific filtering questions and algorithms used to generate the content risk score. This contextual information helps provide transparency in how the score was derived, which can be useful for audits, compliance, or manual reviews.

[0061] FIG. 3 is a flowchart showing an illustrative method for generating a content risk score for digital content items, according to an embodiment. The methodmay be executed by the management server 120.

[0062] At S310, a search term is extracted from user input. Typically, the user is a platform that is trying to get content evaluated. Alternatively, the user may be a person who is looking for accurate content or may be looking to otherwise evaluate content, e.g., prior to posting the content. The user input may include keywords, phrases, or terms provided by the user through a user interface. The extracted search term serves as the basis for detecting contextually relevant digital content items.

[0063] At S320, the search term is converted into a vector. The vector representation enables semantic searching by capturing the contextual meaning of the term, allowing the system to perform more accurate content retrieval.

[0064] At S330, a vector search is performed to detect relevant digital content items from one or more web sources. The management server uses the vector to search for content that is semantically aligned with the original search term.

[0065] At S340, a filtering question is extracted from the user input. The filtering question is utilized to detect potential issues in the content, such as inaccuracies, biased representations, misleading information, and the like. For example, a question might be "Does the content portray sugar-sweetened beverages as healthy for children?". This question helps assess whether the content includes biased or misleading claims that promote unhealthy products.

[0066] At S350, the system generates synonymous filtering questions and synonymous code algorithms. The generated synonymous filtering questions help broaden the evaluation criteria to ensure comprehensive content analysis. The synonymous code algorithms are designed to validate the filtering question, as further described herein.

[0067] At S360, each relevant digital content item, as well as to the generated synonymous filtering questions are applied to a large language model (LLM). The LLM processes the content and provides output for each question, where the output identifies at least one potential issue such as bias, inaccuracies, and the like, if any, for each content.

[0068] At S370, the synonymous code algorithms are applied to the relevant digital content items to validate the filtering question. These algorithms analyze thecontent from various perspectives to detect discrepancies, inconsistencies, misleading information, and so on.

[0069] At S380, a content risk score is generated for each digital content item based on the outputs from the LLM and the validation results from the synonymous code algorithms. The content risk score reflects the likelihood of inaccuracies, bias, or other issues within the content.

[0070] The risk score may be used as a basis for taking further action such as deleting all or part of the content if the content risk score is below a threshold or, if the content risk score is below a threshold, delivering the content to user device or posting it somewhere, e.g., as specified by the user.

[0071] The principles of the disclosure are implemented as hardware, firmware, software, or any combination thereof. Moreover, the software is preferably implemented as an application program tangibly embodied on a program storage unit or computer readable medium. The application program may be uploaded to, and executed by, a machine comprising any suitable architecture. Preferably, the machine is implemented on a computer platform having hardware such as one or more central processing units ("CPUs"), a memory, and input / output interfaces. The computer platform may also include an operating system and microinstruction code. The various processes and functions described herein may be either part of the microinstruction code or part of the application program, or any combination thereof, which may be executed by a CPU, whether or not such computer or processor is explicitly shown. In addition, various other peripheral units may be connected to the computer platform such as an additional data storage unit and a printing unit.

[0072] All examples and conditional language recited herein are intended for pedagogical purposes to aid the reader in understanding the principles of the disclosure and the concepts contributed by the inventor to furthering the art and are to be construed as being without limitation to such specifically recited examples and conditions. Moreover, all statements herein reciting principles, aspects, and embodiments of the disclosure, as well as specific examples thereof, are intended to encompass both structural and functional equivalents thereof. Additionally, it is intended that such equivalents include both currently known equivalents as well as equivalents developed inthe future, i.e., any elements developed that perform the same function, regardless of structure.

[0073] A person skilled-in-the-art will readily note that other embodiments of the disclosure may be achieved without departing from the scope of the disclosed disclosure. All such embodiments are included herein. The scope of the disclosure should be limited solely by the claims thereto.

Claims

CLAIMSWhat is claimed is:

1. A method for generating a content risk score for digital content items by a computing system, the method comprising:extracting, by the computing system, a search term from a first user input received at the computing system;converting, by the computing system, the search term into a vector representation; performing, by the computing system, a vector search to detect contextually relevant digital content items in at least one web source, based on the vector representing the search term;extracting, by the computing system, at least one filtering question from a second user input received at the computing system, wherein the at least one filtering question facilitates detecting at least one of inaccuracies, biased representations, and misleading information within the relevant digital content items;generating, by the computing system, for each of the at least one filtering question a plurality of synonymous filtering questions and a plurality of corresponding synonymous code algorithms that validate each of the at least one question;supplying, by the computing system, each relevant digital content item and the plurality of synonymous filtering questions to a large language model (LLM), wherein the LLM is adapted to provide an output for each respective one of the plurality of synonymous filtering questions;applying, by the computing system, the plurality of synonymous code algorithms to the relevant digital content items so as to produce validation results for the at least one question; andgenerating, by the computing system, a content risk score for each relevant digital content item based on the output of the LLM for each of the plurality of synonymous filtering questions and the validation results of the at least one filtering question.

2. The method of claim 1, wherein the search term is extracted using natural language processing (NLP) techniques.

3. The method of claim 1, wherein converting the search term into a vector representation further comprises:applying, by the computing system, at least one of word embedding and sentence embedding to capture a semantic meaning of the search term.

4. The method of claim 1 , further comprising:assigning, by the computing system, a priority to each of the digital content items based on the content risk score.

5. The method of claim 1, wherein the content risk score includes an indication of a specific type of issue detected.

6. The method of claim 1 , further comprising:storing, by the computing system, the content risk score for each relevant digital content item along with metadata about the relevant digital content item.

7. The method of claim 1, wherein each of the plurality of synonymous code algorithms is tailored to at least one specific aspect of content analysis.

8. The method of claim 1 , wherein at least two of the plurality of synonymous code algorithms operate in parallel.

9. The method of claim 8, wherein the at least two of the plurality of synonymous code algorithms cross-validate the content with respect to a plurality of criteria substantially simultaneously.

10. A non-transitory computer readable medium having stored thereon instructions for causing a processing circuitry to execute a process for generating a content risk score for digital content items by a computing system, the process comprising:extracting, by the computing system, a search term from a first user input received at the computing system;converting, by the computing system, the search term into a vector representation; performing, by the computing system, a vector search to detect contextually relevant digital content items in at least one web source, based on the vector representing the search term;extracting, by the computing system, at least one filtering question from a second user input received at the computing system, wherein the at least one filtering question facilitates detecting at least one of inaccuracies, biased representations, and misleading information within the relevant digital content items;generating, by the computing system, for each of the at least one filtering question a plurality of synonymous filtering questions and a plurality of corresponding synonymous code algorithms that validate each of the at least one question;supplying, by the computing system, each relevant digital content item and the plurality of synonymous filtering questions to a large language model (LLM), wherein the LLM is adapted to provide an output for each respective one of the plurality of synonymous filtering questions;applying, by the computing system, the plurality of synonymous code algorithms to the relevant digital content items so as to produce validation results for the at least one question; andgenerating, by the computing system, a content risk score for each relevant digital content item based on the output of the LLM for each of the plurality of synonymous filtering questions and the validation results of the at least one filtering question.

11. A system for generating a content risk score for digital content items, comprising:a processing circuitry; anda memory, the memory containing instructions that, when executed by the processing circuitry, configure the system to:extract a search term from a first user input received at the computing system; convert the search term into a vector representation;perform a vector search to detect contextually relevant digital content items in at least one web source, based on the vector representing the search term;extract at least one filtering question from a second user input received at the computing system, wherein the at least one filtering question facilitates detecting at least one of inaccuracies, biased representations, and misleading information within the relevant digital content items;generate for each of the at least one filtering question a plurality of synonymous filtering questions and a plurality of corresponding synonymous code algorithms that validate each of the at least one question;supply each relevant digital content item and the plurality of synonymous filtering questions to a large language model (LLM), wherein the LLM is adapted to provide an output for each respective one of the plurality of synonymous filtering questions;apply the plurality of synonymous code algorithms to the relevant digital content items so as to produce validation results for the at least one question; andgenerate a content risk score for each relevant digital content item based on the output of the LLM for each of the plurality of synonymous filtering questions and the validation results of the at least one filtering question.

12. The system of claim 11, wherein the search term is extracted using natural language processing (NLP) techniques.

13. The system of claim 11 , wherein the processing circuitry, when converting the search term into a vector representation, is configured to:apply at least one of word embedding and sentence embedding to capture a semantic meaning of the search term.

14. The system of claim 11, wherein the processing circuitry is further configured to:assign a priority to each of the digital content items based on the content risk score.

15. The system of claim 11 , wherein the content risk score includes an indication of a specific type of issue detected.

16. The system of claim 11, wherein the processing circuitry is further configured to:store the content risk score for each relevant digital content item along with metadata about the relevant digital content item.

17. The system of claim 11, wherein each of the plurality of synonymous code algorithms is tailored to at least one specific aspect of content analysis.

18. The system of claim 11, wherein at least two of the plurality of synonymous code algorithms operate in parallel.

19. The system of claim 18, wherein the at least two of the plurality of synonymous code algorithms cross-validate the content with respect to a plurality of criteria substantially simultaneously.

Images