Source Content Filtering Before Persistence Using Text Mining

Resolve Bottlenecks,
Find Innovative SolutionsGenerate Solutions

Solution Overview

Problem

Existing content analytics software struggles with scalability due to inefficient data ingestion and storage, leading to increased processing time and legal risks, as well as the need for manual data cleaning by data scientists.

Innovation Solution

Implement a source content processor with control logic and filtering rules that leverages a text mining engine to extract metadata, allowing for precise content filtering before data persistence, reducing irrelevant data and enhancing the data ingestion pipeline.

Engineering Contradictions & Design Principles

VSEngineering Contradiction Analysis

1Productivity

If existing content analytics software stores all source content in databases for analysis, then content analytics functionality is provided, but scalability is limited due to large data volume and increased processing time

Engineering Contradiction:

Improvecontent analytics processing speedVSAvoidamount of source content stored

Core Design Contradiction:

ProductivityVSQuantity of substance

Solution Approach 1:

The system performs preliminary text mining and metadata extraction on source content before it is stored in the database. By analyzing content characteristics, extracting entities, concepts, and relationships upfront, the system can make informed decisions about what to store and what to filter out, thereby reducing the quantity of content that needs to be stored while maintaining analytics capability.

Inventive Principle:

Principle #10Preliminary action

Solution Approach 2:

The system extracts only the essential metadata and key content characteristics from the full source content before storage. By taking out only the necessary information (entities, concepts, relationships, content type) and storing that alongside minimal content samples, the system reduces data volume while preserving the ability to perform content analytics.

Inventive Principle:

Principle #2Taking out (Extraction)

2Measurement precision

If all source content is persisted into the repository for analysis, then complete data availability is maintained, but time required to identify relevant content increases significantly

Engineering Contradiction:

Improvecontent relevancy identification accuracyVSAvoidtime to identify relevant content

Core Design Contradiction:

Measurement precisionVSLoss of time

Solution Approach 1:

The system performs preliminary text mining, entity extraction, and content classification before content is persisted. By pre-processing and tagging content with relevant metadata and characteristics upfront, the system enables fast retrieval and identification of relevant content later without needing to search through all stored content, thus reducing time loss while maintaining precision.

Inventive Principle:

Principle #10Preliminary action

Solution Approach 2:

The system introduces an intermediary layer of metadata and extracted characteristics between the source content and the final content repository. This intermediary metadata structure acts as a mediator that enables efficient filtering and rapid identification of relevant content, allowing users to query and identify pertinent information much faster without sacrificing accuracy.

Inventive Principle:

Principle #24Intermediary (Mediator)

3Reliability

If manual data cleaning by data scientists is performed, then data quality is improved, but productivity is reduced due to time-consuming cleaning processes

Engineering Contradiction:

Improvedata qualityVSAvoiddata processing efficiency

Core Design Contradiction:

ReliabilityVSProductivity

Solution Approach 1:

The system performs self-service text mining and metadata extraction automatically on incoming source content. By using automated algorithms to identify entities, concepts, relationships, and content characteristics, the system cleans and prepares data without requiring manual intervention from data scientists, thereby maintaining data quality while significantly improving processing efficiency and productivity.

Inventive Principle:

Principle #25Self-service

Solution Approach 2:

The system replaces the mechanical manual data cleaning process with automated computational text mining techniques. By using algorithms to extract, classify, and clean data automatically, the system eliminates the need for human data scientists to manually review and clean content, thereby maintaining data quality while dramatically improving productivity and reducing time consumption.

Inventive Principle:

Principle #28Mechanics substitution (Replace mechanical system)

Data Source

PatentUS20250284751A1Systems and methods for intelligent content filtering and persistence

Publication Date: 2025.09.11 OPEN TEXT SA ULC

AI summary

A source content processor receives content from a crawler and calls a text mining engine. The text mining engine mines the content and provides metadata about the content. The source content processor applies a source content filtering rule to the content utilizing the metadata from the text mining engine. The source content filtering rule is previously built based on at least one of a named entity, a category, or a sentiment. The source content processor determines whether to persist the content according to a result from applying the source content filtering rule to the content and either stores the content in a data store or deletes the contents from the data ingestion pipeline such that the content is not persisted anywhere. Embodiments disclosed herein can significantly reduce the amount of irrelevant content through the data ingestion pipeline, prior to data persistence.