Entity Record Filtering for Incremental ML Clustering
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Clustering entity records using machine learning models is computationally demanding and requires significant processing time, especially when dealing with large datasets, which leads to high network traffic and storage requirements.
Innovation Solution
Implement a system that filters entity records by identifying changes since the last clustering, generating a subset of records with common fields, and communicating only these filtered records to a machine learning model for clustering, thereby reducing the input data and computational load.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If all entity records are input to the machine learning model for clustering, then clustering accuracy is maintained, but computational demand and processing time increase significantly
Solution Approach 1:
The system performs preliminary filtering of entity records before they are input to the machine learning model. By identifying and selecting only changed entity records (those with updated fields) from the full dataset, the system prepares a reduced subset in advance. This preliminary action maintains clustering accuracy for relevant records while significantly reducing the computational load and processing time required by the model.
2Loss of information
If all entity records are transmitted over the network to the machine learning model, then complete data is available for clustering, but network traffic increases
Solution Approach 1:
The system extracts only the necessary subset of entity records (those with changes) from the complete dataset before transmission. By identifying changed records through field comparison and selecting only these relevant records for clustering, the system removes unnecessary data from the transmission stream. This extraction approach ensures that complete information about changed entities is available for accurate clustering while significantly reducing network traffic and energy consumption.
3Reliability
If the machine learning model processes all entity records, then comprehensive clustering is achieved, but computational resources are consumed
Solution Approach 1:
The system performs preliminary filtering to identify changed entity records before they are processed by the machine learning model. By pre-processing the data to select only relevant changed records based on field updates, the system reduces the input size to the computational model. This preliminary action ensures that comprehensive clustering is performed on all necessary records while minimizing the consumption of computational resources such as CPU cycles, memory, and processing power.
Data Source
AI summary
The techniques may include accessing entity data that identifies updates to a set of entity records. The techniques may include generating a set of changed entity records from a set of entity records based at least in part on the entity data. The techniques may include for each record in the set of changed entity records: identifying at least one updated field of a first entity record of the set of changed entity records; identifying at least one common field of the first entity record and one or more second entity records that are both are associated with a first cluster; comparing the updated field of the first entity record and at least one corresponding field of the second entity records to determine whether there is a change to the number of common fields; and communicating the first entity record and the second entity record to a computer system.