Real Time Data Warehousing Record Matching and Separation
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Current data warehouse systems face limitations in data quality, integrity, and performance due to format discrepancies, incomplete data, record merging issues, inadequate relationship identification, and lack of real-time alert capabilities, leading to inaccurate and unreliable results.
Innovation Solution
A method and system that processes data into a database using standardized message formats, analyzes identifiers, enhances data, creates hash keys, and maintains persistent queries, while identifying relationships and issuing alerts based on user-defined rules, utilizing algorithms to match and separate records, and distribute data across databases for efficient analysis.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If records are merged to handle duplicate entities, then data quality improves, but data integrity deteriorates due to inability to separate records later
Solution Approach 1:
The patent segments the data storage into multiple tables: a raw data table that stores original records without modification and a processed data table that stores merged records. This segmentation allows the system to maintain both the original data integrity for separation needs and the merged data for quality improvement, resolving the contradiction between merging duplicates and preserving data integrity.
Solution Approach 2:
The patent introduces an intermediary processing layer that includes algorithms for matching records and determining merge decisions. This intermediary layer processes raw data into processed data while maintaining traceability, allowing the system to improve data quality through merging while preserving the ability to separate records by referencing the raw data source.
2Measurement precision
If transformation and enhancement processes are applied to received data, then data accuracy improves, but query consistency deteriorates when different algorithms are used
Solution Approach 1:
The patent applies parameter changes by standardizing data formats and transforming data into a consistent representation before storage. The system transforms received data into a standardized format and stores it with metadata about the transformation applied, ensuring that queries using the same transformation parameters will return consistent results, thereby resolving the contradiction between improving data accuracy and maintaining query consistency.
3Speed
If real-time processing is implemented to identify relationships, then response time improves, but system complexity increases
Solution Approach 1:
The patent implements preliminary action by pre-defining alert rules and processing algorithms that are executed automatically when new data is received. The system pre-establishes the logic for identifying relationships and triggering alerts, so that when data arrives, the processing can be performed immediately without complex real-time decision-making, thereby improving response time while keeping system complexity manageable through automated rule-based processing.
Data Source
AI summary
A method and system for processing data into and in a database and for retrieving the processed data is disclosed. The data comprises identifiers of a plurality of entities. The method and system comprises: (a) processing data into and in a database, (b) enhancing received data prior to storage in a database, (c) determining and matching records based upon relationships between the records in the received data and existing data without any loss of data, (d) enabling alerts based upon user-defined alert rules and relationships, (e) automatically stopping additional matches and separating previously matched records when identifiers used to match records are later determined to be common across entities and not generally distinctive of an entity, (f) receiving data queries for retrieving the processed data stored in the database, (g) utilizing the same algorithm to process the queries and (h) transferring the processed data to another database that uses the same algorithm.