Change Determination System for Data Groupings
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing systems face inefficiencies in identifying and extracting changed data from data sources, leading to unnecessary data duplication and resource wastage, especially when dealing with large datasets where only a small portion has changed.
Innovation Solution
A change determination system that compares current and historical data groupings to identify changed data items, using techniques like Bloom filters for efficient comparison, and forwards only the changed data to a destination system, thereby reducing redundant data transfer and storage.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Loss of information
If all data is transferred and stored from the data source, then data completeness is ensured, but data transfer time and storage resources are wasted due to redundant data
Solution Approach 1:
The patent extracts only the changed data items from the data source by comparing current data groupings with historical local versions. The system identifies and extracts solely the modified data items rather than transferring all data, thereby reducing data transfer time while maintaining data completeness for changed items.
Solution Approach 2:
The patent performs preliminary actions by maintaining local versions of data groupings and computing grouping information (such as hashes or checksums) in advance. This preliminary preparation enables efficient identification of changed data through comparison, reducing the time required for actual data transfer while ensuring completeness of changed data retrieval.
2Loss of information
If all data is transferred and stored from the data source, then data completeness is ensured, but storage resources are wasted due to redundant data
Solution Approach 1:
The patent extracts only the changed data items from the data source by comparing current data groupings with historical local versions. The system identifies and extracts solely the modified data items rather than transferring all data, thereby reducing storage resources required while maintaining data completeness for changed items.
3Productivity
If data groupings are compared to identify changes, then data transfer efficiency is improved, but system complexity increases due to comparison operations
Solution Approach 1:
The patent segments data into groupings based on common attributes (such as customer ID, product category, or time period). By dividing the data into manageable segments, the system can efficiently compare only relevant groupings between current and historical versions, improving data transfer efficiency while keeping the comparison mechanism relatively simple through structured organization.
Solution Approach 2:
The patent uses grouping information such as hashes or checksums as identifiers for data groupings. These grouping identifiers act like 'color codes' that enable rapid comparison and identification of changed data without requiring detailed examination of the actual data content, thus improving efficiency while maintaining manageable system complexity.
Data Source
Figure 1
Figure 2A
Figure 2B
AI summary
According to certain aspects, a computer system may be configured to obtain information indicating a plurality of groupings of data stored in a data source, the information indicating a number of data items included in each of the plurality of groupings; determine a first grouping of the plurality of groupings including one or more data items that have changed by comparing a first number of data items included in the first grouping and a historical first number of data items included in a corresponding local version of the first grouping; access data items included in the first grouping from the data source; compare the data items included in the first grouping to data items of the corresponding local version of the first grouping to determine which data items have changed; extract the changed data items of the first grouping; and forward the extracted data items to a destination system.