WORM Storage Merge for IP Category Updates
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing distributed storage and data processing techniques based on the write once read many (WORM) model are not natively compatible with updating large data sets, such as traffic flow records, which are essential for value-added services and summary records in telecommunications networks.
Innovation Solution
A merge operation is performed within a distributed storage architecture that uses the WORM storage model to integrate updated records into existing data sets, employing a delta table and temporary table approach to update the IP categorization table, allowing for category label updates and reflecting changing roles of IP addresses.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If per-record updating is performed on large data sets, then data integrity and accuracy are improved, but compatibility with the WORM model deteriorates
Solution Approach 1:
The patent segments the update operation into two distinct phases: (1) writing updated records to a separate delta table, and (2) performing a merge operation to integrate changes. This segmentation allows the system to maintain per-record updating capability while preserving WORM model compatibility, as each phase operates independently without violating write-once constraints.
Solution Approach 2:
The patent introduces a delta table as an intermediary structure that temporarily holds updated records before merging them into the main data set. This intermediary enables the system to perform updates without directly modifying existing WORM-compliant data, thus maintaining compatibility while achieving data integrity through controlled merge operations.
2Reliability
If distributed storage architecture with WORM model is used, then storage reliability and data protection are improved, but ability to update large data sets deteriorates
Solution Approach 1:
The patent performs preliminary actions by first writing all updated records to a delta table before executing the merge operation. This preliminary step prepares the update data in a WORM-compliant manner, allowing the subsequent merge to proceed without violating storage model constraints while maintaining ease of operation through a structured two-phase process.
Solution Approach 2:
The patent introduces dynamic elements to the otherwise static WORM storage model by implementing a two-phase update mechanism with intermediate delta tables. This dynamic approach allows the system to adapt between the immutability requirements of WORM and the flexibility needs of data updating, achieving both storage reliability and operational ease.
3Loss of information
If merge operation is performed to integrate updated records, then data currency and accuracy are improved, but processing time and computational resources increase
Solution Approach 1:
The patent applies partial action by performing merges at controlled intervals rather than continuously after every update. The delta table accumulates changes and merges are executed periodically or when thresholds are met, reducing the frequency of expensive merge operations while maintaining acceptable data currency through batch processing.
Solution Approach 2:
The patent implements periodic merge operations where updated records in the delta table are integrated into the main data set at scheduled intervals or when specific conditions are met. This periodic approach balances data currency requirements with computational resource constraints, avoiding continuous merging while ensuring data remains current through regular update cycles.
Data Source
AI summary
Category labels, associated with IP addresses, may define a dominant usage role, in a network, of a corresponding IP address. Updates for a number of IP address/category label associations may be received, where the update may reflect changing roles relating to the IP addresses and/or to reflect newly used IP addresses. A merge operation may be performed to integrate the updated category labels into an existing data set that relates category labels to IP addresses. The merge operation may be performed within a distributed storage architecture that uses the write once read many (WORM) storage model.