Streaming Data Management via New File Storage
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Batch-processing engines face challenges in real-time analysis due to locking periods and difficulty in excluding out-of-date records, leading to inefficient data processing and analysis.
Innovation Solution
The solution involves storing updated records in new files instead of updating existing files, maintaining a datastore to indicate the most current records, and processing these records in batches to ensure only the most current data is analyzed, thereby reducing locking periods and data duplication.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Productivity
If existing files are updated with new records, then data is stored efficiently, but locking periods occur and real-time processing is blocked
Solution Approach 1:
The patent segments the data storage system into multiple independent files (e.g., file1, file2, file3) that can be processed in parallel. When a new record arrives, the system determines which file to write to based on hashing the entity ID, allowing multiple writers to operate simultaneously on different files without blocking each other, thus eliminating locking periods and enabling real-time processing.
2Productivity
If all records are processed in batches, then processing is simplified, but out-of-date records are included in analysis
Solution Approach 1:
The patent implements preliminary actions by maintaining metadata information (such as version numbers or timestamps) that indicates which records are current. Before batch processing, the system pre-identifies and filters out outdated records using this metadata, ensuring that only current records are included in the batch analysis, thus maintaining data currency accuracy while preserving batch processing efficiency.
3Quantity of substance
If records are updated in place, then storage space is optimized, but data duplication and version control become difficult
Solution Approach 1:
The patent implements a nested structure where each file contains multiple records, and each record contains both the data and its version metadata. This nested organization allows the system to maintain multiple versions of records efficiently within the same storage space, with each version properly tracked through the nested metadata structure, thus optimizing storage while simplifying version control.
Data Source
AI summary
Streaming data, such as streaming records transmitted from entities, can be managed. For example, a new record associated with an entity can be received. There can be an existing record for the entity within a group of records. The group of records can form a block. A new block for the new record can be generated. A datastore can be updated to indicate that the new block has the most current record for the entity. Entries in the datastore can be filtered to identify a subgroup of blocks that has the most current record for each entity of multiple entities. A combined group of blocks can be generated by joining the new block with the subgroup of blocks. The combined group of blocks can be processed as a batch of data by a processing engine.