Stream Data Partitioning Without Sequence Numbers
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Conventional stream data partitioning methods are inefficient when dealing with data lacking a sequence number, as they require repeated processes to specify partition positions and read records, leading to prolonged processing times.
Innovation Solution
A data partitioning apparatus that stores previous-and-subsequent information for each data point, allowing for dynamic updating of partition boundaries without allocating a sequence number, enabling efficient and real-time partitioning of stream data into equal-sized groups.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Productivity
If conventional stream data partitioning methods are used for data lacking sequence numbers, then data can be partitioned, but processing time is prolonged due to repeated processes to specify partition positions and read records
Solution Approach 1:
The patent applies preliminary action by pre-storing previous-and-subsequent information for each data point when data is initially stored in the storing unit. This allows the system to immediately determine partition positions without repeated searching, as the relationships between data points are already established and stored. When new data arrives, the system updates partition boundaries using this pre-established information, eliminating the need for repeated reading and processing of entire record sets.
2Ease of operation
If sequence numbers are allocated to stream data for partitioning, then partition positions can be easily specified, but device complexity increases due to the additional allocation process
Solution Approach 1:
The patent applies copying by creating and storing metadata (previous-and-subsequent information) that replicates the positional relationships between data points. Instead of modifying the original data with sequence numbers, the system creates a separate copy of the relationship information that can be used for partitioning decisions. This copying approach maintains data simplicity while providing the structural information needed for easy partition position specification.
3Reliability
If all data is scanned to output requested data buckets, then complete data can be provided, but processing time increases
Solution Approach 1:
The patent applies taking out by extracting and storing the essential positional relationship information (previous-and-subsequent pointers) separately from the actual data. When a partition request arrives, the system uses this extracted relationship information to directly identify and retrieve only the required data buckets without scanning through all data. This extraction of key structural information enables selective access while maintaining data completeness.
Data Source
AI summary
A data partitioning apparatus receives stream data and stores with previous-and-subsequent information that specifies data previous and subsequent to the data. Then, for groups, each of which contains the data that are stored and that are partitioned into the number of the groups, the data partitioning apparatus stores top information specifying data located at the top in a corresponding group and end information specifying data located at the end in the corresponding group. Then, when new data is stored, the data partitioning apparatus specifies data previous and subsequent to the new data and stores the new data by associating it with the previous-and-subsequent information that specifies the data previous and subsequent to the new data. Then, when data is inserted, the data partitioning apparatus updates information for each group such that the difference between the number of data belonging to each group is equal to or less than one.