Stateless Data Stream Storage via Offset-Based File Segmentation
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing data stream storage methods face challenges in efficiently managing large, dynamic data streams without pre-known sizes, particularly in removing earlier portions and replicating data across systems with varying file sizes, often requiring indices and stateful processes.
Innovation Solution
The method involves creating successive files of equal size, named based on binary powers of bytes, allowing mapping of data stream offsets to files without an index or state, enabling efficient storage, removal, and replication of data streams across systems with different file sizes.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Device complexity
If data streams are recorded into a single file, then the storage structure is simple, but the file grows to become very large and management becomes difficult
Solution Approach 1:
The patent divides the data stream storage into multiple sequential files of equal size (e.g., 64MB each). As data is written, it automatically transitions from one file to the next when the current file reaches its capacity. This segmentation prevents any single file from becoming excessively large while maintaining a relatively simple storage structure.
2Speed
If indices are used to map data stream portions to files, then file identification is efficient, but the system requires stateful processes and additional storage overhead
Solution Approach 1:
The system uses self-service file naming where each file is automatically named based on its position in the sequence (e.g., data-001, data-002, data-003). The filename itself encodes the file's position and size information, eliminating the need for external indices or stateful tracking. This approach achieves efficient file identification while keeping the system stateless and simple.
3Adaptability or versatility
If files are created dynamically as data arrives, then the storage adapts to data volume, but population must await file creation causing delays
Solution Approach 1:
The system pre-creates a sequence of files with equal sizes before data population begins. This preliminary action ensures that files are ready to receive data immediately, eliminating any delays caused by waiting for file creation during the data population process. The pre-created files are then filled sequentially as data arrives.
4Adaptability or versatility
If different systems use different file sizes, then each system can optimize for itself, but mapping between systems becomes complex
Solution Approach 1:
The patent allows each system (primary and secondary) to use its own local file size optimization while maintaining compatible data mapping. The system determines which file to write to based on the data offset and local file size parameters. This local quality approach enables each system to independently optimize its file sizes while still achieving correct data placement across different systems.
Data Source
AI summary
Storing an incoming data stream using successive files that are consecutively populated. The appropriate file to populate a given data stream portion into is determined by mapping the data stream offset to a file, and potentially also an address within that file. The successive files may be the same size, so that the file can be identified based on the data stream address (or offset) without the use of an index. Furthermore, the files may be easily named by having that size be some multiple of a binary power of bytes. That way, the files themselves can be automatically and named and identified by using the more significant bit or bits of the data stream offset to uniquely identify the file and establish ordering of the files. Replication may occur from a primary to a secondary store by transmitting the offset, and the actual data to be stored.