Logical Circular Buffer for Bounded Data Storage
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
The increasing amount of data from IoT devices poses challenges in designing efficient, scalable, and flexible storage systems, as much of the data generated becomes less relevant over time, requiring effective management of storage architectures to prioritize recent data while handling large volumes of less important older data.
Innovation Solution
A large-scale multi-tenant storage system is implemented using distributed circularly linked append-only buffer elements, decoupling logical data storage from physical storage by allocating logical storage units across physical units to form a logical circular buffer, which replaces oldest data with most recent data, ensuring efficient storage with bounded duration.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Productivity
If data is stored in traditional linear storage structures, then data can be retrieved sequentially, but I/O operations increase significantly when accessing recent data in large storage systems
Solution Approach 1:
The storage system segments data into fixed-size buffer elements arranged in a circular buffer structure, where recent data is stored in dedicated recent data buffers and older data is stored in older data buffers. This segmentation allows the system to quickly access recent data without scanning through entire storage structures, significantly reducing I/O operations while maintaining storage efficiency.
2Quantity of substance
If storage capacity is increased to accommodate growing IoT data, then more data can be stored, but storage system complexity increases
Solution Approach 1:
The patent extracts the management of recent data from the general storage system by creating a separate recent data buffer structure. This extraction allows the storage system to handle recent data with specialized optimized logic while managing older data through more conventional methods, thereby scaling storage capacity without proportionally increasing overall system complexity.
Solution Approach 2:
The storage system dynamically adjusts the boundary between recent data buffers and older data buffers based on data age and access patterns. As data transitions from recent to older, it moves between buffer types, allowing the system to adapt to changing storage needs and access patterns without requiring complete architectural redesign, thus enabling scalable capacity management.
3Ease of operation
If all data is treated equally in storage, then storage management is simple, but recent data cannot be prioritized for faster access
Solution Approach 1:
The storage system applies local quality by treating recent data and older data differently through separate buffer structures. Recent data is stored in recent data buffers with optimized access patterns, while older data is stored in older data buffers with different management characteristics. This local differentiation enables prioritized access to recent data while maintaining manageable complexity through clear structural boundaries.
Data Source
AI summary
A method comprising receiving one or more data storage specifications from a tenant, determining that a plurality of physical storage units in a physical storage system comprises a sufficient system capacity to provision data storage for the tenant, sending a storage request message to request creation of a network storage entity for the tenant according to the data storage specifications, and receiving a storage response message indicating a first of a plurality of logical storage units allocated to the network storage entity according to the data storage specifications, wherein the plurality of logical storage units are distributed across the plurality of physical storage units, and wherein the plurality of logical storage units are arranged in a sequential order to form a logical circular buffer.