Network Data Storage Using Circular Buffers and Location Indices
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Conventional data storage systems are inadequate for efficiently storing and rapidly retrieving network data, leading to insufficient data retention for analysis due to inefficiencies in storage and retrieval processes.
Innovation Solution
A method involving the use of circular buffers and location indices to store network data as flow records, where data is organized by location and time, allowing for efficient storage and rapid retrieval through aggregation and indexing techniques, including the use of 32-bit block IDs and signed integers for timestamps and IP addresses.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Quantity of substance
If conventional data storage systems are used to store network data, then data can be stored, but storage and access times suffer greatly as the amount of data is increased
Solution Approach 1:
The patent divides the storage system into multiple circular buffers organized in a hierarchical structure with record tables at different levels. Each circular buffer handles a specific time range and is organized into record tables that can be independently accessed. This segmentation allows the system to store large amounts of network data while maintaining fast access times by directing queries to specific segments rather than searching the entire dataset.
2Quantity of substance
If more network data is stored for future analysis, then more analysis capability is available, but conventional systems cannot rapidly retrieve the data
Solution Approach 1:
The patent implements preliminary organization of network data into a hierarchical structure with location indices and time-based circular buffers before retrieval is needed. Data is pre-segmented into record tables organized by time periods and locations, with indices that map directly to storage positions. This preliminary action enables rapid retrieval by eliminating the need for complex searches during actual data access operations.
3Measurement precision
If data is stored in detail for all network locations and time periods, then complete analysis is possible, but storage efficiency decreases
Solution Approach 1:
The patent applies local quality by organizing data into location-specific circular buffers and time-period-specific record tables. Each location has its own dedicated storage structure, and data is retained at full detail only for recent time periods. Older data is either archived with reduced detail or overwritten by newer data in the circular buffer structure. This approach maintains complete analysis capability for recent data while efficiently managing storage space through localized, time-aware data retention policies.
Data Source
AI summary
Network data is stored and retrieved from a network data repository configured for rapid data access and efficient usage of storage space. The network data repository includes a plurality of flow record folders. A flow record folder includes a location index and one or more circular buffers of record tables, each circular buffer corresponding to network data collected from a particular location. Network data is aggregated with more or less detail in different flow record folders, and network data in different flow record folders is retained for varying amounts of time.