Network Traffic Storage Using Resource Templates and Difference Extraction
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
The increasing volume of network traffic data poses significant storage and processing challenges, leading to high costs and reduced analysis capabilities due to the need for extensive storage and lengthy processing times, as existing methods store large amounts of duplicate data without efficient compression.
Innovation Solution
The techniques involve comparing network traffic data to previous data to identify differences, using resource templates and hashing algorithms to compress data, and indexing words in a hash table for efficient search and retrieval, allowing for the recreation of network communications with reduced storage needs.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Quantity of substance
If network traffic data is stored in traditional formats, then complete data is preserved, but storage costs and processing time increase significantly
Solution Approach 1:
The patent extracts only the essential and changing elements from network traffic data. Instead of storing complete data packets, the system identifies and stores only unique resources, their properties, and changes between requests. This extraction principle reduces storage volume while preserving the ability to recreate and analyze network communications.
Solution Approach 2:
The patent creates simplified copies of network traffic data in the form of resource templates. These templates contain only the essential structure and properties of resources (HTML, CSS, JavaScript, images, etc.) rather than storing complete data packets. The templates can be used to recreate original communications, maintaining data utility while reducing storage requirements.
2Productivity
If all network traffic data is stored for analysis, then comprehensive analysis is possible, but processing time becomes increasingly lengthy
Solution Approach 1:
The patent segments network traffic data into distinct resource types (HTML, CSS, JavaScript, images, etc.) and stores them separately in organized templates. This segmentation allows the system to process and search only relevant portions of data during analysis, rather than scanning through complete data packets, thereby improving analysis efficiency and reducing processing time.
Solution Approach 2:
The patent performs preliminary processing of network traffic data by organizing it into structured templates with indexed properties during the storage phase. This preliminary action includes parsing resources, identifying properties, and creating searchable structures in advance, which significantly accelerates subsequent analysis and processing operations.
3Quantity of substance
If storage capacity is increased to handle growing traffic, then more data can be retained, but costs become prohibitive
Solution Approach 1:
The patent applies discarding and recovering by eliminating redundant data from storage while preserving the ability to recover original communications when needed. The system discards duplicate and unchanged resource data between requests, storing only templates and their properties. When analysis is required, the system can recover complete communications by reconstructing them from the stored templates, thus reducing storage costs while maintaining data retention capacity.
4Loss of energy
If limited amounts of network traffic are stored, then storage costs are reduced, but the ability to perform useful analysis over time is reduced
Solution Approach 1:
The patent changes the parameters of data storage by transforming complete network traffic packets into compressed template representations with key properties. This parameter transformation includes storing resources in a normalized format with extracted attributes (such as resource type, size, content hash, and relationships) rather than storing raw data packets. The transformed parameters enable effective analysis with reduced storage, maintaining reliability while lowering costs.
Data Source
AI summary
In networked communication systems, a document in a communication (e.g., a response) may be similar between multiple communications involving the same resource, such that duplicate data can be discarded and not stored by a network storage system. Storage of differences in network traffic facilitates compression of storage of network traffic, thereby significantly reducing data storage. Techniques are disclosed for efficient search and retrieval of the compressed data storage. Network traffic may be compared to communications in previous network traffic to identify differences if any. Resource templates may be generated for different (e.g., new) resources identified in network traffic. Storage of the different resources identified in network traffic enables compression of network traffic. Similarity matching may be implemented to improve processing performance for compact storage of network traffic, including determining differences in network traffic for storage.