Compressed Signal Pipeline With Incremental Updates and Time Segmentation
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Traditional data processing systems face challenges with high storage costs, slow retrieval speeds, and inefficient data handling, especially when dealing with large volumes of user-generated data, due to the need for complete decompression and recompression of data for every update, which significantly slows down the process and increases computational load.
Innovation Solution
A compressed compact data pipeline utilizing incremental compression techniques and a Waterfall Data Model organizes data into a hierarchical, time-segmented structure, allowing for efficient storage, retrieval, and incremental updates without full decompression, maintaining data integrity and reducing computational load.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If complete decompression and recompression is performed for every data update, then data integrity is maintained, but processing speed decreases and computational load increases
Solution Approach 1:
The patent segments the data structure into a compressed base record and incremental update records. Instead of recompressing the entire dataset on every update, only the incremental changes are recorded separately. This allows the system to maintain data integrity through structured updates while avoiding the computational overhead of complete decompression and recompression cycles.
Solution Approach 2:
The system performs preliminary compression of the base record once, then stores subsequent updates as incremental changes. This preliminary action of compressing the initial dataset and then using incremental updates eliminates the need for repeated decompression and recompression operations, significantly improving processing speed while maintaining integrity through the structured update mechanism.
2Reliability
If complete decompression and recompression is performed for every data update, then data integrity is maintained, but computational resources increase
Solution Approach 1:
The patent segments the data structure into a compressed base record and incremental update records. Instead of recompressing the entire dataset on every update, only the incremental changes are recorded separately. This allows the system to maintain data integrity through structured updates while avoiding the computational overhead of complete decompression and recompression cycles.
Solution Approach 2:
The system performs preliminary compression of the base record once, then stores subsequent updates as incremental changes. This preliminary action of compressing the initial dataset and then using incremental updates eliminates the need for repeated decompression and recompression operations, significantly improving processing speed while maintaining integrity through the structured update mechanism.
3Device complexity
If traditional separate processing and storage of each data signal is used, then data organization is simple, but storage costs increase and retrieval becomes slower
Solution Approach 1:
The patent merges multiple data signals into a single compressed record structure that maintains temporal ordering. Instead of storing and processing each data signal separately, the system combines them into a unified compressed representation with incremental updates. This merging approach reduces storage costs by eliminating redundancy while improving retrieval speed through efficient compressed data access patterns.
Solution Approach 2:
The system changes the storage parameter from individual signal storage to compressed record storage. By transforming the data representation from separate uncompressioned signals to a compressed format with incremental updates, the system achieves both cost efficiency and performance improvement. The parameter change enables faster retrieval through optimized data access patterns on the compressed structure.
Data Source
AI summary
A computer-implemented method for compressed compact data storage and processing within a cloud-based environment is disclosed. In one aspect, the method for processing data signals, includes receiving a plurality of data signals corresponding to a user, the plurality of data signals includes a plurality of user raw records at corresponding time values, compressing the plurality of data signals using an incremental compression algorithm to form a single compressed iterative record, organizing the single compressed iterative record into hierarchical segments based on predefined time intervals using a waterfall data model, and storing the single compressed iterative record in a first cloud storage system.


