Sensor Dataset Compression Using Suffix Trees and Timestamp Models
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing data compression techniques are inefficient for identifying and classifying data based on sources, leading to cumbersome search mechanisms in databases, especially when dealing with large volumes of business and sensor data.
Innovation Solution
The implementation of a data compression model that uses suffix tree algorithms to determine data compression logic based on attributes such as sensor node identifiers and timestamps, optimizing memory usage and search efficiency by encoding frequently repetitive patterns and storing compressed datasets in an in-memory data store.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Quantity of substance
If known data compression techniques are used without classification, then data storage is achieved, but search efficiency deteriorates and data identification becomes cumbersome
Solution Approach 1:
The patent segments the data storage system by creating separate compressed data stores for different data sources (sensor data, business data, social media data). Each data source has its own compression model and storage structure, enabling efficient targeted searches without scanning entire datasets. This segmentation resolves the contradiction by organizing data into manageable segments that can be independently optimized and searched.
Solution Approach 2:
The patent introduces metadata as an intermediary layer between the compressed data and search operations. Metadata includes data source identifiers, compression parameters, and indexing information that enable efficient search without decompressing the entire dataset. This intermediary structure allows the system to maintain high compression ratios while providing fast search capabilities through metadata filtering.
2Volume of stationary object
If data is compressed without source classification, then storage space is reduced, but data identification and retrieval become difficult
Solution Approach 1:
The patent applies different compression techniques and metadata structures tailored to each data source's characteristics. Sensor data receives specialized compression with timestamp indexing, while business data uses different compression parameters. This local optimization maintains high compression efficiency for each data type while preserving source-specific identification capabilities through customized metadata schemas.
Solution Approach 2:
The patent creates compressed copies of the original data with embedded metadata that preserves identification information. The compression process generates simplified representations that retain essential identifying features (data source, timestamps, key attributes) while removing redundant information. This copying approach reduces storage requirements while maintaining data identification capabilities through the preserved metadata copies.
3Use of energy by moving object
If comprehensive compression is applied to all data, then memory usage decreases, but search complexity increases
Solution Approach 1:
The patent implements dynamic compression where the compression level and technique are adjusted based on data source, access patterns, and resource availability. Frequently accessed data uses lighter compression with more metadata, while less accessed data uses aggressive compression. This dynamic approach optimizes the balance between memory usage and search complexity by adapting compression strategies to actual system conditions and access requirements.
Data Source
AI summary
Methods and system are disclosed that compress a dataset and optimize the dataset for searching data. In one aspect, based on attributes associated with a unified dataset, the dataset including sensor node identifier data, sensor node timestamp data and sensor measurement data are determined. A suffix tree algorithm may be executed to determine data compression logic to compress the dataset including the sensor node identifier data and sensor node measurement data. A sensor node timestamp data compression model may be executed to compress the dataset associated with the sensor timestamp data. The compressed datasets including the sensor node identifier data, the sensor node timestamp data and the sensor measurement data may be stored in an in-memory data store.