Transformed Data Processing for Real-Time Analytics
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
The large volume of raw data stored on slow storage devices hinders real-time analytics due to increased access times, making it challenging to perform efficient analytics on massive datasets, such as time series data, in IT infrastructure and healthcare settings.
Innovation Solution
A data processing system that combines ingest-time and query-time processing, applying transformations like summarization, domain transforms, and filtering during ingest-time to produce compact, transformed data, which reduces query-time processing and allows for faster analytics without sacrificing accuracy, using modules like summarization, domain transforms, and filtering.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Speed
If raw data is stored on slow storage devices, then storage capacity is maintained, but query processing speed deteriorates
Solution Approach 1:
The system performs preliminary transformations on raw data during the ingest phase, creating transformed data representations (summarized, filtered, aggregated) before queries are executed. This preliminary action eliminates the need to process raw data at query time, significantly reducing query processing time and data access delays.
Solution Approach 2:
The data is segmented into multiple representations: raw data for complete accuracy, transformed data for efficient querying, and aggregated data for summary operations. Each segment serves specific query types, allowing the system to quickly select the appropriate data representation rather than processing all raw data for every query.
2Productivity
If transformations are applied during ingest-time, then query-time processing is reduced, but system complexity increases
Solution Approach 1:
Complex transformations (summarization, filtering, aggregation, domain transforms) are performed in advance during data ingestion and storage, converting raw data into multiple transformed representations. This shifts computational complexity from query-time to ingest-time, enabling fast analytics operations without sacrificing processing depth.
Solution Approach 2:
The system changes data parameters through various transformations including summarization (changing data granularity), filtering (changing data selection criteria), aggregation (changing data consolidation level), and domain transforms (changing data representation). These parameter changes create multiple data views optimized for different query types.
3Loss of time
If transformed data is stored in higher-speed memory, then query access time is reduced, but storage cost increases
Solution Approach 1:
Data is segmented across multiple storage tiers and formats: raw data in cost-effective storage, transformed data in higher-speed memory for frequent access, and aggregated data for summary operations. This segmentation allows the system to store transformed representations of data in faster memory without duplicating entire datasets, optimizing the balance between access speed and storage capacity.
Data Source
AI summary
A transformation on raw data is applied to produce transformed data, where the transformation includes at least one selected from among a summary of the raw data or a transform of the raw data between different domains. In response to a query to access data, the query is processed using the transformed data.