Formatted Long-Term Storage for Streaming Data via Parquet Tiers
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Current streaming data storage systems face challenges in efficiently accessing and analyzing historical data due to the use of binary formats that do not preserve the structure of stream events, requiring additional ETL jobs and increasing storage and I/O operations.
Innovation Solution
A formatted long-term storage mechanism that tiers streaming events using open formats like Apache Parquet files, allowing for direct access via both stream and object APIs, eliminating the need for additional ETL jobs and reducing disk I/O and data movement by translating streaming events into objects and preserving their structure.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Quantity of substance
If binary format is used for long-term storage of streaming data, then storage efficiency is improved, but data accessibility and structure preservation deteriorate
Solution Approach 1:
The patent segments streaming data into discrete events with preserved schemas, organizing them into structured formats (Parquet, Avro, ORC) that maintain data integrity while enabling efficient storage and retrieval. This segmentation allows the system to balance storage efficiency with data accessibility by creating manageable, structured units.
Solution Approach 2:
The patent changes the data format parameter from binary to structured columnar formats (Parquet, Avro, ORC). This parameter change enables both efficient storage through compression and easy accessibility through schema preservation, allowing queries and analytics to operate directly on stored data without additional ETL processes.
2Ease of operation
If additional ETL jobs are implemented to access historical data, then data accessibility is improved, but system complexity and processing time increase
Solution Approach 1:
The patent implements self-service by designing the storage system to natively support both streaming writes and batch/interactive reads through its structured format. The system serves its own data access needs directly without requiring external ETL jobs, reducing system complexity while maintaining high data accessibility through unified storage semantics.
3Quantity of substance
If data is stored in binary format, then storage density is improved, but query performance and analytics capability deteriorate
Solution Approach 1:
The patent changes the storage format parameter from binary to columnar formats (Parquet, Avro, ORC) that provide both high storage density through compression and excellent query performance through columnar storage architecture. This parameter change enables efficient analytics operations while maintaining compact storage.
4Ease of operation
If streaming data is processed through multiple systems (streaming storage and object storage), then data accessibility is improved, but data movement and I/O operations increase
Solution Approach 1:
The patent merges streaming storage and object storage into a unified system where the same structured format serves both purposes. This consolidation eliminates redundant data movement between separate systems, reducing I/O operations and energy loss while maintaining comprehensive data accessibility through a single storage infrastructure.
Data Source
AI summary
A system of managing data, comprising a streaming data storage system operating on a processor that causes the processor to receive streaming data and to process the streaming data in response to an append command, a long-term storage system operating on a processor that causes the processor to receive the processed streaming data from the streaming data storage system and to generate a chunk of data, an object storage system operating on a processor that causes the processor to receive the processed streaming data from the streaming data storage system and to generate an object of data and a tables system operating on a processor that causes the processor to receive the processed streaming data from the streaming data storage system and to generate a table of data.