S3-Compatible Logging Platform Using Object Chunking
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Public cloud-computing platform users lack a cost-effective solution for real-time streaming of logging data from virtual machines and containers to Amazon S3 data storage, with existing solutions being 500 times more expensive and requiring complex ETL pipelines.
Innovation Solution
A method and system for building an AI-powered logging, streaming, and analytics platform that uses any S3 compatible store as primary storage, involving data ingestion, metadata storage in virtual machines or containers, packaging data units, breaking objects into chunks, and laying data out by time, while maintaining primary data in an object store and using S3 compatible storage to reduce costs and processing latencies.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Speed
If data is streamed from VMs and containers to vendor cloud-computing platforms, then real-time logging capability is achieved, but cost increases by 500 times
Solution Approach 1:
The patent uses inexpensive S3-compatible object storage as the primary data store instead of expensive vendor cloud platforms. Data is packaged into objects and stored in S3 buckets, eliminating the need for costly proprietary infrastructure while maintaining real-time streaming capabilities through direct S3 integration.
Solution Approach 2:
The patent introduces an intermediary packaging layer that converts streaming data into S3-compatible objects. This packaging mechanism acts as a mediator between the data source (VMs/containers) and the storage destination (S3), enabling cost-effective real-time logging without requiring expensive vendor-specific platforms.
2Productivity
If ETL pipelines are used to process logging data, then data can be stored and processed, but system complexity increases significantly
Solution Approach 1:
The patent extracts the complex ETL processing logic and replaces it with simple S3-compatible object storage operations. Data is packaged directly into objects and stored in S3 buckets without requiring intermediate transformation pipelines, eliminating the complexity of ETL while maintaining data processing capability through S3's native operations.
Solution Approach 2:
The patent uses S3-compatible object storage as a universal platform that performs multiple functions: data ingestion, storage, transformation, and analysis. This multi-functional approach replaces the need for separate ETL pipeline components, reducing system complexity while maintaining full data processing capability.
3Reliability
If data is stored in vendor cloud platforms, then centralized logging is achieved, but processing latency increases
Solution Approach 1:
The patent inverts the traditional centralized logging architecture by using S3-compatible object storage as the primary data store instead of using it as a secondary archive. This inversion enables real-time access to logging data with lower latency while maintaining centralized storage, as S3 operations can be performed directly without data movement overhead.
Data Source
AI summary
In one aspect, a computerized method includes the step of building an information technology operations analytics (ITOA) stack. The method includes the step of ingesting and storing data at rest. The method includes the step of storing a set of metadata of the ingested data in a virtual machine or a container. The method includes the step of maintaining a set of primary data of the ingested data with an object store. The method includes the step of packaging one or more data units of the set of primary data. The method includes the step of writing over objects at select intervals. The method includes the step of breaking objects into chunks to reduce overwrites. The method includes the step of laying the data out as partitioned by time. The method includes the step of creating one or more application boundaries at the time of ingesting.