Metadata-Driven Multilayer Processing for High-Frequency Data Streams
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Conventional data analytics systems are not configured to dynamically provide and flexibly operate a data analytics system operating environment, relying on traditional cloud-based or server-centric infrastructures that are inflexible, resource-intensive, and require continuous maintenance, limiting their ability to handle high-frequency data ingestion and stream processing.
Innovation Solution
A multilayer processing engine is implemented, comprising an event layer, metadata layer, and multi-tier processing layer, operating via a serverless distributed computing system, which dynamically scales infrastructure, generates processing code on-demand, and supports ETL processes without maintenance, enabling efficient handling of high-frequency data.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If conventional cloud-based or server-centric infrastructures are used, then data analytics systems can operate with dedicated resources and fixed bandwidth, but the systems become inflexible, resource-intensive, and require continuous maintenance
Solution Approach 1:
The patent implements a serverless architecture where computing resources are dynamically allocated and scaled based on actual workload demands. The system automatically adjusts infrastructure capacity up or down without manual intervention, transforming the static resource allocation of conventional systems into a dynamic model that adapts to changing data analytics requirements.
Solution Approach 2:
The system employs automated resource provisioning and management capabilities that enable self-service operations. The infrastructure automatically monitors workload conditions, provisions computing resources when needed, and releases resources when no longer required, eliminating the need for continuous manual maintenance and configuration.
2Stability of the object's composition
If traditional cloud-based infrastructures with dedicated resources are used, then data analytics systems can maintain fixed bandwidth and stable operations, but they cannot dynamically scale to changing workloads
Solution Approach 1:
The serverless architecture enables dynamic scaling of data processing throughput by automatically adjusting computing resource allocation based on real-time workload conditions. When data ingestion rates increase, the system provisions additional processing capacity to maintain throughput, and releases resources when demand decreases, achieving both stability and high productivity.
3Power
If server-centric infrastructures are used, then data analytics systems can operate with predetermined computing capacity, but they require continuous maintenance tasks and are not cost-effective
Solution Approach 1:
The system implements self-service automation for resource provisioning, monitoring, and management. Automated orchestration handles workload distribution, resource allocation, and system configuration without requiring manual maintenance tasks, reducing operational overhead while maintaining adequate computing capacity for data analytics workloads.
Solution Approach 2:
The serverless architecture employs ephemeral computing resources that are provisioned on-demand for specific tasks and automatically released afterward. Instead of maintaining permanent server infrastructure, the system uses temporary, automatically managed computing capacity that is allocated only when needed, reducing both cost and maintenance requirements.
4Device complexity
If conventional data analytics systems operate on static servers, then infrastructure configuration is simple, but the systems cannot handle high-frequency data ingestion and stream processing efficiently
Solution Approach 1:
The system dynamically scales computing resources to handle high-frequency data ingestion and stream processing workloads. The serverless architecture automatically provisions additional processing capacity when data arrival rates increase, enabling efficient handling of high-volume streaming data while maintaining simple infrastructure management through automated resource orchestration.
Data Source
AI summary
Methods, systems, and computer storage media for providing a multilayer processing engine of a multilayer processing system. The multilayer processing engine supports an event layer, a metadata layer, and a multi-tier processing layer. The metadata layer can refer to a functional layer that operates via a sequential hierarchy of functional layers (i.e., event layer and multi-tier processing layer) to analyze incoming event streams and configure a downstream processing configuration. The metadata layer provides for dynamic metadata-based configuration of downstream processing of data associated with the event layer and the multi-tier processing layer. The multilayer processing system can be a data analytics system—operating via a serverless distributed computing system. The data analytics system implements the multilayer processing engine as a serverless data analytics management engine for processing high frequency data at scale based on dynamically-generated processing code—generated based on a downstream processing configuration—that supports automatically processing the data.