Heterogeneous Data Platform Indexing Batch and Streaming Data
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Managing multiple data formats in a data platform is challenging due to format-specific idiosyncrasies, leading to increased system complexity, inefficient resource utilization, and reduced data accessibility.
Innovation Solution
A heterogeneous data platform processes and stores both batch and streaming data formats by queuing and aggregating streaming data, generating metadata, and storing data in association with an index, allowing for efficient retrieval and access across various formats.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Adaptability or versatility
If multiple data formats are managed by a data platform, then data accessibility is improved, but system complexity increases
Solution Approach 1:
The patent segments the heterogeneous data into distinct categories (batch data and streaming data) and processes each category through dedicated pipelines. Batch data is processed via batch computation engines while streaming data is handled by streaming computation engines, allowing each segment to be managed with format-appropriate techniques without increasing overall system complexity.
Solution Approach 2:
The patent introduces a metadata layer as an intermediary between the heterogeneous data formats and the query processing system. This metadata layer provides a unified interface that abstracts away format-specific details, allowing queries to be processed uniformly across different data formats without requiring the system to directly manage format-specific complexities.
2Reliability
If multiple data formats are managed by separate data platforms, then format-specific processing is improved, but resource utilization efficiency deteriorates
Solution Approach 1:
The patent merges multiple data platforms that handle different formats into a single unified data platform. This consolidation allows batch and streaming data to coexist in the same system, sharing common infrastructure resources such as storage, computation engines, and query processing capabilities, thereby improving resource utilization efficiency while maintaining format-specific processing through dedicated pipelines.
Solution Approach 2:
The patent creates a universal data platform architecture that can handle multiple data formats through a common interface. The platform uses a unified metadata schema and common query processing mechanisms that work across batch and streaming data, allowing a single platform to perform multiple functions that previously required separate specialized platforms.
3Manufacturing precision
If format-specific techniques are used for each data format, then data processing accuracy is improved, but operational efficiency deteriorates
Solution Approach 1:
The patent changes the parameter representation by introducing a unified metadata schema that describes both batch and streaming data using consistent parameters and data types. This allows the system to maintain format-specific processing accuracy through appropriate computation engines while improving operational efficiency by allowing uniform data definition, validation, and query operations across all data formats.
Data Source
AI summary
In examples, streaming data is received from a data source (e.g., by an edge device associated with a data platform) and is queued and aggregated. Batch data may similarly be received from a data source. The batch data and the aggregated streaming data may be processed to generate metadata accordingly. The data and metadata may be provided to the data platform, where the metadata may be used to update an index and the data may be stored in association with the index. In some instances, the data may be stored in chunks to facilitate subsequent retrieval of the data. In response to a request for the data, the index may be used to identify relevant data, which may include data associated with batch and/or streaming data formats, thereby enabling the client device to access data having any of a variety of formats from the data platform.