Hybrid Data Integration via Machine Learning Automation
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Current data processing systems face challenges in efficiently preparing and integrating diverse data sets, particularly in combining structured and streaming data, which often requires significant operations and is not seamlessly usable across different applications.
Innovation Solution
A computer system equipped with machine learning functionality that automates or recommends data preparation operations such as filtering, joining, aggregating, and enriching data sets, enabling the integration of structured and streaming data into a hybrid data set accessible through various channels, including periodic summaries and real-time streams, using multiple processing pipelines.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Productivity
If manual data preparation operations are performed, then data can be processed and integrated, but the time and effort required for data preparation is significant
Solution Approach 1:
The system performs data preparation operations automatically without requiring manual intervention. The machine learning model analyzes incoming data sets and autonomously determines and executes appropriate preparation operations, allowing the system to serve itself rather than requiring user involvement in the data preparation process.
Solution Approach 2:
The system performs data preparation operations in advance before the data is needed for analysis. By automatically preparing data sets as they are received and before they are required for reporting or analysis, the system eliminates delays and ensures data is ready when needed without requiring manual pre-processing.
2Adaptability or versatility
If diverse data sets are integrated, then comprehensive data availability is achieved, but the complexity of integration operations increases
Solution Approach 1:
The machine learning model automatically analyzes the structure and characteristics of incoming data sets and autonomously determines the appropriate integration operations. The system serves itself by selecting and executing the necessary joins, filters, and transformations without requiring manual configuration of complex integration operations.
Solution Approach 2:
The system adapts its integration approach by changing parameters such as join types, aggregation levels, and data transformation methods based on the specific characteristics of the incoming data sets. This allows the system to handle diverse data types dynamically by adjusting integration parameters rather than using fixed complex operations.
3Adaptability or versatility
If streaming data is combined with structured data, then real-time and historical data are available, but the difficulty of seamless integration increases
Solution Approach 1:
The machine learning model automatically detects whether incoming data is streaming or structured and autonomously determines the appropriate integration method. The system serves itself by selecting the correct integration approach based on data type detection, eliminating the need for manual configuration of complex hybrid integration operations.
Solution Approach 2:
The system segments the data integration process into separate handling paths for streaming and structured data. By dividing the integration logic into distinct segments that handle different data types appropriately, the system can seamlessly combine both types without creating a single complex monolithic integration process.
4Productivity
If automated machine learning operations are applied, then data preparation time is reduced, but the complexity of the system increases
Solution Approach 1:
The machine learning model performs data preparation operations autonomously without requiring manual intervention or complex system configuration. The system serves itself by automatically analyzing data characteristics and executing appropriate preparation operations, which simplifies the overall system architecture compared to manual processes.
Solution Approach 2:
The system replaces manual mechanical data preparation operations with automated machine learning algorithms. By substituting human-based processes with algorithmic automation, the system increases processing speed while actually reducing operational complexity, as the machine learning model handles all preparation tasks automatically without requiring manual intervention.
Data Source
AI summary
Methods, systems, and apparatus, including computer programs encoded on computer-storage media, for enhanced preparation and integration of data sets. In some implementations, data indicating user input that identifies a first data set that includes streaming data and a second data set that includes non-streaming data is received. The first data set and the second data set are integrated to generate a hybrid data set. The data processing system provides access to the hybrid data set through a (i) non-streaming access channel that provides a periodically-refreshed summary of both the streaming data and the non-streaming data and (ii) a streaming access channel that provides a data stream based on combined data of the first data set and the second data set. One or more application programing interfaces are provided. The one or more application programming interfaces allow at least one client device to access the non-streaming access channel and the streaming access channel.