Multi-Cadence Dataset Augmentation via Automated Relational Join
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing systems are unable to combine data from different data sources with varying cadences, such as batch, real-time, and on-demand sources, limiting the ability to create a unified dataset for applications like marketing, where data is spread across multiple sources with different update frequencies.
Innovation Solution
A computing system is configured to automatically combine data from multiple data sources with different cadences by using a graphical user interface (GUI) that allows users to select and join data sources without writing code, utilizing an application that determines metadata and columns for each source, and performs relational joins to create a standardized dataset.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Quantity of substance
If data from multiple data sources with different cadences are combined, then the data surface area and unity of dataset are improved, but the system complexity and difficulty of integration increase
Solution Approach 1:
The patent introduces an intermediary layer (the computing system with metadata module and dataset module) that mediates between multiple data sources with different cadences and the final unified dataset. This intermediary automatically performs cadence reconciliation, data transformation, and joining operations, hiding the complexity from users while enabling integration of batch, real-time, and on-demand data sources.
Solution Approach 2:
The system dynamically adjusts parameters such as cadence timing and data transformation rules based on the characteristics of each data source. The metadata module stores cadence information for each data source, and the system automatically modifies data retrieval and joining parameters to accommodate different update frequencies, enabling seamless integration without manual configuration.
2Ease of operation
If automatic combination of data sources is implemented, then the ease of operation is improved, but the need for code writing increases
Solution Approach 1:
The system performs self-service by automatically discovering data sources, retrieving their metadata including cadence information, and executing the joining process without user intervention. The metadata module autonomously tracks data source characteristics, and the dataset module automatically generates joined datasets based on stored joining information, eliminating the need for users to write code while managing the complexity internally.
3Adaptability or versatility
If data sources with different cadences are integrated, then the versatility of data integration is improved, but the reliability of data synchronization decreases
Solution Approach 1:
The system dynamically adapts to different data source cadences by continuously monitoring and adjusting its data retrieval and joining operations. The metadata module stores cadence information for each data source, and the system dynamically schedules data retrieval operations to match each source's update frequency, ensuring reliable synchronization while supporting diverse cadence patterns.
Data Source
AI summary
A computing system may comprise a server system, a database, and one or more data sources having different cadences, such as a batch data source and a real-time data source. The server system may generate a first dataset based on data from the batch data source, and may generate a second dataset based on data received from the real-time data source. The server system may determine metadata associated with the real-time data source. Based on the metadata, the server system may generate a database table representation of the real-time data source. The server system may be configured to perform a relational join on the first and second datasets. Such a relational join may define a namespace that is based on the first and second datasets.