Federated Data Access via Dynamic Object Transformation
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Conventional data platforms face difficulties in efficiently integrating and analyzing data from disparate source systems, leading to costly and time-consuming processes due to the need for extensive data access and handling large volumes of data, especially when accessing and manipulating objects across decentralized and self-sustained data sources.
Innovation Solution
A system-agnostic approach is implemented using a script to transform federated data into temporary and data platform objects, providing an interface for accessing and modifying these objects, which are cached and updated dynamically in response to changes in the source systems, allowing for seamless data ingestion and promotion without explicit crawling.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Productivity
If data is imported into the object model by applying an ontology to transform data, then data can be ingested into the data platform, but the process becomes costly and time-consuming
Solution Approach 1:
The system pre-applies the ontology to source data before ingestion, transforming data into the object model in advance. This preliminary transformation eliminates the need for time-consuming real-time transformation during data access operations.
Solution Approach 2:
The system creates a copy of the data in the object model format during ingestion, allowing the original source data to remain unchanged while the transformed copy is available for immediate use. This copying approach enables parallel processing and eliminates sequential transformation delays.
2Adaptability or versatility
If federated data is accessed using a front-end analytical application, then data can be retrieved from disparate source systems, but accessing and manipulating objects becomes difficult or costly
Solution Approach 1:
The object model serves as an intermediary layer between disparate source systems and the front-end analytical application. It mediates data access by providing a unified, standardized interface that simplifies object manipulation while maintaining access to diverse underlying data sources.
Solution Approach 2:
The object model provides universal access to all federated data regardless of source system. A single object-oriented interface enables the same operations to be performed on data from multiple disparate sources, eliminating the need for source-specific access methods.
3Adaptability or versatility
If data from many disparate source systems is integrated, then a unified view can be achieved, but the complexity of handling diverse data sources increases
Solution Approach 1:
The system segments the data integration process into two independent phases: (1) applying the ontology to transform source data into object model format, and (2) accessing the transformed data through the front-end application. This segmentation isolates complexity to the transformation phase while keeping the access phase simple and standardized.
Solution Approach 2:
The system changes the data representation parameters by transforming source data into a standardized object model format with consistent properties and methods. This parameter transformation simplifies data handling by providing a uniform interface regardless of the original data source or format.
Data Source
Figure 1
Figure 2
Figure 3
AI summary
Systems and methods are provided that allow federated data from various source systems to be accessed and analyzed through a data analysis platform. The federated data may be stored in different formats. The data analysis platform can receive the federated data in whatever format it has been stored at its respective source system. A script can be used to generate temporary representations (or temporary objects) for the federated data by transforming the federated data. Moreover, the temporary representations or temporary objects can be further transformed into a data analysis platform-specific format. A user of data analysis platform may access and/or manipulate either the temporary representations or objects as well as the data analysis platform-specific objects. Temporary objects can be transformed automatically into corresponding platform-specific objects when necessary to provide an enhanced capability or operation on the objects.