Runtime-Adaptive ETL for Changing Data Locations, Formats, and Schemas
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
ETL processes face challenges in handling changes to data resource locations, formats, and schemas, requiring cumbersome logic adjustments and prone to errors when interacting with catalog services.
Innovation Solution
A single instance of an ETL execution application dynamically determines data resource locations, formats, and schemas at runtime, enabling seamless execution across multiple data clusters without fixed input dependencies.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Adaptability or versatility
If ETL process logic is fixed with fixed input for location, format and schema, then the process is simple and stable, but it cannot adapt when data resource characteristics change
Solution Approach 1:
The patent implements dynamic determination of data resource location, format, and schema at runtime during ETL execution. The system queries catalog service metadata dynamically rather than using fixed pre-defined inputs, allowing the ETL process to adapt automatically to changing data characteristics without modifying the core process logic.
Solution Approach 2:
The patent introduces a catalog service as an intermediary between the ETL execution application and data resources. The catalog service provides metadata about data resources, enabling the ETL process to discover and adapt to location, format, and schema information without direct hardcoding, thus resolving the adaptability-complexity contradiction.
2Reliability
If ETL process requires connecting and authenticating to catalog service, then data resource location and format can be determined, but the process becomes more complex and error-prone
Solution Approach 1:
The patent implements self-service authentication where the ETL execution application automatically handles catalog service connection and authentication using credentials provided in the task definition. The system performs authentication transparently in the background, eliminating the need for users to manually manage connection details while maintaining reliable access to data resources.
Solution Approach 2:
The patent performs authentication and catalog service connection setup in advance during task initialization, before the actual ETL processing begins. By establishing the connection and handling authentication preliminarily, the system ensures reliability is prepared beforehand while keeping the main processing logic simple and avoiding errors during execution.
3Adaptability or versatility
If ETL process uses fixed input for data resource location and format, then the process is easy to implement, but it fails when data characteristics change
Solution Approach 1:
The patent creates a universal ETL execution application that can handle multiple data resource types and locations through a single unified interface. By using the catalog service to dynamically determine data characteristics, the system provides multi-functional capability to process various data formats and locations without requiring separate specialized processes, thus improving flexibility while maintaining ease of operation.
Data Source
AI summary
In some examples, a computing device is able to communicate with a plurality of data clusters. The computing device may determine a location, format and/or schema of data resource(s) stored in the data clusters upon execution of a process. Metadata may be searched and read to determine the location, format and/or schema of data resource(s) and those data resources may be written to other destinations within the data cluster(s). The locations of the destinations may also be determined at runtime. In some examples, parameters may be configured or specified to drive the execution of one or more operations or processes.