Platform-Agnostic Data Processing Jobs Using Source Aliases
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing data processing technologies face inefficiencies when dealing with multiple data sources and processing platforms, leading to redundant efforts in creating data queries and processing jobs that are not reusable across different sources and platforms.
Innovation Solution
The development of platform-agnostic and data-source agnostic data processing systems that allow for the creation of data queries and processing jobs using aliases and code libraries, enabling transformation into specific configurations for target data sources and platforms, and allowing execution outside of interactive development environments.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If data queries and processing jobs are created specifically for each data source and platform, then execution reliability is improved, but device complexity and redundancy increase
Solution Approach 1:
The patent implements platform-agnostic data processing jobs that can be executed across multiple different data processing platforms (e.g., Spark, Flink, Beam) without requiring separate job definitions for each platform. The system uses a universal job format that translates to platform-specific implementations, allowing a single job to serve multiple platforms and data sources, thereby reducing redundancy while maintaining execution reliability through platform-specific optimization.
2Measurement precision
If platform-specific data processing jobs are created for each data source, then data access precision is improved, but loss of time increases due to redundant creation efforts
Solution Approach 1:
The system pre-establishes a library of platform-specific code templates and data source connectors that can be reused when creating data processing jobs. When a new job is needed, the system retrieves appropriate pre-built components from the library and combines them with the user's business logic, significantly reducing the time required to create platform-specific jobs while maintaining the precision needed for accurate data access across different sources.
3Manufacturing precision
If data queries are customized for each data source, then manufacturing precision of data processing is improved, but ease of manufacture decreases due to individual preferences and redundancies
Solution Approach 1:
The patent segments the data processing job into distinct modular components: platform-agnostic business logic, platform-specific execution code, and data source-specific connectors. This segmentation allows users to focus on creating only the business logic portion using intuitive, high-level syntax, while the system automatically handles the more complex platform-specific and data source-specific portions through standardized templates, thereby improving ease of manufacture without sacrificing processing precision.
Data Source
AI summary
Data queries that are agnostic to any particular data source may include a data source alias. The data source alias may be replaced with a data source identifier to obtain a data query configured for a target data source. Data processing jobs may be agnostic to any particular data processing platform. A data processing job may include a data processing task that is agnostic to any particular data processing platform. A code library may provide platform-specific code configured to implement a data processing task on a data processing platform. A data query configured for a particular data source and a data processing task configured for a particular data processing platform may be used to create a data processing job. Configurations that restrict execution of a data processing job to execution via an interactive development environment may be removed to allow its execution directly at the data processing platform itself.