CRON Route Scheduling for Distributed Java ETL Execution
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing ETL systems face challenges in managing large numbers of backend systems for data extraction, transformation, and loading, particularly when dealing with hundreds of thousands of sources and thousands of user requests, which are resource-intensive, prone to user errors, and lack fault tolerance and redundancy, leading to processing and security issues.
Innovation Solution
A CRON-based routing process schedules and executes Java code excerpts across fault-tolerant load-balanced distributed servers, using a routing table to identify classes and methods in Jar files for execution at specified times, transforming and loading data into frontend storage without running complete Java programs.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Productivity
If complete Java programs are executed for each ETL task, then data extraction and transformation can be performed, but resource consumption increases and processing efficiency decreases
Solution Approach 1:
The patent extracts and executes only the necessary code snippets or specific methods from Java programs rather than running complete programs. This is achieved by using a code execution engine that can dynamically load and execute specific code segments, thereby reducing resource consumption while maintaining ETL functionality.
Solution Approach 2:
The patent segments Java programs into smaller, executable code snippets or methods that can be independently executed. This segmentation allows the system to run only the specific portions of code needed for data extraction, transformation, and loading operations, rather than executing entire programs, thus improving efficiency and reducing resource usage.
2Ease of operation
If manual scheduling of ETL jobs is performed, then job execution can be controlled, but labor intensity increases and user errors occur
Solution Approach 1:
The patent implements automated scheduling mechanisms that allow the system to self-manage ETL job execution without manual intervention. The scheduling engine automatically determines when and how to execute code snippets based on predefined rules, data source availability, and system resources, eliminating manual scheduling labor while maintaining control.
Solution Approach 2:
The patent employs preliminary configuration of scheduling parameters and execution rules before ETL jobs are triggered. By pre-defining scheduling policies, resource allocation rules, and execution priorities, the system automates the scheduling process while reducing the likelihood of user errors during actual job execution.
3Productivity
If standalone programs are used for each extraction task, then data extraction can be performed, but system complexity increases and coordination becomes difficult
Solution Approach 1:
The patent merges multiple standalone extraction programs into a unified code execution platform. Instead of running separate programs for each ETL task, the system uses a centralized code execution engine that can dynamically load, execute, and manage code snippets from various sources, thereby reducing system complexity while maintaining extraction capabilities.
Solution Approach 2:
The patent creates a universal code execution environment that can handle multiple types of data extraction, transformation, and loading operations through a single platform. This multi-functional engine can execute different code snippets for various ETL tasks, eliminating the need for separate standalone programs and simplifying system architecture.
4Reliability
If fault tolerance mechanisms are implemented in traditional ETL systems, then system reliability improves, but processing overhead increases
Solution Approach 1:
The patent implements fault tolerance mechanisms that prepare backup execution environments and error handling protocols in advance. By pre-configuring fallback options and monitoring system state before failures occur, the system can quickly respond to errors without significant processing delays, maintaining both reliability and efficiency.
Data Source
AI summary
A routing process schedules and executes Java code excerpts across backend servers. A route table contains routes to be executed in JVM namespaces on servers and identifies for each route a Jar file containing applicable classes and methods to be executed at a CRON time. During execution, methods are retrieved from each applicable class and are executed to extract data from backend devices. The servers transform the extracted data into frontend data and load it onto frontend storage devices. Only the classes and methods in the Jar file for a route need be executed. Complete Java programs need not be run.