Data Processing Engine Routing with Cross-Language Query Conversion
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
The integration of OLAP databases and Spark engines for real-time and offline data processing faces challenges in language compatibility and high operation and maintenance costs, with OLAP systems primarily using C++ and Spark systems in Java, leading to instability and inefficiency in handling large data volumes.
Innovation Solution
A data processing method that determines a target data processing engine based on query complexity and language differences, using a software development kit and language conversion interface to perform data queries, enabling compatibility between OLAP (C++) and Spark (Java) systems.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Adaptability or versatility
If OLAP database and Spark engine are integrated to provide both real-time and offline processing capabilities, then processing versatility is improved, but operation and maintenance cost increases
Solution Approach 1:
The patent implements a unified data processing system that can handle both real-time and offline processing requests through a single entry point (front-end node). The system determines the appropriate processing engine based on request characteristics, allowing one system to serve multiple processing needs without requiring separate maintenance tracks for OLAP and Spark engines.
2Adaptability or versatility
If OLAP system (C++) and Spark system (Java) are integrated to provide both real-time and offline processing, then processing capability is improved, but language compatibility problem arises
Solution Approach 1:
The patent introduces a language conversion interface as an intermediary component that translates query instructions between C++ (OLAP) and Java (Spark) languages. This mediator enables the two different language-based systems to communicate and cooperate without requiring direct compatibility between them, solving the language barrier while maintaining system versatility.
3Speed
If OLAP database is used for real-time processing, then processing speed is improved, but stability problem occurs when processing large amounts of data
Solution Approach 1:
The patent implements a dynamic routing mechanism that determines which processing engine to use based on the characteristics of each request. The system can switch between OLAP (for real-time processing) and Spark (for large data processing) depending on the workload, allowing the system to adapt to different data volumes and maintain both speed and stability.
Data Source
AI summary
Embodiments of the present disclosure disclose a data processing method, an electronic device, and a computer-readable medium. And the data processing method includes: receiving a data query request for a first data processing engine; determining, in a front-end node, a target data processing engine based on the data query request, where computer programming languages used by the first data processing engine and a second data processing engine are different; and when the target data processing engine is the second data processing engine, performing data query on data stored in a target database by using a preset software development kit and calling a language conversion interface based on an offline query request in the second data processing engine, where the language conversion interface is configured to receive a query instruction in a second language and call a query instruction in a first language.