Big Data Query Translation Using Function Mapping
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing systems face challenges in migrating data from one big data format to another, such as when moving from on-premises systems to cloud-based systems, where commands or queries used pre-migration become unusable post-migration, restricting data interaction.
Innovation Solution
A script conversion system that utilizes a mapping of functions to convert queries and scripts from a first big data format to a second format, employing direct and indirect function mappings, and additional formatting to ensure compatibility, while generating a report on conversion results.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Adaptability or versatility
If data is migrated from one big data system to another, then data accessibility and new system capabilities are improved, but query compatibility and operational continuity deteriorate
Solution Approach 1:
The patent introduces a query translation layer that acts as an intermediary between the source and target big data systems. This translation layer converts queries from the source system's query language to the target system's query language, enabling compatibility without requiring changes to the underlying systems or manual query rewriting. The intermediary resolves the contradiction by maintaining query functionality across system migrations.
Solution Approach 2:
The patent employs parameter mapping and transformation techniques to adapt query parameters from one system to another. By dynamically adjusting query parameters, data types, and syntax structures based on the target system's requirements, the translation layer maintains query effectiveness while accommodating differences between systems. This parameter transformation approach preserves operational continuity during migration.
2Manufacturing precision
If manual query conversion is performed during data migration, then query accuracy can be maintained, but migration time and operational efficiency deteriorate
Solution Approach 1:
The patent replaces the manual mechanical process of query rewriting with an automated query translation system. The translation layer automatically parses, transforms, and generates equivalent queries in the target system's language without human intervention. This automation maintains query accuracy through systematic translation rules while eliminating the time-consuming manual conversion process.
Solution Approach 2:
The patent performs preliminary analysis and mapping of query structures, data types, and syntax rules before actual migration occurs. By pre-configuring translation mappings and validation rules, the system prepares the translation layer in advance, enabling rapid and accurate query conversion during migration without requiring time-consuming manual analysis of each query.
3Duration of action of stationary object
If comprehensive query translation is implemented, then operational continuity is improved, but system complexity and development effort deteriorate
Solution Approach 1:
The patent divides the query translation functionality into separate, modular translation layers for different query types, data formats, and system configurations. Each translation module handles specific aspects of query conversion independently, making the overall system more manageable and maintainable. This segmentation reduces complexity by organizing translation logic into discrete, reusable components rather than a monolithic system.
Data Source
AI summary
Disclosed herein are system, method, and computer program product embodiments for converting queries and scripts used in big data systems. A script conversion system converts a script and its queries from a first big data format to a second big data format. The script conversion system uses a configuration mapping that includes direct and/or indirect function mappings. For the direct mapping, the script conversion system replaces the function with a new function in the second big data format. For the indirect mapping, the script conversion system constructs an equivalent function in the second big data format using information extracted from parsing queries in the first big data format. The script conversion system also formats a converted query using a project mapping and/or generates a conversion report indicating successful or unsuccessful conversions. Scripts may be converted from an on-premises data warehouse system to a cloud-based data warehouse system.