Recursive Query Prefetching for Batch Processing Efficiency
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
In batch processing, data to be accessed is often determined during iterative processing, making it impossible to prefetch necessary data beforehand, leading to inefficient database access and decreased program execution efficiency.
Innovation Solution
A system that generates and executes recursive queries to prefetch data from a database before program execution, using an initial query to set up an initial table, a recursive query to generate intermediate tables, and a final query to output the final table, allowing for efficient data retrieval.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Adaptability or versatility
If data is accessed every time iterative processing is performed in batch processing, then the program can dynamically determine and process required data, but the number of database accesses becomes very large and processing efficiency decreases
Solution Approach 1:
The system performs preliminary analysis of the program to identify and prefetch all data that will be needed during iterative processing before the program actually executes. This includes analyzing loop structures, query patterns, and data dependencies to determine the complete set of required data in advance, then loading this data into memory beforehand to eliminate repeated database accesses during execution.
2Productivity
If data is read in advance prior to starting iterative processing, then database access efficiency improves, but data to be accessed cannot be known when it is determined during the process of iterative processing
Solution Approach 1:
The system uses feedback from program analysis to iteratively refine the prefetching strategy. By analyzing the program's logic, loop structures, and data access patterns, the system determines which data will be needed dynamically during execution and adjusts the prefetching scope accordingly, ensuring all potentially required data is loaded in advance while avoiding unnecessary data loading.
Solution Approach 2:
The system changes the parameter of data access from dynamic database queries during execution to static data loading before execution. By transforming the program's data access pattern through analysis, the system converts dynamic queries into predetermined data sets that can be loaded once, changing the state from frequent random access to a single bulk load operation.
3Productivity
If recursive queries are generated and executed prior to program execution, then data retrieval speed and program execution efficiency improve, but the system complexity increases
Solution Approach 1:
The system introduces an intermediary data structure (temporary tables) between the database and the iterative processing program. The recursive queries populate these intermediate tables with all required data before program execution, serving as a buffer that decouples the complex prefetching logic from the simple program execution, thereby managing system complexity while maintaining performance benefits.
Data Source
AI summary
In order to improve the efficiency of execution of a program by prefetching data necessary to execute the program, a system is provided that causes a computer to execute a recursive query prior to a program being subjected to prefetching. This system detects from iterative processing in the program a query to generate a resultant table by selecting a record that satisfies a selection condition from a target table. The system generates an initial query to generate an initial table that includes values of variables that are set prior to starting the iterative processing in the program. Furthermore, the system generates a recursive query to generate the next intermediate table that is referred to in the next and later cycles of the iterative processing, in the recursive query corresponding to each cycle of the iterative processing that is sequentially performed, from an intermediate table that includes resultant tables generated by a target query in the preceding cycles of the iterative processing and the target table. Furthermore, the system generates a final query to generate a final table from the intermediate table sequentially generated by the recursive query.