Parallel Batch Database Streaming Interface
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing database systems face inefficiencies in processing large data streams, as they require maintaining long connections to servers or excessive client storage, especially in scenarios like mapping applications where data records exceed client memory limits, and lack efficient parallel processing interfaces for multiple queries.
Innovation Solution
Implementing a streaming interface that allows multiple parallel database queries by retrieving small batches of data records using primary keys, where primary key retrieval and batch processing tasks operate in parallel to minimize server resource usage and client memory requirements, enabling sequential processing of records without modifying client-side operations.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Ease of operation
If stream data record processing is implemented by maintaining a long connection to the database, then sequential processing of records is enabled, but server resources are excessively consumed
Solution Approach 1:
The patent segments the stream processing task into multiple parallel batch processing operations. Instead of maintaining a single long-lived connection for sequential processing, the system divides the data stream into batches and processes them in parallel using multiple database connections, thereby reducing the resource burden on the server while maintaining stream processing functionality.
Solution Approach 2:
The patent implements dynamic connection management where multiple database connections are created and managed dynamically based on processing needs. Rather than maintaining a static long connection, the system dynamically establishes multiple connections for parallel batch processing and manages their lifecycle to optimize server resource usage.
2Reliability
If all desired data records are read prior to processing at the client, then processing can proceed without server connection, but client resources are excessively consumed
Solution Approach 1:
The patent segments the data retrieval and processing workload into manageable batches. Instead of loading all data records into client memory at once, the system retrieves data in smaller batches through parallel database queries, processes them incrementally, and discards processed batches, thereby reducing client memory requirements while maintaining processing reliability.
Solution Approach 2:
The patent performs preliminary actions on the server side through parallel batch processing before data reaches the client. By pre-processing data batches in parallel on the server and transmitting only processed results to the client, the system reduces the amount of data the client must store and process, thereby reducing client memory requirements while ensuring reliable processing.
3Productivity
If multiple parallel database queries are implemented, then processing efficiency is improved, but system complexity increases
Solution Approach 1:
The patent implements a universal batch processing framework that handles multiple parallel database queries through a standardized interface. This framework provides multi-functional capabilities including connection management, batch creation, parallel query execution, and result aggregation, thereby enabling efficient parallel processing while abstracting away the complexity through a unified system.
Solution Approach 2:
The patent introduces an intermediary batch processing layer that mediates between the client and multiple parallel database queries. This intermediary framework manages the complexity of parallel query coordination, connection handling, and result aggregation, thereby enabling high productivity through parallel processing while shielding the client from the underlying system complexity.
Data Source
Figure 1
Figure 2
Figure 3
AI summary
Retrieving data records from a database is provided. A query is received by a database access API requesting a number of data records stored in the database. A parallel batch processor is created for data transmittal, and the parallel batch processor is returned to a client program. A next data record responsive to the query is requested, and the next data record is removed from a list of records. The next data record is then provided from the parallel batch processor to the client program.