Data Processing Engine Metadata Caching to Reduce Network Overhead
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing big data processing engines face performance deterioration and efficiency loss due to frequent network access for metadata, causing network delays and jitter, especially when accessing external databases.
Innovation Solution
Cache metadata from external databases into the memory of the data processing engine and establish heartbeat channels to ensure data consistency, reducing network access and maintaining efficient data processing.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If metadata is frequently accessed via network from external database catalog, then data processing engine can obtain required metadata information, but network delay and jitter occur causing performance deterioration
Solution Approach 1:
The patent caches metadata from external database catalogs into local memory before it is needed for data processing operations. This preliminary action stores frequently accessed metadata locally, eliminating the need for repeated network accesses during job execution and thereby resolving the contradiction between reliable metadata access and processing efficiency
Solution Approach 2:
The patent introduces a local catalog/memory as an intermediary layer between the data processing engine and external database catalogs. This intermediary stores metadata locally and provides it to the processing engine without requiring direct network access to external databases, thus maintaining reliability while eliminating network-induced performance degradation
2Loss of information
If catalog of external database is accessed multiple times via network, then metadata can be obtained for data processing, but network overheads increase reducing processing performance
Solution Approach 1:
The patent performs preliminary caching of metadata into local memory before data processing jobs require it. This advance preparation ensures metadata is readily available without requiring repeated network accesses, thereby eliminating network overhead while maintaining complete metadata availability for processing operations
3Adaptability or versatility
If network access is used for catalog queries, then external database metadata can be retrieved, but network unavailability impacts job execution
Solution Approach 1:
The patent introduces a local catalog as an intermediary that stores copies of external database metadata locally. This intermediary provides metadata availability independent of network status, ensuring job execution reliability while maintaining the flexibility to access diverse metadata sources through the same interface
4Stability of the object's composition
If heartbeat channel is established to check metadata updates, then metadata consistency is ensured, but additional network communication is required
Solution Approach 1:
The patent implements periodic heartbeat channels that check for metadata updates at scheduled intervals rather than continuously. This periodic approach maintains metadata consistency by detecting changes when they occur, while minimizing network communication overhead by remaining inactive between heartbeat intervals
Data Source
AI summary
A data processing method includes determining a to-be-accessed external database for a data processing job and a target access object in the external database; caching metadata of the target access object from the external database into a memory of the data processing engine; and accessing the metadata from the memory to execute the data processing job. According to the foregoing method, the metadata in the external database is cached into the memory of the data processing engine, and the metadata is accessed from the memory. This can avoid a catalog of the external database from being accessed for a plurality of times via a network.