Distributed Data Preloading Metadata Segmentation
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
In distributed computing environments, existing data preloading methods face inefficiencies due to frequent metadata movement and redundant loading, especially in virtualization setups where data is processed across multiple computers with non-uniform memory access, leading to increased overhead and decreased performance.
Innovation Solution
The solution involves managing local and remote preloading metadata separately, selecting local preloading targets based on the current CPU context, and asynchronously starting local preloading tasks, while checking for redundancy with synchronously loaded data to prevent unnecessary loading.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If data is preloaded in distributed computing environments using existing methods, then data availability is improved, but metadata movement overhead increases excessively
Solution Approach 1:
The patent segments preloading tasks into local preloading tasks and remote preloading tasks based on the location of data relative to the CPU. Local preloading metadata is stored in local memory pages while remote preloading metadata is stored in remote memory pages, preventing unnecessary metadata movement between memory pages and reducing overhead.
Solution Approach 2:
The patent applies local quality by distinguishing between local and remote data sources and managing their metadata separately. Local preloading operations use local memory pages for metadata storage, while remote operations use remote memory pages, optimizing performance based on the specific characteristics of each data source location.
2Productivity
If preloading tasks are executed without redundancy checks, then data loading speed is improved, but redundant loading occurs increasing overhead
Solution Approach 1:
The patent performs preliminary actions by checking whether data is already loaded into memory before executing preloading tasks. The system checks the local preloading metadata to determine if the target data is already present, and only proceeds with preloading if the data is not already loaded, thereby avoiding redundant operations.
Solution Approach 2:
The patent implements feedback mechanisms by continuously monitoring the state of loaded data in memory and using this information to control preloading operations. The system provides feedback about whether data is already present before initiating preloading, dynamically adjusting preloading behavior based on current memory state.
3Quantity of substance
If local and remote preloading metadata are stored in the same memory page, then memory usage is optimized, but frequent metadata movement occurs decreasing efficiency
Solution Approach 1:
The patent segments metadata storage into separate memory pages based on the location of data sources. Local preloading metadata is stored in local memory pages while remote preloading metadata is stored in remote memory pages, preventing frequent metadata movement between memory pages and improving access efficiency.
Solution Approach 2:
The patent introduces an intermediary mechanism by using separate memory pages as intermediaries for different types of preloading metadata. This separation acts as an intermediary layer that prevents direct interaction and movement between local and remote metadata, reducing overhead.
Data Source
AI summary
Disclosed herein are an apparatus for preloading data in a distributed computing environment and a method using the same. The method includes selecting a local preloading target that each of multiple computers connected over a network is to preload into the local memory thereof, registering a local preloading task corresponding to the local preloading target in local preloading metadata, and asynchronously starting the local preloading task at a preset time based on the local preloading metadata. The local preloading metadata is stored in a page other than the page in which remote preloading metadata for managing a remote preloading task is stored.