Large Data Store Access via GPU Address Calculation
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Accessing large data stores over communications networks is hindered by limited broadband, long access times, and the financial burden of commercially available databases, particularly in scenarios like the airline industry where data is frequently updated and accessed by multiple users simultaneously.
Innovation Solution
A method involving a first server allocating a heap of at least 100 megabytes in memory, transmitting data to a second server, and using a graphics processing unit to calculate new memory addresses for efficient data access, reducing the need for multiple licenses and improving access times by leveraging volatile memory and GPUs.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Speed
If large data stores are stored on standard hard disk drives, then storage capacity is sufficient, but access times become unacceptably long
Solution Approach 1:
The system segments the large data store into smaller chunks that can be loaded into memory as needed. Instead of accessing the entire terabyte-scale data store from disk, the system divides it into manageable portions that fit in RAM, allowing faster access to frequently used data while maintaining the ability to store large quantities of data overall.
Solution Approach 2:
The system introduces memory (RAM) as an intermediary between the hard disk drive and the processing units. Data is first loaded from the hard disk into memory, where it can be accessed much faster. This intermediary layer caches frequently accessed data, reducing the need for slow disk I/O operations while maintaining access to large data stores.
2Device complexity
If a single server acts as the requesting node for licensed databases, then license management is simplified, but the system becomes over-burdened during busy periods
Solution Approach 1:
The system merges the functions of multiple requesting nodes into a coordinated multi-server architecture. Instead of having a single overloaded server, multiple servers share the workload of accessing and processing database queries, distributing the burden while maintaining centralized license management through coordination protocols.
Solution Approach 2:
The system transitions from a single-dimensional architecture (one server) to a multi-dimensional distributed architecture. Multiple servers operate in parallel across different nodes, adding spatial distribution as a new dimension to the system. This allows license management to remain centralized while query processing is distributed across multiple dimensions of the system architecture.
3Ease of operation
If data is stored centrally for easy access, then data availability is improved, but network traffic over-burdens the system during high-demand periods
Solution Approach 1:
The system performs preliminary actions by pre-loading data into memory before it is actually needed. Frequently accessed data is cached in advance in the servers' RAM, so when queries are executed, the data is already available locally rather than requiring network transmission. This reduces network traffic while maintaining high data availability.
Solution Approach 2:
The system implements local quality by allowing each server to maintain local copies of data in its own memory. Instead of all servers needing to access data from a central remote location over the network, each server has locally optimized access to relevant data portions, reducing network dependency while maintaining centralized data management capabilities.
Data Source
AI summary
A method for facilitating access to a large data store is disclosed. The method includes reading the large data store, allocating a heap of at least 100 megabytes in a first memory, and storing the data store in the heap, wherein a memory address is associated with each memory element in the heap, and wherein each memory address comprises a base address unique to the first memory and an offset value from the base address. A second server allocates space in a second memory and stores a particular group of memory elements from the first memory in the second memory. Next, a graphics processing unit calculates new memory addresses for each of the memory elements in the particular group by adding a new base address, unique to the second memory, to the offset value of each memory address, and transmits the new memory addresses to the second memory.