Data Loading via Distributed File System Gateways

Resolve Bottlenecks,
Find Innovative SolutionsGenerate Solutions

Solution Overview

Problem

The challenge of efficiently loading data from external distributed file systems, such as Hadoop, into database management systems (DBMS) is exacerbated by the need for sorting and inter-node data transfer, which slows down the process, especially when dealing with large volumes of data and No Primary Index (NoPI) tables.

Innovation Solution

The proposed solution involves configuring multiple clients to acquire specific portions of data from an external distributed file system and interact with specific gateways of the DBMS, where each gateway passes the data directly to access module processors (AMPs) on the same node, eliminating inter-node data transfer and using NoPI tables that do not require sorting, thereby allowing for faster data loading.

Engineering Contradictions & Design Principles

VSEngineering Contradiction Analysis

1Speed

If data is loaded using traditional methods with sorting and inter-node transfer, then data can be loaded into Primary Index tables, but data loading speed deteriorates due to sorting requirements and cross-node transfer overhead

Engineering Contradiction:

Improvedata loading speedVSAvoidtime for sorting and inter-node transfer

Core Design Contradiction:

SpeedVSLoss of time

Solution Approach 1:

The patent segments the data loading process by having multiple clients acquire specific portions of data from the external file system and interact with specific gateways. Each gateway passes data directly to local AMPs without inter-node transfer, dividing the loading task across multiple independent pathways that operate in parallel.

Inventive Principle:

Principle #1Segmentation

Solution Approach 2:

Instead of loading data into Primary Index tables that require sorting, the patent inverts the approach by loading directly into NoPI tables that do not require sorting. This reverses the traditional loading paradigm, sacrificing the ability to quickly find rows by Primary Index column in exchange for dramatically faster loading speeds.

Inventive Principle:

Principle #13The other way round (Inversion)

2Measurement precision

If Primary Index tables are used for data loading, then data can be quickly looked up by Primary Index column, but data loading performance deteriorates due to sorting requirements on each AMP

Engineering Contradiction:

Improvelookup efficiency by Primary Index columnVSAvoiddata loading throughput

Core Design Contradiction:

Measurement precisionVSProductivity

Solution Approach 1:

The patent applies inversion by choosing NoPI tables instead of Primary Index tables for the loading operation. This reverses the traditional approach where PI tables are the default choice, thereby prioritizing loading speed over lookup efficiency during the data ingestion phase.

Inventive Principle:

Principle #13The other way round (Inversion)

Solution Approach 2:

The patent changes the table structure parameter from Primary Index to No Primary Index. This parameter change eliminates the sorting requirement that bottlenecks data loading, allowing AMPs to load data directly without the overhead of computing hash values and sorting rows.

Inventive Principle:

Principle #35Parameter changes

3Productivity

If multiple clients load data concurrently to the same DBMS, then data loading capacity increases, but data transfer efficiency deteriorates due to inter-node data transfer overhead

Engineering Contradiction:

Improveconcurrent data loading capacityVSAvoidnetwork bandwidth consumed by inter-node transfer

Core Design Contradiction:

ProductivityVSLoss of energy

Solution Approach 1:

Inventive Principle:

Principle #1Segmentation

Solution Approach 2:

The patent implements local quality by having gateways pass data directly to AMPs controlled by that specific gateway on the same node. This localizes the data transfer within the same node, eliminating the need for energy-consuming inter-node network transfers while maintaining concurrent loading capacity.

Inventive Principle:

Principle #3Local quality

Data Source

PatentUS8938480B2Techniques for fast loading of data from an external distributed file system to a database management system

Publication Date: 2015.01.20 TERADATA CORP

AI summary

Techniques for fast loading data from an external distributed file system (DFS) to a database management system (DBMS) are provided. Assigning a portion of data from a file located in the DFS to each of a plurality of DFS clients. Configuring each DFS client to interact with a specific gateway from a plurality of gateways associated with the DBMS. Each gateway configured to send its portion of the file received from its designated client to every access module processor (AMP) controlled by that gateway. Each AMP processes a portion of the data received from the gateway to load the data into a table of the DBMS.