Universal Namenode for Tiered Storage Query Engines

Resolve Bottlenecks,
Find Innovative SolutionsGenerate Solutions

Solution Overview

Problem

Existing data management systems face inefficiencies in storage space utilization and data consistency when performing analytics across different, geographically diverse databases with incompatible file systems, requiring an improved method to manage datasets effectively.

Innovation Solution

A universal namenode system that connects clients to diverse storage systems via universal protocols, allowing for unified data management across different file systems, including HDFS and NFS, and enables query execution on multiple tiers of storage systems without data copying, using a catalog to track file locations and adjust storage based on usage levels.

Engineering Contradictions & Design Principles

VSEngineering Contradiction Analysis

1Adaptability or versatility

If data is copied from multiple databases to a central database for analytics, then data analytics can be performed across different databases, but storage space is inefficiently used and data consistency issues arise

Engineering Contradiction:

Improvedata analytics capabilityVSAvoidstorage space utilization

Core Design Contradiction:

Adaptability or versatilityVSQuantity of substance

Solution Approach 1:

The patent introduces a central database as an intermediary that receives data from multiple source databases through replication agents. This mediator enables cross-database analytics without requiring direct copying between all source databases, thereby improving storage efficiency while maintaining analytics capability. The replication agents act as intermediaries that selectively replicate only necessary data portions.

Inventive Principle:

Principle #24Intermediary (Mediator)

Solution Approach 2:

The patent divides the data replication process into segments by using replication agents at each source database that independently identify and replicate only the data portions needed for analytics. This segmentation prevents full data copying, improving storage space utilization while maintaining the ability to perform analytics across distributed databases.

Inventive Principle:

Principle #1Segmentation

2Adaptability or versatility

If data is copied from multiple databases to a central database, then unified data analytics is enabled, but data consistency between databases deteriorates

Engineering Contradiction:

Improveunified data analyticsVSAvoiddata consistency

Core Design Contradiction:

Adaptability or versatilityVSReliability

Solution Approach 1:

The replication agents perform preliminary actions by identifying and replicating data before it is needed for analytics. This advance replication ensures that the central database has up-to-date data while maintaining consistency, as the replication process can be controlled and synchronized with source database updates.

Inventive Principle:

Principle #10Preliminary action

Solution Approach 2:

The system implements feedback mechanisms where replication agents monitor source databases for changes and automatically replicate updated data to the central database. This continuous feedback loop maintains data consistency between source and central databases, ensuring reliability while enabling unified analytics.

Inventive Principle:

Principle #23Feedback

3Adaptability or versatility

If different file systems are used in diverse storage systems, then storage flexibility is improved, but compatibility between systems deteriorates

Engineering Contradiction:

Improvestorage flexibilityVSAvoidsystem compatibility

Core Design Contradiction:

Adaptability or versatilityVSEase of operation

Solution Approach 1:

The patent implements a universal interface layer that enables the central database to interact with multiple different file systems (HDFS, NFS, local filesystems) through a common protocol. This universal interface maintains storage flexibility by supporting diverse file systems while improving compatibility by providing a unified access method for analytics operations.

Inventive Principle:

Principle #6Universality (Multi-functionality)

4Quantity of substance

If data is stored in geographically diverse locations, then storage capacity and accessibility are improved, but system complexity increases

Engineering Contradiction:

Improvestorage capacityVSAvoidsystem complexity

Core Design Contradiction:

Quantity of substanceVSDevice complexity

Solution Approach 1:

The patent merges multiple geographically distributed storage systems into a unified logical structure through the central database. By combining access to diverse storage locations through a single centralized interface, the system maintains high storage capacity and accessibility while reducing operational complexity, as users interact with one unified system rather than multiple distributed systems.

Inventive Principle:

Principle #5Merging (Combining)

Data Source

PatentUS11514046B2Tiering with pluggable storage system for parallel query engines

Publication Date: 2022.11.29 EMC IP HLDG CO LLC

AI summary

A method, article of manufacture, and apparatus for managing data. In some embodiments, this includes determining a usage level of a file, wherein the file is stored in a first storage system, moving the file to a second storage system based on the determined usage level of the file updating location information in a catalog based on the movement of the file, and performing at least a portion of a query on the file after updating location information in the catalog.