Data Connector Retrieval Layer for Distributed Clusters
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Deploying data-intensive distributed applications across clusters of virtual machines is impractical due to latency issues when handling large data sets and accessing multiple data repositories with different data access formats.
Innovation Solution
Implementing a retrieval layer that translates data access requests from processing nodes to the appropriate format for various data repositories using data connector identifiers, creating a unified file system to facilitate access across multiple storage formats like Gluster, HDFS, and NFS repositories.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Productivity
If data-intensive distributed applications are deployed across clusters of virtual machines, then processing capability is improved, but latency increases due to feeding large data sets to the applications
Solution Approach 1:
The patent introduces a retrieval layer as an intermediary component between processing nodes and data repositories. This retrieval layer pre-loads and caches data sets in memory, acting as a buffer that eliminates the latency of feeding large data sets to applications while maintaining high processing capability across virtual machine clusters
2Adaptability or versatility
If multiple data repositories with different data access formats are accessed, then data source versatility is improved, but system complexity increases due to format incompatibilities
Solution Approach 1:
The retrieval layer is designed with multi-functional capability to handle multiple data access formats (HDFS, Gluster, NFS, S3, CloudStore) through a single unified interface. This universal design allows the system to access diverse data repositories without increasing complexity, as the retrieval layer internally manages format conversions and repository-specific protocols
Data Source
AI summary
Systems, methods, and software described herein facilitate interfacing between processing nodes and a plurality of data repositories. In one example, a method of interfacing between a processing node and a plurality of data repositories includes identifying, for the processing node, a data access request using a first data access format, wherein the data access request includes a data connector identifier. The method further includes translating the access request to a second data access format based on the data connector identifier, and identifying a data repository in the plurality of data repositories to service the data access request based on the data connector identifier. The method also provides accessing data for the data access request in the data repository via the second data access format.