Data Movement Engine for Database to Distributed File System Integration
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
The integration of relational databases and distributed file systems, such as Apache Hadoop Distributed File System (HDFS), poses challenges due to differences in data access and security mechanisms, making it difficult to efficiently move and manage data across these systems.
Innovation Solution
A data movement engine that creates activities overriding existing security, defines source and target tables, and provides security credentials to move data from relational databases to distributed file systems, allowing users to access the data through database-like table structures while ensuring security and governance.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If data is moved from relational databases to distributed file systems using traditional security mechanisms, then security is maintained, but data movement efficiency and user accessibility deteriorate
Solution Approach 1:
The patent segments security management by creating activity-level security contexts that are separate from traditional user-level security. Each activity becomes an independent security domain with its own credentials and permissions, allowing data movement operations to proceed with activity-specific security validation rather than requiring complex user permission checks across heterogeneous systems.
Solution Approach 2:
The patent introduces an intermediary activity context that mediates between the relational database security model and the distributed file system security model. This activity context acts as a security translator, converting between different security mechanisms and providing a unified security framework that enables efficient data movement without compromising security.
2Adaptability or versatility
If heterogeneous data sources are integrated, then data accessibility is improved, but system complexity increases
Solution Approach 1:
The patent creates a universal activity context framework that can handle multiple data sources (relational databases, distributed file systems, and future sources) through a common interface. The activity context serves as a multi-functional container that manages security, credentials, and data movement operations across different system types, eliminating the need for separate integration logic for each data source.
Solution Approach 2:
The activity context acts as an intermediary layer between diverse data sources and users, providing a unified view and access mechanism. This intermediary abstracts the underlying complexity of heterogeneous systems, allowing users to access data from multiple sources through consistent table-like interfaces without directly interacting with the complexity of each source system.
3Adaptability or versatility
If security credentials are overridden for data movement, then data movement flexibility is improved, but security control deteriorates
Solution Approach 1:
The patent implements dynamic security credential assignment where the activity context can adopt different security credentials based on the specific data movement operation. The activity context dynamically switches between using the author's credentials, the user's credentials, or activity-specific credentials, allowing flexible security adaptation while maintaining controlled access through the activity-level security framework.
Solution Approach 2:
The patent changes the security parameter from static user-level permissions to dynamic activity-level credentials. The activity context can modify security parameters by selecting different credential sets for different operations, enabling flexible data movement while maintaining security through structured credential management and activity-based access control.
Data Source
AI summary
Provided are techniques for data movement. An activity is created that overrides existing security, that is associated with a user, that defines one or more source tables in a relational database and one or more target tables that are available to the user for the activity, and that provides security credentials of an author of the activity for use by the user, wherein the one or more target tables represent data in one or more target files in a file system. The activity is executed using the security credentials of the author to move data in a source table of the one or more source tables to a target file of the one or more target files, wherein a target table of the one or more target tables is defined over the target file. Access is provided to the user to the moved data using the target table.