Distributed Computing Platform Application Storage Format Mapping
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Current distributed computing platforms, such as Hadoop-based systems, face inefficiencies in processing large datasets due to suboptimal data storage and retrieval methods, leading to increased resource usage and processing time.
Innovation Solution
A method is introduced where a distributed computing platform generates a mapping between applications and their appropriate storage formats, storing this information in a name node to direct data storage across multiple data nodes, ensuring data is stored in formats optimized for each application, thereby improving processing efficiency and resource utilization.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Productivity
If data is stored in a distributed file system without application-specific storage format mapping, then the system structure remains simple, but processing efficiency and resource utilization deteriorate
Solution Approach 1:
The system performs preliminary action by pre-establishing mappings between applications and their optimal storage formats in the name node before data storage operations. This allows the system to automatically select appropriate storage formats without real-time complex decisions, improving processing efficiency while keeping the architecture relatively simple.
Solution Approach 2:
The name node acts as an intermediary component that maintains and manages the mappings between applications and storage formats. It mediates between applications and data nodes, enabling efficient data placement without requiring direct complex interactions between applications and the storage system, thus improving productivity with minimal added complexity.
2Loss of time
If data is stored without considering application-specific storage formats, then the system operation remains simple, but processing time increases
Solution Approach 1:
The system implements self-service by automatically using pre-established application-storage format mappings to optimize data placement without requiring manual intervention or complex operational decisions. The name node automatically selects appropriate storage formats based on the application type, reducing processing time while maintaining operational simplicity.
3Use of energy by moving object
If a mapping system between applications and storage formats is implemented, then resource utilization improves, but system complexity increases
Solution Approach 1:
The name node performs multiple functions: it manages the distributed file system, maintains application-storage format mappings, and directs data placement decisions. This multi-functionality allows the system to improve resource utilization through intelligent mapping without adding separate dedicated components, thus improving resource efficiency with minimal increase in overall system complexity.
Data Source
AI summary
A method and system for a distributing and computing platform are provided that generates a mapping between each of a plurality of applications and a plurality of application storage formats, wherein each application has a mapping to at least one application storage format; stores the mapping of the application and the application storage format in the name node; obtains a data node storage format information for each of the plurality of data nodes; stores the data node storage format information for each of the plurality of data nodes in the name node; and stores data generated by at least one of the plurality of applications in at least one data node based on the mapping information stored in the name node. Through the above embodiment, the distributing and computing platform may store the data in the data node with the storage format which is proper for the application.