Decoupled Hadoop Storage and Processing Allocation
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Processing and storing large genome datasets efficiently is challenging due to the inefficiency of existing systems, which often lead to underutilization of resources and lack of sharing among researchers within the same institution, resulting in idle systems and inefficient resource allocation.
Innovation Solution
A distributed processing system that decouples storage and processing resources, allowing for the allocation of virtual resources using automation and orchestration tools like vCloud Automation Center and Pivotal HD, enabling simultaneous servicing of multiple requests while maintaining data confidentiality and cost-effectiveness through user-based resource allocation and sharing options.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Ease of operation
If researchers build their own systems for processing genome data, then each researcher can process their own data, but the systems sit idle most of the time leading to resource underutilization
Solution Approach 1:
Multiple individual researcher systems are merged into a single shared distributed processing system. The system allows different researchers to submit processing requests that are queued and executed on the same infrastructure, ensuring continuous utilization while maintaining individual data processing capabilities through virtualization and resource allocation mechanisms.
Solution Approach 2:
The shared processing system is designed to serve multiple researchers and multiple types of genome data processing tasks simultaneously. It provides universal access to processing resources through a common interface while supporting diverse computational workloads, making the system adaptable to various research needs without requiring separate dedicated systems.
2Productivity
If distributed systems are used to handle very large datasets, then processing capacity increases, but resource allocation becomes inefficient and systems remain idle
Solution Approach 1:
The system implements dynamic resource allocation where processing resources are allocated based on real-time demand from multiple researchers. The workload manager continuously monitors system state and distributes processing tasks dynamically across available resources, ensuring that computational power is fully utilized when needed and preventing idle time when the queue is empty, thus adapting to varying research demands.
3Productivity
If new systems are built to process and store very large datasets, then processing capability improves, but resource sharing and collaboration are reduced
Solution Approach 1:
The system provides universal access to multiple researchers through a common platform that supports various types of genome data processing and storage operations. It enables collaboration by allowing researchers to share data, workflows, and computational resources while maintaining individual project isolation, thus enhancing both processing capability and collaborative adaptability simultaneously.
Data Source
AI summary
It is decided whether to increase a total amount of storage in a pool of Hadoop storage and whether to increase a total amount of processing in a pool of Hadoop processing. If it is decided to increase the total amount of storage and not increase the total amount of processing, the total amount of storage is increased without increasing processing. If it is decided to not increase the total amount of storage and increase the total amount of processing, the total amount of processing is increased without increasing storage. In response to receiving a request to perform a process on a set of data, processing is allocated from the pool of processing and storage is allocated from the pool of storage where the allocated processing and storage are used to perform the process on the set of data.