Mobile Hadoop Cluster Segmentation for Field Analytics
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Conventional Hadoop clusters are not mobile due to their large size and high power consumption, limiting their usability for applications requiring real-time data collection and analysis in the field, such as signal intelligence gathering.
Innovation Solution
A mobile Hadoop cluster with a small footprint that can be contained within a vehicle, capable of storing a snapshot of data from a main cluster and processing analytic tasks autonomously in real-time without connection to the main cluster, integrating newly acquired data seamlessly upon reconnection.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Quantity of substance
If a conventional Hadoop cluster is used to provide large-scale data storage and processing, then data storage capacity and processing capability are improved, but the system size and power consumption increase significantly, making it non-mobile
Solution Approach 1:
The system is divided into two segments: a stationary main Hadoop cluster that provides large-scale data storage, and a mobile cluster that can be transported in vehicles. The mobile cluster contains a simplified Hadoop implementation with reduced data nodes and metadata nodes, enabling it to function autonomously while mobile while maintaining essential data processing capabilities.
Solution Approach 2:
The mobile cluster creates and maintains a copy (snapshot) of the main cluster's data and metadata structures. This copying approach allows the mobile cluster to have a smaller footprint while still providing data storage and processing capabilities by replicating essential portions of the main cluster's data structure.
2Quantity of substance
If a conventional Hadoop cluster is used to provide large-scale data storage, then data storage capacity is improved, but power consumption increases significantly
Solution Approach 1:
The system divides data storage functions between the main cluster (stationary, high capacity) and the mobile cluster (portable, reduced capacity). The mobile cluster handles only essential data storage and processing while mobile, significantly reducing power consumption compared to running a full-scale Hadoop cluster in a mobile platform.
Solution Approach 2:
The mobile cluster maintains a copy of essential data and metadata from the main cluster, allowing it to provide data processing capabilities with minimal power consumption by operating on a reduced dataset rather than requiring full cluster capacity.
3Ease of operation
If data collection equipment operates in the field without connection to a Hadoop cluster, then mobility is maintained, but real-time data analysis capability is lost
Solution Approach 1:
The system provides offline data analysis capability by segmenting the Hadoop functionality into a mobile cluster that can operate autonomously. The mobile cluster includes necessary data nodes and metadata nodes to perform data processing and analysis locally, enabling real-time analytics while mobile without requiring connection to the main cluster.
4Weight of moving object
If the mobile cluster stores a snapshot of main cluster data, then data storage capacity is reduced for mobility, but autonomous operation capability is improved
Solution Approach 1:
The mobile cluster creates a snapshot copy of the main cluster's data and metadata structures. This copying enables the mobile cluster to operate autonomously by having local copies of necessary data, while the snapshot approach keeps the data footprint manageable for mobile deployment.
Data Source
AI summary
Techniques for mobile clusters for collecting telemetry data and processing analytic tasks, are disclosed herein. The mobile cluster includes a processor, a plurality of data nodes and an analysis module. The data nodes receive and store a snapshot of at least a portion of data stored in a main Hadoop storage cluster and real-time acquired data received from a data capturing device. The analysis module is operatively coupled to the processor to process analytic tasks based on the snapshot and the real-time acquired data when the storage cluster is not connected to the main storage cluster.