Application Containers for Large Scale Data Processing Overhead
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing large-scale data processing systems face inefficiencies due to the overhead requirements of virtual machines, which can limit the resources available for data-intensive applications, especially in distributed environments where multiple machines are needed to handle large data sets.
Innovation Solution
A service architecture that utilizes application containers, such as Docker or Linux containers, to share resources like the operating system kernel and provide large-scale processing frameworks like Hadoop or Spark, reducing the need for individual operating systems and minimizing overhead by using a cache service for data retrieval and sharing memory between the host and containers.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If virtual machines are used to execute large scale processing frameworks, then isolation and resource management are improved, but memory overhead and resource consumption increase significantly
Solution Approach 1:
Multiple processing frameworks share a single operating system instance through containerization, merging resource usage while maintaining isolation. The patent implements this by allowing multiple Hadoop, Spark, or other processing frameworks to run concurrently on one OS, sharing kernel resources and reducing duplicate memory consumption while preserving framework-level isolation through namespace separation.
Solution Approach 2:
A single operating system instance serves multiple processing frameworks simultaneously, making the OS universal for different data processing workloads. The patent enables one OS to host diverse frameworks (Hadoop, Spark, Presto, etc.) through container isolation, eliminating the need for separate OS instances for each framework and optimizing resource utilization across different processing needs.
2Reliability
If multiple operating systems are deployed for different processing frameworks, then framework isolation is maintained, but system complexity and resource consumption increase
Solution Approach 1:
Multiple processing frameworks are merged into a single operating system environment using container isolation mechanisms. The patent combines Hadoop, Spark, and other frameworks under one OS umbrella, using namespace isolation (PID, network, filesystem namespaces) to maintain framework boundaries while reducing system complexity by eliminating multiple OS deployments.
Solution Approach 2:
Containerization technology acts as an intermediary layer between the operating system and processing frameworks, providing isolation without requiring separate OS instances. The patent uses this intermediary mechanism to manage framework interactions, resource allocation, and isolation boundaries, simplifying system architecture compared to direct multi-OS deployment.
3Adaptability or versatility
If virtual machines with full operating systems are used, then resource allocation flexibility is improved, but startup time and resource provisioning speed decrease
Solution Approach 1:
The heavy operating system layer is extracted and shared among multiple processing frameworks, leaving only the essential framework-specific components in isolated containers. The patent extracts the OS kernel and shared libraries into a single instance, eliminating redundant OS boot sequences for each framework while maintaining flexibility through container-level resource controls and rapid container instantiation.
Data Source
AI summary
Systems, methods, and software described herein provide for enhancements to large scale data processing architectures. In one implementation, a service architecture for large scale data processing includes a host computing system, and a virtual machine executing on the host computing system. The service architecture further includes a plurality of application containers executing on the virtual machine, wherein each of the application containers comprises a large scale processing node running one or more java virtual machines.