Big Data Cluster Shell Isolation via Segmented Memory
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Current big data cluster systems face challenges in managing independent data processing environments, where local variables are not accessible across different user shells, leading to inefficiencies in task distribution and data management across worker systems.
Innovation Solution
A cluster system architecture that includes a master system for task distribution and data management, where each worker system maintains independent local memory spaces for local variables associated with specific user shells, and a tracking mechanism to manage shared data distribution across multiple worker systems.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Adaptability or versatility
If worker systems share a common data environment, then data accessibility is improved, but data isolation between user shells deteriorates
Solution Approach 1:
The patent segments the data environment by creating separate local memory spaces for each user shell while maintaining a shared data structure. Each worker system maintains independence through its own local variables and data context, preventing cross-shell contamination while enabling controlled sharing of the underlying data structure through the cluster system's task distribution mechanism.
2Ease of operation
If local variables are shared across user shells, then variable accessibility is improved, but task execution correctness deteriorates
Solution Approach 1:
The patent applies local quality by allowing variables to have different accessibility properties depending on their location and context. Local variables maintain shell-specific isolation for correctness-critical data, while the shared data structure provides controlled accessibility for read-only or explicitly shared data, creating a differentiated variable accessibility model that prioritizes correctness.
3Speed
If worker systems access shared data directly, then data retrieval speed is improved, but data management complexity deteriorates
Solution Approach 1:
The patent introduces the cluster system as an intermediary layer between worker systems and the shared data structure. This mediator manages data distribution, version control, and access coordination, enabling fast direct access to shared data while centralizing management complexity in the cluster system that coordinates updates and ensures consistency across all worker systems.
4Reliability
If independent local memory spaces are maintained for each user shell, then data isolation is improved, but memory resource utilization deteriorates
Solution Approach 1:
The patent merges the underlying data storage into a shared data structure that can be efficiently utilized by multiple worker systems, while maintaining separate local memory spaces only for shell-specific variables. This combining approach allows worker systems to share common data without duplication while preserving isolation for user-specific context, optimizing overall memory resource utilization.
Data Source
AI summary
A cluster system includes an interface and a processor. The interface is to receive a request from a user associated with one of a plurality of shells. The processor is to determine a plurality of tasks to respond to the request; determine a local set of data and a shared set of data for a task of the plurality of tasks, wherein the local set of data is associated with the one of the plurality of shells; and provide the task, a local set indication, and a shared set indication to a worker associated with the task, wherein the local set indication refers to the local set of data and the shared set indication refers to the shared set of data.