Spark DAG Caching for Cross-Application Data Sharing
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
The Spark architecture performs repeated computations and stores duplicate data due to its inability to share data among application programs, leading to reduced computing speed and efficiency.
Innovation Solution
A method is introduced where a server cluster determines and caches shareable resilient distributed datasets (RDDs) across application programs by creating directed acyclic graphs (DAGs) and storing them in main memory, reducing redundant computations and accelerating computing speed.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If Spark executes application programs with isolated driver programs and worker programs, then each application program can run independently, but data cannot be shared among application programs leading to repeated computations
Solution Approach 1:
The patent merges the isolated execution environments of multiple application programs by introducing a shared memory space where driver programs can access each other's data structures. Specifically, the driver program of the first application program accesses the directed acyclic graph (DAG) and resilient distributed datasets (RDDs) created by the driver program of the second application program through shared memory, enabling data sharing while maintaining independent execution capabilities.
Solution Approach 2:
The patent makes the memory space universal by allowing it to be shared across multiple application programs. The driver programs store their DAGs and RDDs in a common memory area that can be accessed by any driver program in the cluster, transforming previously program-specific resources into universally accessible assets that benefit all applications.
2Reliability
If Spark stores data separately for each application program, then data isolation is maintained, but duplicate data is stored consuming excessive memory resources
Solution Approach 1:
The patent combines previously separate memory spaces into a unified shared memory area. Instead of each application program having its own isolated memory for storing DAGs and RDDs, all driver programs store their data structures in a common memory space, allowing duplicate data to be stored only once and shared across multiple applications.
Solution Approach 2:
The patent eliminates the need for physical copying of data between application programs. By using shared memory, the same data structures (DAGs and RDDs) can be referenced and accessed by multiple driver programs simultaneously without creating duplicate copies, significantly reducing memory consumption.
3Reliability
If Spark performs repeated computations on the same data, then each application program processes data independently, but computing speed and efficiency are reduced
Solution Approach 1:
The patent implements preliminary action by having driver programs store their DAGs and RDDs in shared memory in advance. When subsequent application programs need the same data, they can directly access the pre-stored structures without performing redundant computation, significantly improving computing efficiency while maintaining independent processing capabilities.
Solution Approach 2:
The patent introduces a feedback mechanism where driver programs can query and access previously created DAGs and RDDs in shared memory. This feedback loop allows the system to learn from previous computations and avoid repeating the same processing work, with subsequent applications benefiting from the computational history stored in shared memory.
Data Source
AI summary
A method for processing shared data, an apparatus, and a server are provided, and relate to the field of communications technologies, so that shared data can be cached across application programs, which facilitates data sharing across applications programs, can reduce a quantity of repeated computations, and helps accelerate a computing speed of a Spark architecture. The method includes: receiving a first instruction; starting a first Spark context for a first application program, to create a DAG of the first application program, and caching the DAG of the first application program in a first area of a first server; receiving a second instruction; starting a second Spark context for a second application program reading m DAGs from the first area; and caching the to-be-cached shareable RDDs in a main memory of a second server, where the shareable RDD is an RDD included in at least two DAGs of the m DAGs.