Stream Processing Deployment Using Precompiled Dependency Libraries
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing stream processing systems face high latency and computational inefficiencies due to the need for deploying application code with its dependencies as 'fat jars', which slows down deployment and stresses network resources, especially in environments that require frequent and concurrent updates.
Innovation Solution
The technique involves deploying application code without its dependencies to a container initialized with precompiled libraries, allowing the code to link with dependencies at runtime, eliminating the need for recompilation and reducing deployment time significantly.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If application code is deployed with its dependencies as 'fat jars', then all required libraries are available at runtime, but deployment time increases and network resources are stressed
Solution Approach 1:
The system pre-compiles dependency libraries into a precompiled repository before deployment. When deploying application code, the system identifies required dependencies and retrieves only those specific precompiled libraries from the repository, rather than deploying all dependencies as in traditional fat jar approaches. This preliminary preparation of dependencies enables fast, targeted retrieval during deployment.
Solution Approach 2:
The system segments the dependency set into individual compilable units that are precompiled separately and stored in a repository. Instead of bundling all dependencies together as a single fat jar, each dependency is compiled independently and can be retrieved on-demand. This segmentation allows the deployment system to fetch only the specific segments (dependencies) needed for a given application, reducing deployment time and network stress.
2Reliability
If application code is deployed with its dependencies as 'fat jars', then all required libraries are available at runtime, but the size of deployed code increases to 100 MB to 1 GB
Solution Approach 1:
The system extracts the dependency retrieval process from the traditional fat jar model. Instead of embedding all dependencies within the application package, the system extracts dependencies into a separate precompiled repository. At runtime, the application code retrieves only the specific dependencies it needs from this external repository, rather than carrying all possible dependencies. This extraction dramatically reduces the size of deployed code while ensuring required libraries are available.
3Adaptability or versatility
If frequent and concurrent updates are deployed using traditional methods, then code modifications can be applied, but network resources are stressed and deployment efficiency decreases
Solution Approach 1:
The system performs preliminary compilation of all possible dependencies before deployment operations. This pre-compilation creates a ready-to-use repository of compilable units that can be rapidly retrieved during frequent updates. When concurrent updates are deployed, the system efficiently retrieves precompiled dependencies rather than performing compilation during deployment, maintaining high productivity even with frequent code modifications.
Data Source
AI summary
The technology disclosed provides a novel and innovative technique for compact deployment of application code to stream processing systems. In particular, the technology disclosed relates to obviating the need of accompanying application code with its dependencies during deployment (i.e., creating fat jars) by operating a stream processing system within a container defined over worker nodes of whole machines and initializing the worker nodes with precompiled dependency libraries having precompiled classes. Accordingly, the application code is deployed to the container without its dependencies, and, once deployed, the application code is linked with the locally stored precompiled dependencies at runtime. In implementations, the application code is deployed to the container running the stream processing system between 300 milliseconds and 6 seconds. This is drastically faster than existing deployment techniques that take anywhere between 5 to 15 minutes for deployment.