Automated Notebook Processing for Cross-Cloud Job Execution
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Conventional systems for executing business projects in engineering teams require manual intervention, making it difficult to audit input parameters, analyze project requirements, and suggest improvements, leading to project delays, cost overruns, and poor product quality, especially in cross-cloud environments.
Innovation Solution
An automated notebook processing system with a decision force assistant and engine that parses job requests, launches virtual machines, and executes web-based notebooks, enabling automated job execution and output generation, thereby streamlining project execution and reducing manual intervention.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Ease of operation
If manual intervention is used to execute jobs in conventional systems, then users can control job execution steps, but auditing of input parameters becomes difficult and productivity decreases
Solution Approach 1:
The system enables self-service automation where the automated assistant independently executes jobs by automatically parsing job requests, launching virtual machines, fetching input files, running notebooks, and generating outputs without requiring manual user intervention at each step, thereby maintaining ease of operation while significantly improving productivity
2Adaptability or versatility
If conventional systems require domain knowledge for workflow automation, then complex jobs can be executed, but accessibility for users without expertise decreases
Solution Approach 1:
The automated assistant acts as an intermediary that bridges the gap between users without domain knowledge and complex workflow automation requirements. It handles the complexity of parsing job requests, launching virtual machines, managing cloud resources, and executing notebooks internally, while users simply need to submit job requests through a user-friendly interface, making the system accessible to users regardless of their technical expertise
3Reliability
If data scientists and engineers collaborate on job execution, then expertise is leveraged, but back-and-forth communication increases and self-service usage decreases
Solution Approach 1:
The system implements self-service automation where the automated assistant independently handles the entire job execution workflow including parsing requests, launching virtual machines, fetching input files from external storage repositories, running notebooks sequentially, and generating outputs. This eliminates the need for back-and-forth collaboration between data scientists and engineers while maintaining job execution quality through automated best practices and validation
4Device complexity
If conventional systems use standard methods for job execution, then simplicity is maintained, but ability to analyze project requirements and suggest improvements is lost
Solution Approach 1:
The automated assistant performs preliminary actions by automatically analyzing project requirements before job execution. It parses job requests to understand project context, identifies required resources, launches appropriate virtual machines, and prepares the execution environment in advance. This preliminary analysis capability enables the system to suggest improvements and optimize job execution without requiring users to manually analyze requirements, maintaining simplicity while adding advanced analytical capabilities
Data Source
AI summary
A system for notebook processing to handle job execution in cross-cloud environment is disclosed. A decision force assistant to receive one or more job requests representative of execution of one or more projects, parses the one or more job requests received; a decision force engine launches one or more virtual machines on a cloud-based platform, sends one or more job instructions associated with the one or more job requests to the decision force assistant, enables the decision force assistant to fetch at least one input file corresponding to the one or more job instructions; a job execution engine runs one or more web-based notebooks in a sequential manner, enables the decision force assistant to fetch the at least one input file for execution of the one or more job requests on the one or more web-based notebooks, generates a job associated output, to generate a job execution status.