Cloud Pipeline Data Prepositioning for Low-Latency Distributed Workflows
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Cloud-based systems experience latency issues when distributing data to geographically distributed users in production pipelines, leading to inefficiencies and frustration due to slow file migration.
Innovation Solution
The system represents the production pipeline as two flow graphs: a flow graph of tasks and a flow graph of data, proactively gathering and delivering data to users based on their tasks, and generates reduced-fidelity data versions when high-resolution data cannot be delivered in time, allowing users to work on lower resolution data while high-resolution data is uploaded.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Ease of operation
If data is distributed to geographically distributed users through cloud-based systems, then users can access data remotely, but server latency causes slow file migration and inefficiency
Solution Approach 1:
The system performs preliminary actions by proactively gathering and pre-positioning data at edge locations before users need it. The orchestration system monitors task flow graphs and data dependencies to anticipate when data will be needed, transferring files to edge servers in advance, thus eliminating wait time when users access the data.
Solution Approach 2:
The patent introduces edge servers as intermediary components between the central cloud storage and end users. These edge servers act as mediators that cache and serve data locally, reducing the distance and latency between cloud storage and geographically distributed users, thereby solving the slow file migration problem.
2Manufacturing precision
If high-resolution data is transferred to users, then data quality is maintained, but transfer time increases due to large file size
Solution Approach 1:
The system applies local quality by providing different data resolutions to different users based on their specific needs and circumstances. The orchestration system analyzes task requirements and can deliver high-resolution data to users who need it for quality-critical tasks, while providing lower-resolution versions to others, optimizing the balance between quality and transfer time.
Solution Approach 2:
The patent implements parameter changes by dynamically adjusting data resolution parameters during transfer. The system can modify file size, compression level, and quality parameters based on user preferences, network conditions, and task requirements, allowing flexible trade-offs between data quality and transfer speed.
3Reliability
If the system waits for all data to be ready before users start tasks, then data completeness is ensured, but user productivity decreases due to waiting
Solution Approach 1:
The system performs preliminary actions by pre-gathering and pre-positioning data that will be needed for upcoming tasks in the workflow. The orchestration system analyzes the task flow graph to identify data dependencies and proactively transfers required files to edge servers before users initiate their tasks, ensuring data is ready when needed without causing user waiting time.
Solution Approach 2:
The system implements feedback mechanisms where the orchestration system continuously monitors user task progress, data usage patterns, and workflow state. This feedback allows the system to dynamically adjust data gathering and transfer operations, ensuring data completeness while optimizing delivery timing to match actual user needs and maintain productivity.
Data Source
AI summary
The system efficiently distributes data in a cloud pipeline to geographically distributed users. To enable efficient distribution of resources to multiple users, the production pipeline can be represented as two flow graphs: a flow graph of tasks as well as a flow graph of data. The flow graph of tasks can indicate a sequence of tasks, departments, and users in a production pipeline. The flow graph of data can indicate location of data needed for each task in the flow graph of tasks, and location of the users assigned to the particular task. Based on the next task that needs to be executed in the flow graph of tasks, the system can proactively gather the data from the various locations and deliver the data to the users assigned to executing the next task.