Containerized Workflow Engine for Scalable Media Processing
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Developing scalable media applications that can efficiently and reliably execute custom algorithms in a high-throughput fashion is complex, requiring significant manual effort and technical knowledge, and existing solutions like distributed monoliths and generalized distributed computing frameworks face challenges such as inflexible updates and difficult OS dependency management.
Innovation Solution
A computer-implemented method using a split-map-collect workflow within containers that includes proper OS dependencies, where a split function partitions media items, a map function applies the custom algorithm, and a collect function combines results, reducing manual effort and technical knowledge needed for secure and efficient execution across distributed computing resources.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If a distributed monolith architecture is used to execute custom algorithms on media items, then the system provides integrated media processing capabilities, but the system becomes inflexible and requires complete platform rebuilds for any modifications
Solution Approach 1:
The patent segments the monolithic media processing platform into independent containerized services. Each service (ingestion, processing, delivery) is packaged as a separate container that can be independently deployed, updated, and scaled. This segmentation enables flexible modifications without requiring complete platform rebuilds while maintaining integrated capabilities through orchestration.
Solution Approach 2:
The patent implements dynamic service registration and discovery mechanisms that allow new processing services to be added to the platform without static configuration changes. The system dynamically adapts to new services through automated service registry updates and orchestration adjustments, enabling continuous evolution of the platform architecture.
2Ease of manufacture
If a generalized distributed computing framework is used to execute custom algorithms, then the system enables independent application development, but the system struggles with OS dependency management and execution environment consistency
Solution Approach 1:
The patent incorporates all operating system dependencies, libraries, and execution environments into the container image during the build stage. This preliminary action ensures that the complete execution environment is captured beforehand, eliminating runtime dependency issues and ensuring consistent execution across different compute instances regardless of their underlying OS configurations.
Solution Approach 2:
The patent creates portable container images that copy the entire execution environment including OS dependencies, application code, and configuration files. These immutable container images can be replicated across any compute instance, ensuring identical execution environments without requiring the underlying infrastructure to match specific OS requirements.
3Reliability
If manual configuration is used to build scalable media applications with custom algorithms, then the system can meet specific processing requirements, but the development process requires significant manual effort and technical knowledge
Solution Approach 1:
The patent implements self-service automation where the orchestration system automatically handles service registration, configuration generation, resource allocation, and workflow orchestration based on high-level service definitions. This eliminates the need for manual configuration of complex distributed system parameters, reducing development complexity while maintaining the ability to fulfill specific processing requirements through automated service composition.
Data Source
AI summary
In various embodiments, a workflow engine executes a custom algorithm on a media item. In operation, the workflow engine generates split specifications based on a split function included in a container image. Each split specification is associated with a different portion of the media item. Subsequently, the workflow engine generates map output files based on the split specifications and a map function included in the container image. The workflow engine then generates one or more final output file(s) based on the map output files and a collect function included in the container image. The final output file(s) are subsequently used to perform at least one of an evaluation operation on, a modification operation on, and a representation operation with respect to the media item.