Docker Operator for Multimedia Analytics in Spark RDD
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Current cluster computing frameworks face challenges in processing multimedia objects, as they are designed for textual data and integrating multimedia processing algorithms and native libraries, which requires complex adaptations and is not efficient.
Innovation Solution
Extending the cluster computing framework with multimedia processing containers using a docker operator to manage and execute multimedia processing algorithms, allowing multimedia objects to be processed independently and mapped into a resilient distributed dataset (RDD) in a textual format, leveraging pre-created executable images for secure isolated execution.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Adaptability or versatility
If cluster computing frameworks are designed for textual data only, then the framework structure remains simple and well-defined, but the capability to process multimedia objects is limited or non-existent
Solution Approach 1:
The patent introduces Docker containers as an intermediary layer between the existing cluster computing framework and multimedia processing algorithms. The framework tasks interact with Docker containers, which in turn execute the multimedia processing algorithms and native libraries. This mediator approach allows the framework to process multimedia objects without requiring direct integration complexity into the core framework structure.
Solution Approach 2:
The patent makes the cluster computing framework universal by enabling it to handle both textual data (original capability) and multimedia objects (new capability) through the same framework infrastructure. The Docker operator and containerization approach provide a unified mechanism that works for different data types, making the framework multi-functional without creating separate processing paths.
2Productivity
If multimedia processing algorithms and native libraries are integrated directly into the cluster computing framework, then processing efficiency improves, but the integration complexity and adaptation requirements increase significantly
Solution Approach 1:
Docker containers serve as an intermediary that bridges the cluster computing framework and multimedia processing algorithms. The framework tasks submit work to Docker containers, which then execute the algorithms and native libraries in isolated environments. This approach maintains processing efficiency by enabling direct algorithm execution while avoiding direct integration complexity in the framework itself.
Solution Approach 2:
The patent prepares multimedia processing algorithms and their required native libraries in advance by packaging them into pre-configured Docker images. These images are created beforehand with all necessary dependencies, libraries, and algorithms already integrated and tested. When processing is needed, the pre-prepared containers are simply deployed and executed, eliminating the need for complex real-time integration.
3Adaptability or versatility
If new frameworks are created specifically for multimedia processing, then multimedia processing capability is optimized, but the cost of developing and maintaining new frameworks increases
Solution Approach 1:
The patent extends the existing cluster computing framework's functionality to include multimedia processing, making it universal rather than creating a separate specialized framework. The same framework tasks, RDD structures, and processing pipelines that handle textual data are extended to handle multimedia objects through Docker container integration, avoiding the need for parallel framework development.
Solution Approach 2:
The system allows the existing framework to serve itself by processing multimedia data through the same task execution mechanism. Framework tasks are automatically assigned to execute Docker containers that process multimedia objects, and the results are integrated back into the framework's data structures. This self-service approach eliminates the need for separate maintenance teams and development resources for a new framework.
4Productivity
If multimedia objects are processed using traditional cluster computing methods, then platform independence is maintained, but processing efficiency and algorithm integration are compromised
Solution Approach 1:
Docker containers act as an intermediary that preserves platform independence while enabling efficient multimedia processing. The containers encapsulate all platform-specific dependencies, native libraries, and algorithm implementations within isolated environments. The host framework remains platform-agnostic, while the Docker containers provide the necessary platform-specific optimizations and efficiencies for multimedia processing.
Data Source
AI summary
A computer implemented method of mapping multimedia analytics of multimedia objects into a resilient distributed dataset (RDD), comprising one or more processors adapted to obtain an RDD of a cluster computing framework executed by a cluster comprising a plurality of computing nodes, the RDD comprises a plurality of entries each comprising a pointer to one of a plurality of multimedia objects stored in a shared storage, instruct each of a plurality of framework tasks executed by at least some members of the cluster to apply a docker operator for retrieving and executing one of a plurality of multimedia containers each associated with a respective one of the multimedia objects and comprising a multimedia processing algorithm for processing the respective multimedia object, receive from the framework tasks multimedia analytics results generated simultaneously by the multimedia containers and map the multimedia analytics results into the RDD.