API Management for Batch Processing Predictive Outputs
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
In batch scoring processes, software applications face challenges in accessing predictive results and metadata due to changing storage locations and formats, requiring additional logic to find and retrieve data from distributed storage systems.
Innovation Solution
An application programming interface (API) is built on top of the batch scoring engine, updating its endpoint with storage locations of input features, output predictions, and metadata, providing a fixed location for software applications to query and retrieve predictive outputs and metadata without worrying about underlying storage intricacies.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Productivity
If batch scoring results and metadata are stored in distributed storage systems with changing locations and formats, then the system can handle large data sets asynchronously, but software applications face challenges in finding and retrieving the data requiring additional logic
Solution Approach 1:
The patent introduces an intermediary component (batch scoring engine with standardized interface) that mediates between the distributed storage system and software applications. This intermediary abstracts the complexity of locating and retrieving data from distributed storage by providing a unified access mechanism, thereby resolving the contradiction between maintaining batch processing productivity and reducing application complexity.
Solution Approach 2:
The batch scoring engine is designed with universal functionality to handle multiple storage locations and formats through a single standardized interface. This multi-functional capability allows the system to access batch scoring results regardless of where they are stored or in what format, eliminating the need for application-specific retrieval logic while maintaining high productivity.
2Adaptability or versatility
If the format and location of model output change over time across different nodes and databases, then the system can flexibly manage storage, but applications must continuously adapt to find results
Solution Approach 1:
The standardized interface of the batch scoring engine acts as an intermediary that shields applications from storage location and format changes. This intermediary layer absorbs the complexity of adapting to different storage configurations, allowing the system to maintain high storage flexibility while preserving ease of operation for applications.
Solution Approach 2:
The system segments the data access functionality into a separate batch scoring engine component that handles all variations in storage location and format. This segmentation isolates the adaptability requirements from applications, allowing the engine to handle storage flexibility internally while presenting a simple, consistent interface to applications.
3Productivity
If metadata is stored in a different location from predictive results, then the system can organize data independently, but applications face challenges in accessing both together
Solution Approach 1:
The batch scoring engine provides universal access to both predictive results and metadata through its standardized interface. This multi-functional capability allows applications to retrieve both types of data through a single access mechanism, maintaining data organization efficiency while eliminating the need for complex metadata access logic in applications.
Data Source
AI summary
An example operation may include one or more of storing a batch scoring engine and an application programming interface (API) for the batch scoring engine, receiving a trigger to perform a batch prediction process, reading input data from a source data store and executing, via the batch scoring engine, one or more predictive models on the input data to generate a predictive output and metadata associated with the predictive output, storing the predictive output and the metadata in a target data store, and updating the API with a location of the predictive output within the target data store and a location of the metadata within the target data store.