Universal Model Execution System for Small and Big Data
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing machine learning models are limited by their ability to operate only in specific data environments, either small or big data, which restricts their scalability and efficiency, requiring separate models and pre-processing steps for each environment, leading to inefficiencies in data processing and model execution.
Innovation Solution
A system and method that allows a single model to be executed in both small and big data environments using a common programming language, with a user interface to determine data size and automatically select the appropriate environment for execution, utilizing containerization and micro-services for efficient data processing and model deployment.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If separate models are built for small data and big data environments, then model execution reliability is improved, but device complexity increases
Solution Approach 1:
The patent creates a universal model building system that can execute the same machine learning model in both small data and big data environments. The system uses a single model definition that adapts to different data sizes through automated environment selection, eliminating the need for separate model versions while maintaining execution reliability across different scales.
Solution Approach 2:
The patent introduces an intermediary layer (the model building system with automated environment selection) that mediates between the model and different data environments. This intermediary automatically determines whether to execute the model in a small data or big data environment based on the input data size, simplifying the overall system architecture while ensuring reliable execution.
2Measurement precision
If separate pre-processing steps are applied for small and big data, then measurement precision is improved, but loss of time increases
Solution Approach 1:
The patent implements dynamic pre-processing that automatically adapts to the size of the input data. The system selects appropriate pre-processing steps based on whether the data is small or big, eliminating the need for manual intervention and reducing the time required to build and execute models while maintaining processing precision through context-appropriate transformations.
Solution Approach 2:
The model building system performs self-service by automatically determining the appropriate execution environment and applying suitable pre-processing steps without requiring user specification. The system autonomously analyzes the input data size and configures the processing pipeline accordingly, reducing both time loss and improving precision through automated decision-making.
3Adaptability or versatility
If a single model handles both small and big data, then adaptability is improved, but device complexity increases
Solution Approach 1:
The patent designs a universal model execution framework that can handle both small and big data with a single model definition. The system automatically selects the appropriate execution environment (small data or big data) based on the input data size, providing adaptability across different scales while maintaining a unified and relatively simple system architecture through automated environment routing.
4Ease of operation
If automated environment selection is implemented, then ease of operation is improved, but device complexity increases
Solution Approach 1:
The system implements automated environment selection that operates autonomously without requiring user intervention. The model building system automatically analyzes the input data size and selects the appropriate execution environment (small data or big data), significantly improving ease of operation. The added complexity is confined to the automated selection logic, which transparently manages the underlying architectural decisions.
Data Source
AI summary
According to some embodiments, system and methods for building a model are provided, comprising a display; a memory storing processor-executable process steps; and a processor to execute the processor-executable process steps to cause the system to: present a user interface on a display, the user interface including one or more user-entry fields to build a model, user-entry fields is associated with a selection of big data or small data for use with the model; receive at least one data source in a user-entry field associated with the model; determine if data in the data source includes big data or small data; and in response to the determination of big data or small data in the data source, execute the model with data from the data source in a big data or small data execution environment. Numerous other aspects are provided.