Intelligent Big Data System for Model Training and Prediction
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Users of intelligent big data systems require extensive knowledge of deep learning algorithms and software/hardware architecture to build a platform for model training and prediction, leading to high development costs and knowledge barriers.
Innovation Solution
An intelligent big data system with a data warehouse, distributed storage, and a cluster of computing nodes that includes a resource scheduling component, model training/prediction components, and supports heterogeneous hardware like CPU, GPU, and FPGA, allowing users to perform model training and prediction without building a custom platform, using a unified Linux environment and REST API for service integration.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Adaptability or versatility
If users build their own software and hardware platform for model training and prediction, then they can have full control and customization, but the knowledge requirements and development costs increase significantly
Solution Approach 1:
The patent introduces a cloud-based intelligent big data platform as an intermediary between users and the complex underlying infrastructure. Users interact with simplified APIs and pre-configured services rather than directly managing hardware and software platforms. The platform abstracts away the complexity of deep learning framework deployment, GPU resource management, and model training orchestration while providing full customization capabilities through configuration files and parameter settings.
2Adaptability or versatility
If users build their own software and hardware platform for model training and prediction, then they can have full control and customization, but the development costs increase significantly
Solution Approach 1:
The platform implements self-service capabilities where users can independently configure, deploy, and manage their own model training and prediction workflows through web-based interfaces and API calls. The system automatically handles resource allocation, framework installation, and environment configuration without requiring users to manually build infrastructure. This eliminates the need for expensive dedicated platform development teams while maintaining full customization through user-configurable parameters.
3Manufacturing precision
If users need to understand deep learning algorithm and software hardware architecture, then they can optimize model performance, but the knowledge barrier increases
Solution Approach 1:
The patent segments the complex model training process into distinct, manageable components: data preprocessing, model selection, training configuration, execution, and evaluation. Each component can be independently configured and optimized. Users can start with default configurations for immediate results and progressively dive deeper into specific segments (such as adjusting hyperparameters or selecting different loss functions) only when needed, rather than requiring comprehensive knowledge of the entire deep learning stack.
4Ease of operation
If a unified platform is provided for model training and prediction, then knowledge requirements and development costs are reduced, but the system complexity increases
Solution Approach 1:
The patent implements a universal platform that handles multiple functions through a single unified architecture: data storage, model training, model evaluation, prediction serving, and resource management. The system supports multiple deep learning frameworks (TensorFlow, PyTorch, MXNet), various hardware accelerators (GPU, FPGA, ASIC), and different deployment scenarios through a common interface and configuration system. This multi-functionality is achieved through modular design where core components can be configured for different purposes without requiring separate systems.
Data Source
AI summary
A system comprises: a data warehouse, a storage device and a cluster including a plurality of computing nodes; the data warehouse is configured to store task data obtained from the user; at least one computing node in the cluster includes a resource scheduling component, and is configured to perform resource scheduling for the task and determine a computing node executing the task; the computing node executing the task comprises a model training component and/or a prediction component; the model training component is configured to, according to task data, invoke a corresponding type of learning model from the storage device; use sample data and training target included in the task data to train the learning model, to obtain the prediction model corresponding to the task and store the prediction model in the storage device; the prediction component is configured to obtain a prediction result output by the prediction model.