AI Data Collection System with On-Demand Pre-Processing
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Current AI systems face challenges in data collection and storage, particularly in network management, where large volumes of raw data are difficult to manage, leading to usability and security issues due to the need for constant monitoring and storage of extensive network data.
Innovation Solution
An AI system that collects data on demand using a predetermined data configuration, including measurement, pre-processing, and storage profiles, allowing for efficient data collection, processing, and storage only of necessary data, thereby reducing storage space requirements and enhancing data suitability and security.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Quantity of substance
If large volumes of raw network data are collected and stored for AI model development, then data availability and model learning capability are improved, but storage space requirements and security risks increase
Solution Approach 1:
The system performs preliminary data configuration and collection setup before actual data gathering begins. The data collection module is pre-configured with collection profiles that define what data to collect, where to collect from, and how to process it, enabling targeted data acquisition that avoids unnecessary storage of irrelevant data volumes
Solution Approach 2:
The system changes the parameters of data processing by applying pre-processing profiles that transform raw data into processed data suitable for AI model development. This transformation reduces the volume of data that needs to be stored while maintaining the essential characteristics needed for model training, effectively converting large volumes of raw data into smaller, more manageable processed data sets
2Quantity of substance
If large volumes of raw network data are collected and stored for AI model development, then data availability and model learning capability are improved, but security risks and management complexity increase
Solution Approach 1:
The system segments the data management process into distinct modules: data collection module, data pre-processing module, and AI model development module. Each module has specific functions and operates independently, reducing the complexity of managing large data volumes by dividing the overall system into manageable, specialized components with clear responsibilities
Solution Approach 2:
The data pre-processing module acts as an intermediary between data collection and AI model development. It receives raw data, applies processing rules, and outputs processed data suitable for model training, thereby simplifying the management burden by handling data transformation and preparation tasks centrally rather than requiring complex management across the entire system
3Duration of action of stationary object
If constant monitoring and storage of extensive network data is performed, then data completeness for AI development is improved, but usability and security issues arise
Solution Approach 1:
The system implements periodic data collection and processing rather than continuous monitoring of all network data. Data collection occurs at intervals defined by collection profiles, and processing is performed periodically on collected data batches. This periodic approach maintains data completeness for AI development while reducing the burden of constant monitoring and storage management, thereby improving usability
Solution Approach 2:
The system extracts only the necessary features and characteristics from raw network data through the data pre-processing module, rather than storing and managing complete raw data sets. By extracting essential information needed for AI model development and discarding redundant data, the system maintains data completeness for training purposes while significantly improving usability and reducing security risks
Data Source
AI summary
A method and an AI system for collecting data on demand by starting data collection based on a predetermined data configuration of data required for development of AI model when design of the AI model starts on the AI system; storing raw data collected through the data collection and generating data processed for AI model learning or machine learning (ML) by pre-processing the raw data; and completing the development of the AI model by learning and validating the AI model designed based on the raw data and/or pre-processed data are provided.