Chunked Edge Machine Learning for Early Malicious File Detection
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing systems for classifying streaming files, particularly at edge devices, are inefficient and create latency due to the need to wait for the entire file to be received before classification, which is impractical given memory constraints and leads to delayed action on potentially malicious files.
Innovation Solution
A system that classifies streaming files at edge devices by processing chunks sequentially using a machine learning model, allowing for early detection and action on malicious files before the entire file is received, utilizing a streamlined deep learning technique and state information to ensure deterministic classification.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If the entire streaming file is received before classification, then classification accuracy is improved, but latency increases and memory constraints are violated
Solution Approach 1:
The streaming file is divided into fixed-size chunks that are processed sequentially. The machine learning model classifies each chunk independently, enabling early detection before the entire file is received. This segmentation allows the system to make classification decisions on partial data, reducing latency while maintaining acceptable accuracy through cumulative chunk analysis.
Solution Approach 2:
The system performs classification actions preliminarily by analyzing chunks as they arrive rather than waiting for complete file reception. This preliminary classification enables immediate detection and response to malicious files, reducing the time loss while the model continues to process subsequent chunks for confirmation or refinement of the classification.
2Reliability
If the entire streaming file is stored in memory for classification, then classification completeness is improved, but memory constraints at edge devices are violated
Solution Approach 1:
The file processing is segmented into fixed-size chunks that are loaded into memory sequentially rather than loading the entire file at once. This approach ensures that memory usage remains bounded and manageable on edge devices with limited resources, while the cumulative analysis of all chunks maintains classification completeness.
Solution Approach 2:
The system performs classification with partial data (individual chunks) rather than requiring complete file data in memory. This partial action approach allows the model to make classification decisions based on available chunk data, ensuring reliability through progressive analysis while adhering to memory constraints by never loading more than one chunk at a time.
3Speed
If streaming files are processed in real-time, then response speed is improved, but processing accuracy may deteriorate due to incomplete data
Solution Approach 1:
The streaming file is segmented into fixed-size chunks processed in sequence. Each chunk is classified independently by the machine learning model, enabling real-time response as chunks arrive. The segmentation strategy balances speed and accuracy by allowing early detection with partial data while maintaining systematic processing of subsequent chunks for refined classification.
Solution Approach 2:
The classification process continues continuously as chunks arrive, with the model performing repeated classification actions on each chunk. This continuous useful action ensures that detection occurs as early as possible (improving response speed) while the cumulative effect of processing multiple chunks maintains detection accuracy through consistent pattern recognition across the streaming data.
Data Source
AI summary
The present application discloses a method, system, and computer system for classifying stream data at an edge device. The method includes obtaining a stream of a file at the edge device, processing a set of chunks associated with the stream of the file using a machine learning model, and classifying, at the edge device, the file before processing an entirety of the file.