Streaming File Classification Using Chunk-Based Edge ML Detection

Resolve Bottlenecks,
Find Innovative SolutionsGenerate Solutions

Solution Overview

Problem

Existing systems for classifying streaming files, particularly at edge devices, are inefficient due to the need to wait for the entire file to be received before classification, leading to latency and memory constraints, making them unsuitable for real-time detection of malicious files.

Innovation Solution

A system that classifies streaming files at edge devices by processing chunks sequentially using a machine learning model, allowing classification before the entire file is received, and implementing chunk alignment to ensure consistent byte analysis.

Engineering Contradictions & Design Principles

VSEngineering Contradiction Analysis

1Measurement precision

If the system waits for the entire file to be received before classification, then the classification accuracy is improved, but the latency increases and real-time detection becomes impossible

Engineering Contradiction:

Improveclassification accuracyVSAvoidlatency

Core Design Contradiction:

Measurement precisionVSLoss of time

Solution Approach 1:

The patent divides the streaming file into discrete chunks that are processed sequentially. The machine learning model receives and analyzes chunks one at a time as they arrive in the stream, enabling classification decisions to be made without waiting for the complete file. This segmentation allows the system to maintain accuracy by analyzing multiple chunks while reducing latency by processing data as it streams in rather than requiring full file reception.

Inventive Principle:

Principle #1Segmentation

2Reliability

If the system processes the entire file before classification, then the detection reliability is improved, but the memory requirements increase

Engineering Contradiction:

Improvedetection reliabilityVSAvoidmemory requirements

Core Design Contradiction:

ReliabilityVSQuantity of substance

Solution Approach 1:

By segmenting the file into manageable chunks, the system maintains detection reliability through sequential analysis of multiple chunks while significantly reducing memory requirements. Instead of loading the entire file into memory, the system processes and stores only the necessary chunk data, allowing reliable detection of malicious patterns without the memory burden of holding complete files.

Inventive Principle:

Principle #1Segmentation

3Measurement precision

If the system uses traditional file classification methods, then the analysis completeness is improved, but the speed of detection decreases

Engineering Contradiction:

Improveanalysis completenessVSAvoiddetection speed

Core Design Contradiction:

Measurement precisionVSSpeed

Solution Approach 1:

The patent implements continuous processing of streaming file chunks as they arrive, eliminating the need to wait for complete file reception. The machine learning model continuously analyzes incoming chunks in real-time, maintaining analysis completeness through ongoing evaluation while significantly improving detection speed. This continuous action allows the system to detect malicious files as they stream in rather than requiring complete file transfer.

Inventive Principle:

Principle #20Continuity of useful action

Data Source

PatentUS20250365296A1Machine learning architecture for detecting malicious files using stream of data

Publication Date: 2025.11.27 PALO ALTO NETWORKS INC

AI summary

The present application discloses a method, system, and computer system for classifying stream data at an edge device. The method includes obtaining a stream of a file at the edge device, processing a set of chunks associated with the stream of the file using a machine learning model, and classifying, at the edge device, the file before processing an entirety of the file.