Parallelized Custom Data Processing Without Indexing

Resolve Bottlenecks,
Find Innovative SolutionsGenerate Solutions

Solution Overview

Problem

Existing data processing systems are inefficient in handling unstructured data, requiring costly indexing and scanning that is limited in detecting embedded content within compound files, leading to undetected cybersecurity threats and missed evidence of wrongdoing.

Innovation Solution

A system that processes data in volatile memory without creating an index, using parallel processing and heuristics to analyze content and metadata, allowing real-time detection of specific characteristics and immediate action on unstructured data.

Engineering Contradictions & Design Principles

VSEngineering Contradiction Analysis

1Productivity

If indexing is used to process data, then search capability is improved, but cost and time consumption increase

Engineering Contradiction:

Improvesearch capabilityVSAvoidtime consumption

Core Design Contradiction:

ProductivityVSLoss of time

Solution Approach 1:

The system performs preliminary extraction of content and metadata from compound files into volatile memory before processing. This allows the data to be ready for immediate analysis without requiring time-consuming indexing operations, resolving the contradiction between search capability and time consumption

Inventive Principle:

Principle #10Preliminary action

Solution Approach 2:

The system creates a copy of the compound file contents in volatile memory for processing, rather than working with the original indexed structures. This copying approach enables rapid analysis without the overhead of maintaining indexes, improving productivity while reducing time loss

Inventive Principle:

Principle #26Copying

2Difficulty of detecting and measuring

If scanning is used to examine data, then detection capability is improved, but ability to examine embedded content is limited

Engineering Contradiction:

Improvedetection capabilityVSAvoidability to examine embedded content

Core Design Contradiction:

Difficulty of detecting and measuringVSAdaptability or versatility

Solution Approach 1:

The system segments the compound file into its constituent components (embeds, objects, streams) and processes each segment separately in volatile memory. This segmentation enables thorough examination of embedded content while maintaining detection capability, resolving the contradiction between detection ability and adaptability to embedded structures

Inventive Principle:

Principle #1Segmentation

Solution Approach 2:

The system implements a nested processing structure where compound files are extracted and processed, which may contain embedded files that are themselves processed recursively. This nested approach allows the system to penetrate multiple levels of embedding while maintaining detection capability, improving both detection and adaptability

Inventive Principle:

Principle #7Nested doll (Nesting)

3Ease of operation

If indexing is performed, then data organization is improved, but storage resources and cost increase

Engineering Contradiction:

Improvedata organizationVSAvoidstorage resources

Core Design Contradiction:

Ease of operationVSQuantity of substance

Solution Approach 1:

The system extracts only the necessary content and metadata from compound files into volatile memory for processing, rather than creating comprehensive indexes that require additional storage. This extraction approach provides sufficient data organization for processing while minimizing storage resource consumption

Inventive Principle:

Principle #2Taking out (Extraction)

Solution Approach 2:

The system uses volatile memory as a temporary, disposable storage medium for processing data, eliminating the need for persistent indexing structures that consume long-term storage resources. This approach provides adequate data organization during processing while avoiding the storage cost of maintaining indexes

Inventive Principle:

Principle #27Cheap short-living objects (Disposable)

4Speed

If traditional data processing is used, then processing speed is maintained, but detection of security threats is delayed

Engineering Contradiction:

Improveprocessing speedVSAvoiddetection delay

Core Design Contradiction:

SpeedVSLoss of time

Solution Approach 1:

The system performs preliminary extraction and analysis of data in volatile memory before any security assessment is needed. This preliminary processing prepares data for immediate security evaluation, enabling real-time threat detection without the delays associated with traditional indexing and scanning processes

Inventive Principle:

Principle #10Preliminary action

Solution Approach 2:

The system maintains continuous processing of data in volatile memory, enabling ongoing security monitoring without interruption. This continuous action eliminates the start-stop nature of traditional batch processing, improving both processing speed and detection responsiveness

Inventive Principle:

Principle #20Continuity of useful action

Data Source

PatentUS12413617B2Systems and methods for parallelized custom data-processing and search

Publication Date: 2025.09.09 NUIX

AI summary

This invention provides systems and methods for data processing by means of an ongoing background process on an end-user's computer. As a user receives and generates data, files are analyzed. A container file is opened into the volatile memory and its contents (including data and metadata) are extracted, without requiring an index to be created. The extracted components are analyzed based on predefined characteristics.