Binary Code Similarity Detection via N-Gram Fingerprinting

Resolve Bottlenecks,
Find Innovative SolutionsGenerate Solutions

Solution Overview

Problem

Conventional methods for detecting code similarity and malware variants are inefficient in processing speed and library capacity, making it difficult to rapidly compare input samples against a large library of references for bulk processing.

Innovation Solution

An apparatus and method that utilize lossy compression to generate 'fingerprints' or 'DNA' strings from binary code, allowing for efficient clustering and recursive analysis of malware, with n-gram based scoring and integer hashing to reduce processing time and space requirements, while maintaining the ability to identify near-duplicate code variants and malware.

Engineering Contradictions & Design Principles

VSEngineering Contradiction Analysis

1Productivity

If conventional code similarity detection methods are used, then code similarity can be identified, but processing speed and library capacity are insufficient for rapid bulk processing

Engineering Contradiction:

Improveprocessing speedVSAvoidcode similarity detection accuracy

Core Design Contradiction:

ProductivityVSMeasurement precision

Solution Approach 1:

The code is segmented into n-grams (sequences of tokens) to create a fingerprint representation. This segmentation allows the system to process code in manageable units, enabling rapid comparison while preserving the ability to detect similarity patterns. The n-gram approach divides the code into overlapping sequences of n tokens, which are then hashed and compared efficiently.

Inventive Principle:

Principle #1Segmentation

Solution Approach 2:

The system changes the parameter of code representation from full code text to hashed n-gram fingerprints. By transforming the code into a different parameter space (hash values of n-grams), the system enables fast comparison operations while maintaining the ability to detect similarity. The hashing function compresses the n-gram sequences into fixed-size integers that can be rapidly compared.

Inventive Principle:

Principle #35Parameter changes

2Adaptability or versatility

If a large library of reference samples is maintained, then more code variants can be detected, but storage space and processing overhead increase

Engineering Contradiction:

Improvemalware variant detection capabilityVSAvoidreference library size

Core Design Contradiction:

Adaptability or versatilityVSVolume of stationary object

Solution Approach 1:

Instead of storing complete reference code samples, the system stores only their fingerprint representations (hashed n-grams). This copying approach creates a simplified representation that occupies minimal space while retaining the essential characteristics needed for similarity detection. The fingerprint acts as a compact copy that enables rapid comparison without requiring the original full-size code samples.

Inventive Principle:

Principle #26Copying

Solution Approach 2:

The system performs preliminary processing of reference samples by converting them to fingerprints before storage. This preliminary action of hashing and storing only the compressed representation allows the library to grow large without proportionally increasing storage requirements or processing overhead. The fingerprints are pre-computed and stored, enabling rapid query-time comparisons.

Inventive Principle:

Principle #10Preliminary action

3Reliability

If duplicate reference samples are stored in the library, then comprehensive coverage is achieved, but processing efficiency decreases

Engineering Contradiction:

Improvecode similarity detection completenessVSAvoidbulk processing speed

Core Design Contradiction:

ReliabilityVSProductivity

Solution Approach 1:

The system uses feedback from fingerprint comparison to identify and eliminate duplicate reference samples. When a new reference sample is added, its fingerprint is compared against existing fingerprints in the library. If a match is found (indicating a duplicate), the new sample is not added to the library. This feedback mechanism ensures the library contains only unique representations, maintaining processing efficiency while preserving detection completeness.

Inventive Principle:

Principle #23Feedback

Data Source

PatentUS10152518B2Apparatus and method for efficient identification of code similarity

Publication Date: 2018.12.11 JOHNS HOPKINS UNIVERSITY

AI summary

A method for identifying similarity between query samples and stored samples in an efficiently maintained reference library may include receiving a binary query sample and processing the binary query sample via operations including producing a query sample fingerprint from the binary query sample, scoring the query sample fingerprint with each previously stored fingerprint in the reference library to produce a matching score, and for each previously stored fingerprint for which the matching score meets or exceeds a predetermined threshold, reporting a corresponding reference sample unique identifier associated with the previously stored fingerprint and the matching score. Each previously stored fingerprint in the reference library has been determined, prior to storage, as not being duplicative of another fingerprint in the reference library.