Binary Input Structure Extraction Using Recursive State Machines
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing methods for analyzing program inputs, particularly in unknown or undocumented software, struggle to identify complex and nested structures, and require specific parsing strategies or ASCII-encoded inputs, limiting their applicability in analyzing malware and other programs with obscure or complex input structures.
Innovation Solution
A method using dynamic taint analysis infers a recursive state machine to describe the space of acceptable input for arbitrary binary programs, without requiring a distinct parser, and represents complex structures as recursive state machines, capable of handling loops and recursion, and combining state machines from different input data.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If dynamic taint analysis is used to monitor data flow and construct parse trees, then input structure can be identified, but complex and nested structures cannot be properly abstracted and represented
Solution Approach 1:
The patent applies nesting by representing complex and nested input structures as nested state machines, where each state machine can contain other state machines within its transitions or states. This allows the parse tree to properly abstract and represent nested structures at multiple levels, resolving the limitation of flat parse tree representations.
Solution Approach 2:
The patent introduces dynamic state machines that can adapt to different input structures through runtime analysis. The state machines are not static but can dynamically adjust their transitions and states based on the actual data flow patterns observed during execution, enabling them to handle complex and varying nested structures effectively.
2Measurement precision
If known methods track direct data flow in a white-box manner, then grammar can be reconstructed, but the method is limited to scannerless recursive descent parsing with optional lookahead
Solution Approach 1:
The patent creates a universal state machine representation that can model various parsing strategies and input structures without being limited to a specific parsing approach. The state machines can represent different parsing behaviors, loops, recursion, and control flow patterns, making the system adaptable to multiple parsing strategies and input formats simultaneously.
Solution Approach 2:
The patent changes the parameters of the state machine representation to accommodate different parsing strategies. By allowing state machines to have variable transitions, states, and control flow characteristics, the system can adapt to different parsing requirements without being constrained by a fixed parsing paradigm.
3Measurement precision
If function calls are annotated on the function call trace, then grammar rules can be created, but the method relies on how function calls flow in a recursive descent parser
Solution Approach 1:
The patent extracts the grammar representation from the specific recursive descent parsing implementation and creates a standalone state machine model. This separates the grammar extraction process from the parsing implementation details, allowing grammar rules to be derived directly from data flow patterns without depending on the specific parsing strategy used in the program.
Solution Approach 2:
The patent introduces state machines as an intermediary representation between the program's data flow and the final grammar rules. This intermediary layer abstracts away the parsing implementation details and provides a universal representation that can be used to derive grammar rules independent of the specific parsing approach.
4Ease of manufacture
If input is expected to be ASCII characters and parser is a distinct phase, then analysis can be simplified, but applicability to malware and complex input structures is limited
Solution Approach 1:
The patent changes the parameter of input encoding from ASCII-specific to arbitrary binary data. The state machines can process and represent any byte sequence, making the system applicable to malware analysis and programs with complex or non-ASCII input structures, while maintaining a unified analysis approach.
Solution Approach 2:
The patent creates a universal analysis framework that can handle multiple input formats and program types through a single state machine-based approach. The same state machine machinery can analyze ASCII inputs, binary data, network protocols, and malware payloads, eliminating the need for separate analysis pipelines for different input types.
Data Source
AI summary
Herein disclosed is a method for automatically automatically infer a recursive state machine (RSM) describing the space of acceptable input of an arbitrary binary program. This method automatically identifies atomic fields of fixed and variable lengths and syntactic elements, such as separators and terminators, and generalizes them into regular expression tokens. It constructs an RSM of tokens to represent structures such as arrays and records. Further, it constructs nested states in RSM to represent complex, nested structures. The RSM may serve as an independent parser for the program's acceptable input.