OS Signal Handler for ROP Attack Detection and Response

Resolve Bottlenecks,
Find Innovative SolutionsGenerate Solutions

Solution Overview

Problem

Existing systems fail to effectively detect and respond to Return-Oriented Programming (ROP) attacks on transformed program binaries, leading to abrupt termination of processes without providing diagnostic information, resulting in potential data loss and service disruptions.

Innovation Solution

Implementing a signaling mechanism at the operating system level, utilizing signal handlers to collect and analyze in-memory artifacts, provide real-time notifications, and enable customizable responses to invalid states, thereby differentiating between malicious attacks and errors, and identifying attack signatures.

Engineering Contradictions & Design Principles

VSEngineering Contradiction Analysis

1Reliability

If binary transformation is applied to mitigate ROP attacks, then security against ROP attacks is improved, but the system becomes vulnerable to detection failures and lacks active response capabilities

Engineering Contradiction:

Improvesecurity against ROP attacksVSAvoidattack detection capability

Core Design Contradiction:

ReliabilityVSDifficulty of detecting and measuring

Solution Approach 1:

The patent implements a feedback mechanism by installing a signal handler that actively monitors program execution states. When an invalid state is detected (indicating a ROP attack), the handler generates signals to notify the user, creating a closed-loop detection and response system that transforms passive binary transformation into an active monitoring and response system.

Inventive Principle:

Principle #23Feedback

Solution Approach 2:

The patent introduces an intermediary layer in the form of a signal handler between the transformed binary execution and the user. This intermediary monitors execution states, detects attacks, and mediates the response by generating notifications to the user, thereby bridging the gap between detection and user awareness.

Inventive Principle:

Principle #24Intermediary (Mediator)

2Object-affected harmful factors

If the system crashes upon detecting memory corruption, then security response is simplified, but data loss and service disruption increase

Engineering Contradiction:

Improvesecurity response effectivenessVSAvoiddata loss and service disruption

Core Design Contradiction:

Object-affected harmful factorsVSLoss of substance

Solution Approach 1:

The patent converts the harmful effect of memory corruption (which normally causes crashes) into a beneficial detection opportunity. By installing a signal handler that captures invalid state conditions, the system transforms what would be a silent crash into an actionable signal that notifies the user, allowing for informed response decisions that prevent data loss and service disruption.

Inventive Principle:

Principle #22Blessing in disguise (Convert harm into benefit)

Solution Approach 2:

The patent applies beforehand cushioning by pre-installing a signal handler that is ready to capture and notify about invalid states before they cause data loss or service disruption. This preparatory measure cushions against the harmful effects of memory corruption by enabling proactive user notification and response.

Inventive Principle:

Principle #11Beforehand cushioning (Prior cushioning)

3Device complexity

If no signal handler is installed, then system complexity is minimized, but diagnostic information and user notification are lost

Engineering Contradiction:

Improvesystem complexityVSAvoiddiagnostic information

Core Design Contradiction:

Device complexityVSLoss of information

Solution Approach 1:

The patent applies segmentation by separating the signal handling functionality from the main binary execution flow. The signal handler is implemented as a distinct, modular component that independently monitors execution states and generates notifications, thereby adding diagnostic capability without significantly complicating the core binary transformation mechanism.

Inventive Principle:

Principle #1Segmentation

Data Source

PatentUS11816211B2Active signaling in response to attacks on a transformed binary

Publication Date: 2023.11.14 RUNSAFE SECURITY INC

AI summary

An apparatus and method for responding to an invalid state occurrence encountered during execution of a third-party application program is included. The apparatus performing the method which includes registering a trap signal handler with a kernel of an operating system. The method also including intercepting calls from the third-party application program to the operating system and processing an exception signal corresponding to the invalid state to generate a response. The response including performing a signal reporting process.