Real-Time Instrumentation Data Analytics via Hybrid Memory

Resolve Bottlenecks,
Find Innovative SolutionsGenerate Solutions

Solution Overview

Problem

Current systems lack the infrastructure and APIs to provide real-time access to machine-generated instrumentation data, leading to inefficiencies in data analysis and delayed detection of errors or performance issues, as data is typically written to disk and analyzed after a significant time lag.

Innovation Solution

Implementing a hybrid approach using in-memory buffers (DRAM) and server-attached flash memory to store and access machine-generated instrumentation data in real-time, allowing immediate streaming into analytics frameworks like Apache Spark, and aggregating data across multiple machines without consolidating raw data, enabling real-time analytics and reducing the need for I/O operations.

Engineering Contradictions & Design Principles

VSEngineering Contradiction Analysis

1Quantity of substance

If data is written to disk for storage and analysis, then data persistence and capacity are improved, but access time and analysis speed deteriorate due to significant time lag

Engineering Contradiction:

Improvedata capacityVSAvoidaccess time

Core Design Contradiction:

Quantity of substanceVSLoss of time

Solution Approach 1:

The patent segments data storage into two distinct layers: volatile memory (DRAM) for hot, frequently accessed data requiring real-time analysis, and non-volatile memory (disk) for cold, historical data requiring long-term persistence. This segmentation allows the system to optimize access time for recent instrumentation data while maintaining adequate data capacity through disk storage of older data.

Inventive Principle:

Principle #1Segmentation

Solution Approach 2:

The patent changes the storage medium parameter from exclusively non-volatile (disk) to a hybrid of volatile and non-volatile memory. By altering the physical state and volatility parameters of the storage system, the patent enables real-time access to instrumentation data in DRAM while maintaining data capacity through periodic migration to disk, thus resolving the time-capacity contradiction.

Inventive Principle:

Principle #35Parameter changes

2Speed

If data is stored in volatile memory for real-time access, then access speed and analysis time are improved, but data persistence and reliability worsen due to volatility

Engineering Contradiction:

Improveaccess speedVSAvoiddata persistence

Core Design Contradiction:

SpeedVSReliability

Solution Approach 1:

The patent implements preliminary action by periodically migrating data from volatile memory to non-volatile memory before the volatile data is lost. This proactive migration ensures that data is persisted to disk at scheduled intervals, maintaining reliability while allowing the system to benefit from fast volatile storage during the intervals between migrations.

Inventive Principle:

Principle #10Preliminary action

Solution Approach 2:

The patent maintains continuous useful action through the periodic migration process, where data continuously flows from volatile to non-volatile storage at scheduled intervals. This continuous migration ensures that no data is permanently lost and that the system maintains both the speed benefits of volatile memory and the reliability of persistent storage.

Inventive Principle:

Principle #20Continuity of useful action

3Adaptability or versatility

If data is consolidated from multiple machines for centralized analysis, then data aggregation capability is improved, but I/O operations and processing overhead increase

Engineering Contradiction:

Improvedata aggregation capabilityVSAvoidprocessing efficiency

Core Design Contradiction:

Adaptability or versatilityVSProductivity

Solution Approach 1:

The patent extracts only the necessary subset of data from volatile memory for migration to non-volatile storage, rather than consolidating all data from multiple machines. This selective extraction approach maintains data aggregation capability by allowing analytics applications to access data from multiple sources while minimizing I/O operations by only moving data that requires persistent storage.

Inventive Principle:

Principle #2Taking out (Extraction)

4Measurement precision

If frequent I/O operations are performed for data migration and access, then data freshness and analytics accuracy are improved, but system performance and energy consumption worsen

Engineering Contradiction:

Improveanalytics accuracyVSAvoidenergy consumption

Core Design Contradiction:

Measurement precisionVSUse of energy by moving object

Solution Approach 1:

The patent implements periodic action by performing data migration from volatile to non-volatile memory at scheduled intervals rather than continuously or on every access. This periodic migration maintains adequate data freshness for analytics accuracy while dramatically reducing I/O operations and energy consumption compared to frequent or continuous migration approaches.

Inventive Principle:

Principle #19Periodic action

Data Source

PatentUS10579506B2Real-time analytics of machine generated instrumentation data

Publication Date: 2020.03.03 INTERNATIONAL BUSINESS MACHINE CORPORATION

AI summary

An aspect includes generating instrumentation data for software that is executing on a computer and writing the instrumentation data to a memory located on the computer. An analytics application is provided with access to the instrumentation data. The writing the instrumentation data and the providing access are performed in real-time with respect to the generating. A request is received from the analytics application for at least a subset of the instrumentation data. Based on receiving the request, at least a subset of the instrumentation data is transmitted to the analytics application. A portion of the instrumentation data is migrated on a periodic basis to a non-volatile memory device that is external to the computer.