Containerized Data Analytics Pipelines for Component Failover

Resolve Bottlenecks,
Find Innovative SolutionsGenerate Solutions

Solution Overview

Problem

Existing data storage systems face challenges in maintaining high availability without replicas, leading to potential data loss and system downtime during failures.

Innovation Solution

Implementing a storage system with dual storage array controllers that share drives for failover, utilizing non-volatile random access memory (NVRAM) for fast write operations, and employing erasure coding and redundancy schemes to ensure data integrity and availability.

Engineering Contradictions & Design Principles

VSEngineering Contradiction Analysis

1Reliability

If traditional storage systems use replicas for high availability, then data availability is improved, but storage cost and system complexity increase

Engineering Contradiction:

Improvedata availabilityVSAvoidsystem complexity

Core Design Contradiction:

ReliabilityVSDevice complexity

Solution Approach 1:

The patent segments data into multiple shards distributed across different storage nodes. Instead of creating full replicas of entire datasets, the system divides data into manageable segments that can be independently stored and recovered, reducing overall system complexity while maintaining availability

Inventive Principle:

Principle #1Segmentation

Solution Approach 2:

The system changes the redundancy parameter from full replication (100% copy) to erasure coding where only portions of data need to be stored. By using erasure coding parameters (k data shards, m parity shards), the system achieves the same reliability with less storage overhead and reduced complexity

Inventive Principle:

Principle #35Parameter changes

2Speed

If data is written directly to storage drives, then storage capacity is utilized, but write speed and data integrity during power failure are reduced

Engineering Contradiction:

Improvewrite speedVSAvoiddata integrity

Core Design Contradiction:

SpeedVSReliability

Solution Approach 1:

The system performs preliminary writing of data to NVRAM before finalizing to persistent storage. This preliminary action in a volatile but fast medium allows for quick writes while maintaining the ability to recover data if power fails, as the NVRAM contents can be flushed to storage drives atomically

Inventive Principle:

Principle #10Preliminary action

Solution Approach 2:

NVRAM serves as an intermediary between the host system and persistent storage drives. It provides a buffer that enables fast writes to be acknowledged immediately while the actual data transfer to slower storage media occurs in the background, decoupling write speed from storage capacity

Inventive Principle:

Principle #24Intermediary (Mediator)

3Reliability

If storage systems implement redundancy schemes, then data integrity is improved, but storage efficiency and capacity utilization decrease

Engineering Contradiction:

Improvedata integrityVSAvoidstorage efficiency

Core Design Contradiction:

ReliabilityVSQuantity of substance

Solution Approach 1:

The system changes the redundancy approach from traditional RAID with fixed parity blocks to erasure coding with flexible parameters. By adjusting the (k, m) parameters, administrators can optimize between data integrity (higher m) and storage efficiency (lower m), achieving better storage efficiency than traditional RAID while maintaining data integrity

Inventive Principle:

Principle #35Parameter changes

Solution Approach 2:

Instead of copying entire data blocks with full redundancy, the system creates distributed fragments through erasure coding where any k out of (k+m) shards can reconstruct the original data. This partial copying approach significantly improves storage efficiency while maintaining data integrity through mathematical reconstruction

Inventive Principle:

Principle #26Copying

4Speed

If the system uses volatile RAM for fast writes, then write performance is improved, but data loss risk during power failure increases

Engineering Contradiction:

Improvewrite performanceVSAvoiddata loss risk

Core Design Contradiction:

SpeedVSReliability

Solution Approach 1:

Data is preliminarily written to NVRAM which maintains data persistently even without power. This preliminary persistent storage allows the system to acknowledge fast writes while ensuring data survives power failures, as NVRAM retains data without requiring continuous power

Inventive Principle:

Principle #10Preliminary action

Solution Approach 2:

The system replaces volatile RAM with NVRAM technology that combines the speed characteristics of volatile memory with the persistence of non-volatile storage. This substitution eliminates data loss risk during power failure while maintaining fast write performance through NVRAM's superior speed compared to traditional storage drives

Inventive Principle:

Principle #28Mechanics substitution (Replace mechanical system)

Data Source

PatentUS12360865B2Creating a containerized data analytics pipeline

Publication Date: 2025.07.15 PURE STORAGE INC

AI summary

Creating a containerized data analytics pipeline, including: creating a data analytics pipeline having a component that is deployed within a container that executes in a container cluster that is external to shared storage accessible by components of the data analytics pipeline; and responsive to detecting that a component of the data analytics pipeline has failed, deploying another instance of the component in a failover container that is configured to support one or more components of the data analytics pipeline.