Multipart Upload Micro-Transaction Data Structure for Large Event Storage

Resolve Bottlenecks,
Find Innovative SolutionsGenerate Solutions

Solution Overview

Problem

Current streaming data storage systems have limitations on event size, making it difficult to handle large events such as high-definition video streams that exceed the standard 8 MB limit, leading to resource overhead and potential data loss or truncation issues when handling large data sets.

Innovation Solution

Implementing a multipart upload approach using micro-transactions that allow large events to be split into smaller parts, uploaded, and reassembled, with optional order-on-start or order-on-end options to manage data consistency and resource usage, enabling events up to 1024 MB in size without overwhelming system resources.

Engineering Contradictions & Design Principles

VSEngineering Contradiction Analysis

1Productivity

If a streaming data storage system enforces a strict event size limit (e.g., 8 MB) to maintain high ingestion rates, then system performance and throughput are improved, but the system cannot handle large events such as high-definition video streams

Engineering Contradiction:

Improveevent ingestion rateVSAvoidability to handle large events

Core Design Contradiction:

ProductivityVSAdaptability or versatility

Solution Approach 1:

The patent divides large events exceeding the size limit into multiple smaller subparts, each within the acceptable size limit. The system creates a multipart upload event where the large event is segmented into manageable chunks that can be processed individually while maintaining the overall event integrity through associated metadata and reconstruction capabilities.

Inventive Principle:

Principle #1Segmentation

2Quantity of substance

If the system allows events larger than the standard limit, then the ability to store large data sets is improved, but resource overhead increases and system performance deteriorates

Engineering Contradiction:

Improveevent size capacityVSAvoidresource overhead

Core Design Contradiction:

Quantity of substanceVSUse of energy by moving object

Solution Approach 1:

By segmenting large events into smaller subparts, the system processes and stores data in manageable units that fit within resource constraints. Each subpart is handled independently through the existing high-performance ingestion pathway, avoiding the need to allocate excessive resources for single large events while maintaining the capability to store and retrieve complete large events when needed.

Inventive Principle:

Principle #1Segmentation

Solution Approach 2:

The system processes only the necessary portions of large events at any given time through multipart upload mechanisms. Rather than loading entire large events into memory simultaneously, the system processes subparts incrementally, using just enough resources to handle the current batch while maintaining the ability to reconstruct complete events on demand.

Inventive Principle:

Principle #16Partial or excessive action

3Reliability

If the system processes large events as single units, then data integrity is maintained, but the complexity of managing resource allocation and data consistency increases

Engineering Contradiction:

Improvedata integrityVSAvoidresource management complexity

Core Design Contradiction:

ReliabilityVSDevice complexity

Solution Approach 1:

The patent segments large events into subparts while maintaining data integrity through an associated data structure that tracks the composition of the original event. This structure stores metadata linking subparts to their parent event and to each other, enabling the system to verify completeness and reconstruct the original event accurately without treating it as an indivisible unit, thereby reducing resource management complexity.

Inventive Principle:

Principle #1Segmentation

4Adaptability or versatility

If the system implements multipart upload functionality, then the ability to handle large events is improved, but the complexity of the upload process and data reconstruction increases

Engineering Contradiction:

Improvelarge event handling capabilityVSAvoidupload process complexity

Core Design Contradiction:

Adaptability or versatilityVSDevice complexity

Solution Approach 1:

The system implements automatic reconstruction of large events from subparts through metadata-driven processes. The associated data structure contains sufficient information to guide the reassembly process, allowing the system to automatically reconstruct complete events from their segmented components without requiring complex manual intervention or overly sophisticated reconstruction algorithms, thereby managing complexity while maintaining versatility.

Inventive Principle:

Principle #25Self-service

Data Source

PatentUS11349906B1Multipart upload for large data events in streaming data storage

Publication Date: 2022.05.31 EMC IP HLDG CO LLC

AI summary

A streaming data storage system facilitates appending of large events (e.g., up to one gigabyte) to a data segment of a streaming data storage system in a multipart upload operation. A micro-transaction data structure is created for a multipart upload of a large event, to which subparts of the multipart upload are appended during write operations. Order of the subparts is preserved, including when not appended in order, to provide for reading of the subparts in order. An event reference to the micro-transaction data structure is maintained in a data segment corresponding to the large event, and when the event reference encountered during reading, results in reading from the micro-transaction data structure when the multipart upload is complete. The reading from the micro-transaction data structure maintains the order of the large event's subparts, such that raw data is returned to an upstream reader application as the large event.