Data Object Segmentation for Distributed Storage Efficiency

Resolve Bottlenecks,
Find Innovative SolutionsGenerate Solutions

Solution Overview

Problem

Existing data processing and storage methods are inefficient when handling large data objects, as they often divide data into equal portions, ignoring structural features and introducing dependencies, leading to increased network traffic and inefficiencies.

Innovation Solution

A technique that splits data objects into multiple portions at boundaries within the object, transforms them into individually processable segments, and distributes these segments across multiple computing nodes for storage and processing, allowing for true parallel processing and reduced network communication.

Engineering Contradictions & Design Principles

VSEngineering Contradiction Analysis

1Quantity of substance

If data objects are divided into equal portions for distributed storage, then storage capacity is improved, but processing efficiency deteriorates due to structural ignorance and dependency introduction

Engineering Contradiction:

Improvestorage capacityVSAvoidprocessing efficiency

Core Design Contradiction:

Quantity of substanceVSProductivity

Solution Approach 1:

The patent divides data objects into segments based on structural boundaries (rows, columns, or both) rather than equal portions. This segmentation preserves the integrity of data structures while enabling independent processing of each segment across distributed storage nodes, thus maintaining both storage capacity and processing efficiency.

Inventive Principle:

Principle #1Segmentation

Solution Approach 2:

The patent applies different segmentation strategies (row-based, column-based, or both) to different portions of the data object based on local structural characteristics. This allows each segment to be optimized for its specific data type and access patterns, improving processing efficiency without compromising storage capacity.

Inventive Principle:

Principle #3Local quality

2Ease of operation

If data portions are cut at arbitrary boundaries to form equal-sized segments, then storage distribution is improved, but data integrity deteriorates by splitting rows

Engineering Contradiction:

Improvestorage distributionVSAvoiddata integrity

Core Design Contradiction:

Ease of operationVSStability of the object's composition

Solution Approach 1:

The patent segments data objects along structural boundaries (row boundaries, column boundaries) rather than arbitrary byte positions. This ensures that each segment contains complete rows or columns, preserving data integrity while still enabling distributed storage across multiple nodes.

Inventive Principle:

Principle #1Segmentation

Solution Approach 2:

The patent accepts that segments may be of unequal sizes to maintain data integrity. By allowing asymmetric segment sizes that align with structural boundaries, the system prioritizes data integrity over uniform distribution, while still achieving effective distributed storage.

Inventive Principle:

Principle #4Asymmetry

3Stability of the object's composition

If complete rows are stored together to maintain data integrity, then data integrity is improved, but network traffic increases due to transferring entire rows

Engineering Contradiction:

Improvedata integrityVSAvoidnetwork traffic

Core Design Contradiction:

Stability of the object's compositionVSLoss of energy

Solution Approach 1:

The patent further segments rows into column-based segments, allowing the system to transfer only the specific columns needed for a query rather than entire rows. This reduces network traffic while maintaining data integrity through structured segmentation and selective retrieval.

Inventive Principle:

Principle #1Segmentation

Solution Approach 2:

The patent extracts and stores data in columnar segments, enabling the system to retrieve only the specific columns required for processing. This extraction approach minimizes network traffic by transferring only necessary data portions while preserving complete row information when needed through reconstruction from column segments.

Inventive Principle:

Principle #2Taking out (Extraction)

4Reliability

If all data portions are gathered to a single node for processing, then processing completeness is improved, but network traffic and processing time increase

Engineering Contradiction:

Improveprocessing completenessVSAvoidprocessing time

Core Design Contradiction:

ReliabilityVSLoss of time

Solution Approach 1:

The patent segments data into independently processable units (row segments, column segments, or both) that can be processed in parallel across multiple nodes. Each segment contains sufficient metadata and structure information to be processed independently, ensuring processing completeness while enabling concurrent execution and reducing overall processing time.

Inventive Principle:

Principle #1Segmentation

Solution Approach 2:

The patent creates segments with universal interfaces and metadata structures that allow any segment to be processed by any computing node with the appropriate software. This universality enables flexible parallel processing distribution while maintaining processing completeness, as any node can handle any segment type.

Inventive Principle:

Principle #6Universality (Multi-functionality)

Data Source

PatentUS20230342347A1Partitioning, processing, and protecting data

Publication Date: 2023.10.26 AIRMETTLE INC

AI summary

A technique for managing data objects in a storage cluster includes splitting a data object into multiple portions at boundaries within the data object. The technique further includes transforming the portions of the data object into segments that provide individually processable units, and distributing the segments among multiple computing nodes of the storage cluster for storage therein.