Data Lake Table Column Reordering and LSM Compression Selection

Resolve Bottlenecks,
Find Innovative SolutionsGenerate Solutions

Solution Overview

Problem

Existing data lake table storage systems face inefficiencies in optimizing file storage and retrieval due to suboptimal compression techniques and file layout, leading to wasted resources and computational overhead, particularly in log structured merge (LSM) trees.

Innovation Solution

A storage controller optimizes file storage by reordering columns based on historical access patterns and selecting appropriate compression techniques, considering the level of the LSM tree and access frequency, to minimize read coalescing and resource waste.

Engineering Contradictions & Design Principles

VSEngineering Contradiction Analysis

1Productivity

If columnar data is stored with initial ordering and compressed using a single compression technique, then storage operation is simple, but query performance and storage efficiency are suboptimal

Engineering Contradiction:

Improvequery performanceVSAvoidstorage optimization complexity

Core Design Contradiction:

ProductivityVSDevice complexity

Solution Approach 1:

The system performs preliminary actions by reordering columns based on historical access patterns before storage, and by pre-computing multiple compression versions. This allows the storage system to optimize for future queries without requiring complex real-time analysis during query execution.

Inventive Principle:

Principle #10Preliminary action

Solution Approach 2:

The system changes multiple parameters including column ordering and compression techniques to optimize storage. By exploring different compression techniques and selecting based on metrics like storage cost and processing cost, the system achieves optimal balance between storage efficiency and query performance.

Inventive Principle:

Principle #35Parameter changes

2Productivity

If multiple compression techniques are applied to columnar data, then storage efficiency and query performance are optimized, but computational overhead and processing cost increase

Engineering Contradiction:

Improvestorage efficiencyVSAvoidcomputational overhead

Core Design Contradiction:

ProductivityVSUse of energy by moving object

Solution Approach 1:

The system applies partial action by selecting only the necessary compression techniques to explore based on the LSM tree level and data characteristics. Not all compression techniques are applied uniformly - the system strategically selects a subset that provides sufficient optimization while controlling computational overhead.

Inventive Principle:

Principle #16Partial or excessive action

Solution Approach 2:

The system changes the compression technique parameter dynamically based on multiple factors including LSM tree level, historical access patterns, and computed metrics like storage cost and processing cost. This adaptive parameter selection optimizes the balance between compression efficiency and computational expense.

Inventive Principle:

Principle #35Parameter changes

3Productivity

If columns are reordered based on historical access patterns, then read operations are minimized, but data reorganization complexity increases

Engineering Contradiction:

Improvedata retrieval speedVSAvoidcolumn reordering complexity

Core Design Contradiction:

ProductivityVSDevice complexity

Solution Approach 1:

The system performs column reordering as a preliminary action during data ingestion or compaction operations. By analyzing historical access patterns in advance and reordering columns accordingly, the system optimizes future read operations without requiring complex real-time reorganization during query execution.

Inventive Principle:

Principle #10Preliminary action

4Productivity

If data is stored at multiple locations with different compression techniques in LSM tree, then query performance is optimized, but storage cost and management complexity increase

Engineering Contradiction:

Improvequery performanceVSAvoidstorage cost

Core Design Contradiction:

ProductivityVSQuantity of substance

Solution Approach 1:

The system changes storage parameters including compression technique selection and data location based on multiple factors such as LSM tree level, historical access patterns, and computed metrics. This adaptive parameter adjustment allows optimal storage configuration while controlling storage cost through intelligent decision-making.

Inventive Principle:

Principle #35Parameter changes

Data Source

PatentUS12481630B1Optimizing file storage in data lake tables

Publication Date: 2025.11.25 GOOGLE LLC

AI summary

A method for optimizing file storage includes receiving columnar data to store at a columnar data store with columns ordered with an initial ordering. The method includes determining, based on historical access patterns for the columnar data store, an updated ordering for the columns. The method includes storing the columnar data at a first location of the columnar data store using the updated ordering. The method includes determining that the stored columnar data is to be compacted and compressing at least a portion of the columnar data using each of a plurality of compression techniques. The method includes, based on compressing the at least a portion of the columnar data, selecting one of the plurality of compression techniques. The method includes storing the columnar data at a second location of the columnar data store using the selected one of the plurality of compression techniques.