Data Lake Loader Batch Processing Schema Adaptation

Resolve Bottlenecks,
Find Innovative SolutionsGenerate Solutions

Solution Overview

Problem

Existing data loaders face challenges in efficiently processing and transforming event data from streaming platforms to object storage, leading to computation overhead and scalability issues due to varying event data sizes and schema changes.

Innovation Solution

A data loader system that includes a memory for caching event data, a local file system for storing event data, and a data processor configured to receive batches of event data, cache and transform them, and combine them into an output file for loading into object storage, while dynamically determining the schema and managing metadata.

Engineering Contradictions & Design Principles

VSEngineering Contradiction Analysis

1Ease of operation

If the data loader processes and transforms event data before loading to object store, then the computation burden on users is reduced, but the computation overhead and time consumption by the data loader increases

Engineering Contradiction:

Improvecomputation burden on usersVSAvoidcomputation overhead by data loader

Core Design Contradiction:

Ease of operationVSUse of energy by moving object

Solution Approach 1:

The data loader performs schema determination and data transformation in advance before loading event data to the object store. This preliminary action includes determining the schema of event data batches, transforming the data according to the schema, and only then loading to object storage, thereby shifting computation burden from users to the data loader while maintaining efficiency through batch processing

Inventive Principle:

Principle #10Preliminary action

2Loss of time

If the data loader outputs large files to object storage, then data querying efficiency is improved, but the processing time and memory requirements increase

Engineering Contradiction:

Improvedata querying timeVSAvoidfile processing time

Core Design Contradiction:

Loss of timeVSDuration of action of moving object

Solution Approach 1:

The system determines the schema and combines multiple batches of event data into large consolidated files before loading to object storage. This preliminary consolidation action creates large files that enable efficient querying by allowing data queries to skip query-irrelevant data, while the schema determination ensures proper data structure for efficient processing

Inventive Principle:

Principle #10Preliminary action

Solution Approach 2:

The data loader processes event data in batches and combines them into large consolidated files. This segmentation approach allows the system to manage processing in manageable batches while achieving the benefit of large file sizes for efficient querying, balancing processing time and query efficiency

Inventive Principle:

Principle #1Segmentation

3Adaptability or versatility

If the data loader handles varying sizes of event data, then the system becomes more adaptable, but the processing capacity may be bottlenecked

Engineering Contradiction:

Improveadaptability to schema changesVSAvoidprocessing capacity

Core Design Contradiction:

Adaptability or versatilityVSProductivity

Solution Approach 1:

The data loader dynamically determines the schema of event data batches rather than relying on fixed predefined schemas. This dynamic approach allows the system to adapt to varying data sizes and schema changes over time, maintaining productivity by adjusting to the actual characteristics of incoming event data batches

Inventive Principle:

Principle #15Dynamics

4Loss of time

If the data loader loads event data as large files, then querying efficiency is improved, but the ability to handle real-time streaming data is reduced

Engineering Contradiction:

Improvequery execution timeVSAvoiddata loading speed

Core Design Contradiction:

Loss of timeVSSpeed

Solution Approach 1:

The data loader performs preliminary combination of multiple batches of event data into large consolidated files before loading to object storage. This preliminary action creates large files for efficient querying while maintaining the ability to handle streaming data through continuous batch processing and consolidation

Inventive Principle:

Principle #10Preliminary action

Data Source

PatentUS20250068642A1Data lake loader

Publication Date: 2025.02.27 SNOWPLOW ANALYTICS LTD

AI summary

A data loader for loading event data from a streaming platform to an object storage, the data loader comprising: a memory for caching event data; a local file system for storing event data; and a data processor, wherein the data processor is configured to: receive a plurality of batches of event data from the streaming platform; for each batch of event data from the plurality of batches of event data: cache the batch of event data to the memory; write the batch of event data from the memory to the local file system; and combine the plurality of batches of event data stored in the local file system into an output file; and load the output file from the local file system to the object storage.