Zero ETL Data Analytics Offloading Processing to Storage
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing data analytics systems face inefficiencies in processing large datasets due to the need for entire objects to be read into compute nodes for processing, leading to increased resource requirements and costs.
Innovation Solution
The implementation of zero extract transform load (zETL) systems and methods, which offload ETL functions to storage devices capable of compute, such as computational storage drives, allowing for processing near where the data is stored and reducing data transfer.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Productivity
If entire objects are read into compute nodes for processing, then data processing can be performed, but resource requirements and costs increase
Solution Approach 1:
The patent extracts the ETL processing function from the compute node and places it directly on the storage device. Only the necessary processed results are transferred back to the compute node, rather than reading entire objects into memory for processing. This extraction principle resolves the contradiction by enabling data processing capability while minimizing the quantity of data that needs to be transferred and processed in memory.
Solution Approach 2:
The storage device acts as an intermediary that performs ETL processing locally. Instead of the compute node directly processing raw data after reading it entirely into memory, the storage device serves as a mediator that transforms the data in-place and returns only the processed results. This intermediary approach maintains productivity while reducing resource requirements.
2Productivity
If data is transferred from storage to compute nodes, then processing can occur, but data transfer overhead increases
Solution Approach 1:
The ETL processing is performed preliminarily on the storage device before data is transferred to the compute node. By transforming, filtering, and preparing the data in-place on storage, the system ensures that only the necessary processed results need to be transferred, thereby maintaining processing throughput while minimizing transfer time.
Solution Approach 2:
The patent extracts only the processed results from the storage device rather than transferring entire data objects. This selective extraction of transformed data reduces the volume of data transfer while preserving processing capability, effectively resolving the time-loss contradiction.
3Adaptability or versatility
If traditional ETL processes are used, then data can be transformed, but system complexity increases
Solution Approach 1:
The patent merges the ETL processing functionality directly into the storage device, combining data storage and data transformation capabilities in a single location. This integration reduces system complexity by eliminating separate ETL processing components and data movement infrastructure, while maintaining full data transformation capability through the unified storage-compute architecture.
Data Source
AI summary
Provided are systems, methods, and apparatuses for data analytics based on zero extract transform load (ETL). In one or more examples, the systems, devices, and methods include generating a data request for at least a subset of data stored in a database; converting the data request to an object storage request; determining, based on parsing the object storage request, that the object storage request includes an extract transform load (ETL) request; creating an ETL command based on the object storage request; routing the ETL command to a storage device based on the storage device comprising at least one ETL function requested in the data request; and determining information from result data that is received from the storage device in response to routing the ETL command to the storage device.