Distributed Data Representation Generation for Storage Efficiency
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing methods for storing large data sets, such as 'big data,' face challenges in balancing data retrieval speed and storage efficiency, with normalization reducing data size but requiring costly denormalization for retrieval, and denormalization increasing storage needs while speeding retrieval but at a high cost.
Innovation Solution
A system that distributes data blocks across multiple node devices, using metadata and block maps to efficiently generate and retrieve two-dimensional representations of data, allowing for parallel processing and reduced storage requirements.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Quantity of substance
If data is stored in normalized form with compression, then storage efficiency is improved, but data retrieval speed deteriorates due to required denormalization and decompression operations
Solution Approach 1:
The system pre-computes and stores intermediate representation data (partial denormalized forms) alongside the normalized compressed data. When a retrieval request arrives, the system can directly use these pre-prepared representations without performing full denormalization and decompression at query time, thus resolving the contradiction between storage efficiency and retrieval speed.
Solution Approach 2:
The data retrieval process is segmented into multiple stages: the system divides the large normalized data blob into smaller manageable chunks, pre-processes them into intermediate representations, and stores these segments separately. This segmentation allows selective retrieval of only necessary portions without processing the entire dataset, improving retrieval speed while maintaining storage efficiency.
2Speed
If data is stored in denormalized form, then data retrieval speed is improved, but storage requirements increase requiring prohibitively large node device grids
Solution Approach 1:
Instead of fully denormalizing the entire dataset, the system applies denormalization locally and selectively to specific portions of data that are frequently accessed or currently needed. The normalized compressed form is maintained for storage efficiency, while local denormalized representations are created on-demand or pre-computed for specific query patterns, thus improving retrieval speed without proportionally increasing storage requirements.
Solution Approach 2:
The system performs partial denormalization only to the extent necessary for efficient retrieval of specific data subsets, rather than fully denormalizing the entire dataset. This partial action approach creates intermediate representations that speed up retrieval for common query patterns while avoiding the excessive storage overhead of complete denormalization.
3Speed
If heavy denormalization is applied to achieve desired retrieval speeds, then data retrieval speed is improved, but the overall data size increases requiring complex and costly grid infrastructure
Solution Approach 1:
The system pre-computes intermediate denormalized representations and stores them alongside the normalized data. This preliminary action eliminates the need for complex real-time denormalization operations during retrieval, simplifying the query processing logic and reducing the computational complexity required in the grid infrastructure while maintaining fast retrieval speeds.
Solution Approach 2:
The system introduces intermediate representation data as a mediator between the normalized compressed storage and the query processing layer. These intermediaries contain pre-processed data in a format that is easier to query, reducing the complexity of the grid operations required for retrieval while maintaining fast access speeds.
Data Source
AI summary
An apparatus includes a processor and a storage storing instructions causing the processor to receive representation metadata indicating features of representation data to be generated from a plurality of representation portions, receive a command to generate at least one row of the representation data, determine a subset of data blocks of a data blob required to generate the at least one row, and a subset of node devices that store the subset of data blocks, for each node device of the subset of node devices, derive a node block map identifying at least one data item of a data block for generating a representation portion, transmit the node block maps to the subset of node devices; and transmit a command to the subset of node devices to each generate at least one row of one of the plurality of representation portions.