Data Descriptor Array for Compressed Storage Location
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing data processing techniques face challenges in efficiently locating and accessing specific portions of data within storage systems, particularly in data-intensive operations like neural network processing, where aggressive compression leads to unpredictable compression rates and variable encoding, making it impractical to track precise locations of data weights.
Innovation Solution
The use of task-based and job-based parameters to determine the position of data descriptors within an array, allowing for efficient retrieval and processing of specific data portions by utilizing the predetermined size of each data descriptor to facilitate predictable access and compression, even in compressed data streams.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Quantity of substance
If data compression is applied to reduce storage footprint, then storage efficiency is improved, but data location predictability deteriorates
Solution Approach 1:
The patent segments data into fixed-size units (e.g., 256 bytes) and organizes them in predictable patterns within compressed streams. Each segment maintains a consistent structure that allows deterministic location calculation using task-based parameters, resolving the contradiction between compression and location predictability.
Solution Approach 2:
The patent introduces task-based parameters (such as task index, data descriptor size, and offset) that change predictably with compression. These parameters allow the system to calculate data locations deterministically despite compression, maintaining measurement precision while reducing storage footprint.
2Productivity
If data descriptors are stored in arrays with predetermined sizes, then data access efficiency is improved, but storage requirements increase
Solution Approach 1:
The patent creates a universal data descriptor array structure that serves multiple functions: storing metadata, enabling random access, and supporting compression. The predetermined size of each descriptor ensures consistent memory layout while the array itself becomes a multi-functional component for efficient data management.
Solution Approach 2:
The patent transitions from storing individual data locations to storing data in a structured array dimension with predetermined sizes. This dimensional organization allows efficient access through index calculations while compressing the overall storage footprint by eliminating redundant location information.
3Quantity of substance
If compression rates are made aggressive to reduce storage, then storage efficiency is improved, but data retrieval reliability deteriorates
Solution Approach 1:
The patent performs preliminary organization of data into fixed-size segments with predetermined structures before compression. This preliminary structuring ensures that even with aggressive compression, the data can be reliably retrieved using task-based parameters, as the organizational framework is established in advance.
Solution Approach 2:
The patent introduces data descriptors as intermediary structures between the compressed data and the retrieval process. These descriptors act as mediators that maintain reliable access to compressed data by storing essential location and size information, enabling reliable retrieval even with aggressive compression.
Data Source
AI summary
A processor to: receive a task to be executed, the task comprising a task-based parameter associated with the task, for use in determining a position, within an array of data descriptors, of a particular data descriptor of a particular portion of data to be processed in executing the task. Each of the data descriptors in the array of data descriptors has a predetermined size and is indicative of a location in a storage system of a respective portion of data. The processor derives, based on the task, array location data indicative of a location in the storage system of a predetermined data descriptor, and obtains the particular data descriptor, based on the array location data and the task-based parameter. The processor obtains the particular portion of data based on the particular data descriptor and processes the particular portion of data in executing the task.