Transformed Dataset Reuse via Fingerprint Selection in AI Pipelines
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing storage systems face inefficiencies in data management and reliability, particularly in flash storage systems, due to redundant write operations and lack of direct control by the operating system over storage processes, leading to reduced reliability and increased latency.
Innovation Solution
Implementing a direct-mapped flash storage system where the operating system directly addresses data blocks without translation by storage controllers, offloading device management responsibilities, and utilizing non-volatile RAM as a buffer for quick data access and power loss protection, along with dual storage array controllers for failover and distributed management.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Adaptability or versatility
If storage controllers translate data blocks between logical and physical addresses, then data management flexibility is improved, but write operations become redundant and reliability decreases
Solution Approach 1:
The patent extracts the address translation function from the storage controller to the operating system. The OS maintains a direct mapping between logical block addresses and physical flash memory locations, eliminating the storage controller's translation layer that caused redundant writes and reliability issues.
Solution Approach 2:
The patent segments the storage management responsibilities between the operating system (address translation and data management) and the storage controller (only handling raw I/O operations). This segmentation allows the OS to optimize write operations and prevent redundant transfers to flash memory.
2Ease of operation
If storage controllers manage device operations, then device control is centralized, but operating system control and reliability are reduced
Solution Approach 1:
The patent extracts device management control from the storage controller and transfers it to the operating system. The OS directly manages flash memory operations including wear leveling, bad block management, and data placement, while the storage controller becomes a simple I/O interface.
3Speed
If data is directly written to flash memory without buffering, then write speed is improved, but data loss occurs during power loss
Solution Approach 1:
The patent implements a buffering mechanism where data is first written to a buffer in the storage controller or host memory before being committed to flash memory. This preliminary buffering allows for power loss protection and ensures data integrity while maintaining write performance.
4Reliability
If multiple write operations are performed for the same data, then data redundancy is improved, but write latency increases
Solution Approach 1:
The patent performs write operations in advance and consolidates them before transferring to flash memory. The operating system batches multiple write requests and schedules them efficiently, reducing the number of actual flash write operations and minimizing latency while maintaining data redundancy through proper placement strategies.
Data Source
AI summary
Generating a transformed dataset for use by a machine learning model in an artificial intelligence infrastructure that includes one or more storage systems and one or more graphical processing unit (‘GPU’) servers, including: storing, within one or more storage systems, a transformed dataset generated by applying one or more transformations to a dataset that are identified based on one or more expected input formats of data received as input data by one or more machine learning models to be executed on one or more servers; and transmitting, from the one or more storage systems to the one or more servers without reapplying the one or more transformations on the dataset, the transformed dataset including data in the one or more expected formats of data to be received as input data by the one or more machine learning models.