File Sharding With Simultaneous Vector Database Storage
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
The existing methods for storing vectors in vector databases rely on manual processing, leading to low storage and update efficiency of files.
Innovation Solution
Perform a shard process on the file only once and store the file and its corresponding vectors simultaneously, using pre-configured storage configuration information to determine storage size, model, and path, thereby improving storage efficiency and update efficiency.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Productivity
If manual processing is used to upload files and store vectors separately, then the system maintains simple architecture, but the storage efficiency and update efficiency are low
Solution Approach 1:
The patent combines the file storage process and vector storage process into a single automated workflow. When a file is uploaded, the system automatically performs shard processing and stores both the file shards and their corresponding vectors in one operation, eliminating the need for separate manual processing steps and significantly improving storage efficiency
Solution Approach 2:
The system pre-configures storage parameters including shard size, vectorization models, and storage paths for different folders. This preliminary configuration enables the automated processing pipeline to operate efficiently without requiring manual intervention during file upload, resolving the contradiction between automation and system complexity
2Productivity
If separate shard processes are performed for file storage and vector generation, then the processing steps are clear and manageable, but the overall storage efficiency is reduced
Solution Approach 1:
The patent implements a continuous automated processing pipeline where file upload triggers immediate shard processing and vector generation without interruption. The system continuously processes files as they are uploaded, eliminating idle time and manual intervention gaps, thereby improving storage efficiency while reducing total processing time
Solution Approach 2:
Storage configuration parameters such as shard size, vectorization models, and storage paths are pre-configured before file upload. This preliminary setup eliminates the need for manual configuration during processing, enabling the system to execute the entire workflow automatically and efficiently
3Productivity
If vectors are stored manually after file upload, then the system maintains operational control, but the update efficiency of the vector database is low
Solution Approach 1:
The system performs self-service by automatically generating vectors and storing them in the vector database immediately after file upload. The automated pipeline includes built-in vectorization using pre-configured models and automatic storage with proper indexing, eliminating the need for manual vector database updates and significantly improving update efficiency
Solution Approach 2:
The system implements feedback mechanisms where the storage configuration information guides the automated processing pipeline. The pre-configured parameters for shard size, vectorization models, and storage paths provide continuous feedback to ensure consistent and efficient automated processing, resolving the contradiction between automation level and update efficiency
Data Source
AI summary
Embodiments of the present disclosure provide a file storage method, a device, and a storage medium. The method includes: in response to receiving an upload request for a target file, determining a target folder where the target file is located, and determining storage configuration information corresponding to the target folder from storage configuration information corresponding to multiple folders that are pre-configured, the storage configuration information including storage size configuration information, model configuration information and storage path; performing a shard process on the target file according to the storage size configuration information to obtain multiple shards corresponding to the target file; storing the multiple shards corresponding to the target file into a relationship database, determining vectors corresponding to the multiple shards respectively according to the model configuration information, and storing the vectors corresponding to the multiple shards respectively into a vector database according to the storage path configuration information.