Semantic File Storage Using Metadata Embeddings and LLM Search
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing data search tools are limited in their search criteria, making it difficult for users to find files or data objects when they only remember certain attributes or content, especially in large data storage systems like cloud-based applications.
Innovation Solution
A data storage system that uses a tagging module to generate metadata based on content and non-content information, an indexing module to create efficient search indices, and a Large Language Model (LLM) to translate free-text requests into structured commands, allowing for a wider range of search criteria and improved retrieval of related files or data objects.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Adaptability or versatility
If traditional structured search tools are used, then search operations are simple to implement, but search capability is limited and cannot handle vague queries
Solution Approach 1:
The patent introduces an LLM-based intermediary layer that translates natural language queries into structured search commands. This mediator handles the complexity of understanding vague user intent while presenting a simple natural language interface to users, resolving the contradiction between versatile search capability and system complexity.
Solution Approach 2:
The system employs self-service mechanisms where the LLM automatically analyzes user intent, generates appropriate search queries, and retrieves relevant results without requiring users to learn complex search syntax or formats. This enables adaptable search capability while keeping the user interface simple.
2Ease of operation
If users search with vague descriptions, then ease of operation improves, but search accuracy decreases
Solution Approach 1:
The system implements feedback mechanisms where the LLM refines search queries based on initial results and user interactions. When users provide vague descriptions, the system iteratively improves search accuracy by analyzing context, adjusting query parameters, and learning from user preferences, thus maintaining both ease of operation and search precision.
Solution Approach 2:
The patent dynamically changes search parameters based on the vagueness of user queries. The LLM adjusts search depth, result ranking criteria, and query expansion levels according to the specificity of the input, enabling accurate results even from vague descriptions while keeping the interface simple.
3Productivity
If data is organized in traditional file systems, then storage structure is simple, but retrieval time increases with large data volumes
Solution Approach 1:
The patent segments the storage system into multiple components: traditional file system for simple organization, LLM processing layer for intelligent query understanding, and vector database for semantic search. This segmentation enables fast retrieval through specialized components while maintaining the simplicity of the underlying file system structure.
Solution Approach 2:
The system adds a semantic dimension to traditional file-based storage by implementing vector embeddings and semantic indexing alongside the conventional file hierarchy. This additional dimension enables rapid retrieval based on meaning and context rather than just file paths, improving productivity without completely replacing the simple storage structure.
Data Source
AI summary
A set of metadata is generated for a file based on file characteristics and a vector embedding is calculated using the set of metadata. A distance between the vector embedding and at least one other vector embedding is used to determine the file storage location. The at least one other vector embedding represents at least one other corresponding set of metadata generated for at least one other file. In one aspect, a combined access latency for the file and the at least one other file is considered in determining the storage location. In another aspect, a text based request is received to search for at least one file indicating a criterion not specifically identifying the at least one file. The text based request is converted into a structured command using a Large Language Model (LLM) to identify at least one storage location for the at least one file.