Metadata Extraction Engine for Memory Resource Conservation
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Current file management systems in computer systems and networks face challenges in effectively storing and presenting metadata without depleting memory resources, especially in complex data structures where tracking the source of information across multiple repositories is difficult.
Innovation Solution
A system comprising a database, memory, extraction engine, enrichment engine, portal, and transmission engine that extracts and associates metadata from disparate file repositories, enriches the data by identifying the source, and loads it into memory, allowing for efficient display and communication of end-to-end dataflow, thereby conserving memory resources and improving processor functionality.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Ease of operation
If metadata is stored alongside data in the same memory, then data access is simplified, but memory resources are depleted
Solution Approach 1:
The system divides memory resources into separate segments: one for storing data and another for storing metadata. The database memory stores only data, while a separate processing system stores and manages metadata about the data. This segmentation allows data access to remain simple while preventing memory depletion by dedicating specific memory regions to specific purposes.
Solution Approach 2:
A separate processing system acts as an intermediary between the database and the metadata management needs. This intermediary system extracts, enriches, and manages metadata without requiring the database memory to store both data and metadata, thus conserving database memory resources while maintaining ease of data access.
2Device complexity
If a centralized system stores and processes both data and metadata, then processing is simplified, but memory resources are depleted
Solution Approach 1:
The system architecture is segmented into a database system for data storage and a separate processing system for metadata management. This structural segmentation reduces the complexity of the database system itself while distributing the processing load appropriately, and it prevents memory depletion by keeping data and metadata in separate memory spaces.
Solution Approach 2:
Metadata is extracted from the data storage system and managed separately by a dedicated processing system. This extraction allows the database to focus solely on data storage and retrieval, simplifying its processing structure, while the separate metadata system manages information about the data without competing for the same memory resources.
3Productivity
If processor resources are dedicated to managing and processing incoming data, then data management is improved, but processor functionality for other tasks is reduced
Solution Approach 1:
The metadata management functions are extracted from the main processor responsibilities and assigned to a separate processing system. This allows the main processor to focus on core data management tasks with improved productivity, while the separate system handles metadata extraction, enrichment, and management, preserving the main processor's functionality for other versatile tasks.
Solution Approach 2:
A separate processing system acts as an intermediary that handles metadata management tasks, freeing the main processor from these responsibilities. This intermediary approach improves data management productivity through specialized processing while maintaining the main processor's adaptability and versatility for other computational tasks.
Data Source
AI summary
A system for file management in data structures is disclosed. The system includes a memory, an extraction engine, an enrichment engine, a portal, and a transmission engine. The memory stores a first database and a memory. The extraction engine extracts columns and fields and associates them with extracted terms. The enrichment engine determines an end-to-end dataflow of the data from extracted metadata and loads the associated data into the memory. Enrichment engine performs either a full load comprising loading all the associated data onto the memory or a delta load comprising comparing the extracted data with data stored in the memory and loading any different data from the extracted data onto the memory. The enrichment engine also generates a journal recording metadata associated with the full or delta load. A portal displays the end-to-end dataflow of the associated data and a transmission engine transmits communication identifying incomplete associated data.