Catalog Update Archiving via Object Bundling and Indexing
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Electronic catalog systems face challenges in managing and querying large quantities of small data updates, as archive storage systems are inefficient for handling hundreds of millions to billions of kilobyte-sized records, and retrieving combined archived and non-archived data is difficult, impacting performance and user experience.
Innovation Solution
Organizing item data updates into objects of predefined sizes (16 MB to 64 MB) with keys and versions, allowing for efficient storage and querying in archive storage systems without affecting the electronic catalog's performance, by creating collections of records and using metadata for indexing and compression.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Quantity of substance
If data is moved to archive storage systems, then storage capacity is improved, but data retrieval efficiency deteriorates
Solution Approach 1:
The patent introduces an intermediary indexing mechanism that stores metadata about archived data objects. This index acts as a mediator between the query system and the archived data, enabling efficient location and retrieval of specific objects without requiring full scans of the archive storage system.
Solution Approach 2:
The system performs preliminary actions by creating and maintaining indexes of archived data objects before retrieval operations. The index is built in advance with information about data object locations, allowing for rapid retrieval when queries are executed against the archived data.
2Ease of operation
If all update data is kept in a single database, then data accessibility is improved, but system performance deteriorates
Solution Approach 1:
The patent divides the large volume of catalog update data into two separate segments: recently accessed data remains in the active database for fast access, while older or less frequently accessed data is moved to archive storage. This segmentation allows the system to maintain high performance for active operations while preserving historical data.
Solution Approach 2:
The system extracts data objects from the active database that are no longer frequently accessed and moves them to archive storage. This extraction process maintains data accessibility through the indexing mechanism while freeing up database resources and improving overall system performance.
3Reliability
If archive storage systems store small records, then storage efficiency deteriorates, but data preservation is improved
Solution Approach 1:
The patent merges multiple small catalog update records into consolidated data objects for archiving. By combining numerous small records into larger bundled objects, the system achieves more efficient utilization of archive storage systems while preserving all the individual record data for future retrieval.
Data Source
AI summary
Systems and methods are provided for archiving and querying data updates received or generated by an update processing system associated with an electronic catalog. In some embodiments, item data updates associated with items represented in an electronic catalog are formed into new objects consisting of collections of records of item data updates. The new objects may have keys and versions associated with each record within them. The use of these objects allows item data updates, which are generally on the order of a few kilobytes, to be efficiently stored in archive storage systems, and to be queried without affecting the performance of the system that hosts the electronic catalog.