Air Freight Rate Data Caching Sharding for Scalability
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Conventional caching systems for air freight rate data lack horizontal scalability due to the use of whole set data copies, leading to memory resource bottlenecks as the amount of cached data increases.
Innovation Solution
The method involves transforming air freight rate data into a data format for a first-level cache, performing data sharding based on origin and destination cities to generate data shards, and storing these shards into a second-level cache, where each data node includes a copy of the whole set and subsets of data dividable by the sharding algorithm, improving horizontal scalability.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Adaptability or versatility
If all cached data copies are whole sets, then data integrity is maintained, but memory resources become a bottleneck that limits horizontal scalability
Solution Approach 1:
The patent divides the air freight rate data into multiple shards based on origin and destination cities. Each data node stores only a portion of the data (one subset) rather than complete copies, enabling horizontal scalability while reducing memory consumption. The sharding algorithm partitions data such that different nodes hold different segments, allowing the system to scale by adding more nodes with smaller data portions.
2Adaptability or versatility
If data is distributed across multiple nodes with subsets, then horizontal scalability is improved, but data retrieval complexity increases
Solution Approach 1:
The patent introduces a data agent as an intermediary component that manages data sharding and retrieval operations. The data agent coordinates between the user, the cache system, and the underlying data storage, handling the complexity of locating and assembling data from multiple shards. This mediator abstracts the distributed data access complexity from the user, presenting a unified interface while enabling scalable data distribution.
3Quantity of substance
If data sharding is performed based on origin and destination cities, then memory efficiency is improved, but data access time may increase due to distributed storage
Solution Approach 1:
The patent implements local quality by organizing data shards based on geographic characteristics (origin and destination cities). Each data node is optimized to store data relevant to specific routes, allowing efficient local access for queries involving those routes. This geographic-based partitioning ensures that commonly accessed route data is locally available at appropriate nodes, reducing access time while maintaining memory efficiency through targeted data distribution.
Data Source
AI summary
An air freight rate data caching method and system. The method includes converting air freight rate data into a data format of a first-level cache, and storing same in the first-level cache; performing, on the basis of a flight origin city and a flight destination city, data fragmentation on the air freight rate data stored in the first-level cache so as to generate fragmented data; and storing the fragmented data, after same is validated, in a second-level cache. Each data node of the fragmented data cached in the second-level cache only includes part of the air freight rate data on which a fragmentation algorithm can be performed, and therefore, the horizontal expansion capacity of a cache system is improved relative to the case where cached data copies are all complete sets.