Distributed File System Historical Call Lookup
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Large-scale data systems, such as mobile call data systems, face high latency and resource consumption when querying historical records due to their large volumes, which are typically stored in separate systems with higher access complexities, leading to slow response times and system bottlenecks, especially when handling thousands of queries daily.
Innovation Solution
The system divides call data into separate record type groups, stores metadata in an application database, partitions each group by date in a historical record database, and uses hash values to optimize queries, generating record-specific query forms and search expressions to efficiently retrieve data from distributed file systems, thereby reducing latency and resource consumption.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Quantity of substance
If historical records are stored in separate storage systems with hard disk storage, then large data volumes can be stored, but access latency increases and instruction complexity increases
Solution Approach 1:
The patent segments the large historical data set into multiple smaller partitions organized in a distributed file system. Each partition contains data for a specific time period and is stored in separate files. This segmentation allows the system to store large volumes of data while enabling faster access by querying only relevant partitions rather than scanning the entire data set, thus resolving the contradiction between storage capacity and access latency.
2Productivity
If traditional relational database systems are used to query historical records, then data can be retrieved, but system resources are heavily consumed and response times are slow
Solution Approach 1:
The patent extracts the query processing logic from traditional relational database systems and implements a custom query engine optimized for the distributed file system structure. This custom engine directly queries the partitioned data files using efficient search algorithms, eliminating the overhead of relational database operations. The extraction of query processing from the database system allows for tailored optimization that reduces resource consumption while maintaining or improving query throughput.
3Loss of time
If in-memory database systems are used, then fast access is achieved, but large data volumes cannot be stored
Solution Approach 1:
The patent transitions from a single-dimensional in-memory storage approach to a multi-dimensional distributed file system architecture. Data is organized across multiple dimensions including time periods, data types, and storage nodes. This dimensional organization allows the system to leverage hierarchical storage mechanisms where frequently accessed data can be cached in memory while the bulk of historical data resides on disk, effectively extending the addressable memory space and enabling both fast access and large storage capacity.
Data Source
AI summary
Systems and methods described herein facilitate the search and presentation of historical data for wireless network usage and provide a mechanism for high-redundancy, low-latency record retrieval of data from large data sets. Network devices divide data for a historical data store into separate record type groups, store metadata for each record type in an application database, partition each record type group by date in a historical record database that is different from the application database, and form, within each date partition, buckets of common hash values of a key parameter from each record. When a user performs a query, the network devices generate a record-specific query form based on the record type metadata to obtain lookup parameters; generate a search hash value using a key parameter from the lookup parameters; and generate a query expression based on the record type, lookup parameters, and the search hash value.