Selective Bandwidth Offloading for Large-Scale Data Indexing
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing on-premises systems are inadequate for indexing large-scale data sets due to locked data integrations in static formats, which restrict granular searching and require significant bandwidth, often exceeding processing capabilities, leading to inefficient data management.
Innovation Solution
A system that utilizes a primary server to determine if the bandwidth required for indexing large-scale data sets exceeds a threshold, and if so, allocates secondary server resources to index the data in parallel, employing a machine-learned model to estimate bandwidth needs and reserve necessary resources, allowing for efficient indexing and searching of locked data.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Ease of manufacture
If on-premises systems use locked data integrations in static formats, then data processing is simplified, but granular searching capability deteriorates and bandwidth requirements increase excessively
Solution Approach 1:
The patent segments the data indexing process by creating separate index data structures that are independent from the original locked data integrations. This allows granular searching capabilities while maintaining the simplicity of the original data formats, resolving the contradiction between processing simplicity and search versatility.
Solution Approach 2:
The patent introduces index data as an intermediary between the locked data integrations and search queries. This intermediary layer enables granular searching without requiring changes to the original data structure, maintaining both processing simplicity and search capability.
2Quantity of substance
If on-premises systems process large-scale data sets, then data integration is achieved, but processing capacity becomes inadequate and bandwidth requirements exceed thresholds
Solution Approach 1:
The patent extracts the indexing operation from the main data processing flow and handles it separately through secondary servers. This allows large-scale data integration while distributing the processing load, preventing bandwidth overload and maintaining adequate processing capacity.
Solution Approach 2:
The patent adds a dimensional separation between primary data processing and secondary indexing operations. By handling indexing as a separate dimension, the system can process large data sets without overwhelming the primary processing capacity or bandwidth.
3Productivity
If secondary servers are allocated for indexing, then bandwidth requirements are met and indexing speed improves, but system complexity increases
Solution Approach 1:
The patent implements preliminary action by pre-determining bandwidth thresholds and pre-configuring secondary server allocation policies. This allows the system to handle high-speed indexing requirements while maintaining manageable complexity through predetermined rules rather than complex real-time decision-making.
Solution Approach 2:
The patent implements self-service mechanisms where the system automatically monitors bandwidth usage and triggers secondary server allocation when thresholds are exceeded. This autonomous operation improves indexing speed while minimizing the need for complex manual configuration and control.
Data Source
AI summary
A system and a method are disclosed for receiving, by a server, based on input by a user, a request to lock a set of data. Responsively, the server modifies the set of data to be locked, and determines whether an amount of bandwidth required to index the locked data exceeds a threshold. Responsive to determining that the amount of bandwidth exceeds the threshold, the server instructs a secondary server to allocate bandwidth to index a first portion of the locked data. The server indexes a second portion of the locked data in parallel with the secondary server indexing the first portion of the locked data, and generates an index by collating the indexed first and second portions of the locked data. The server receives a search request for a portion of the locked data, and retrieves the portion of the locked data based on referencing the index.