Central Server Data Slicing and Asynchronous Pull for Security Systems
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing security systems face performance bottlenecks and downtime due to time-consuming snapshot generation, table level locks, and the need for perfect database replicas, which hinder scalability and require costly re-installations during upgrades and installations.
Innovation Solution
Implementing a central server with parallelism, stateless queries, data slicing, and asynchronous pull mechanisms, allowing for configurable database upgrades, different database versions, and avoiding table level locks through intelligent thread management and data queuing, enabling high scalability and efficient data processing.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If Microsoft replication with distributor/publisher architecture is used to replicate data from regional servers to central server, then data consistency is maintained, but snapshot generation becomes time-consuming (taking days for large databases) and slows down system startup
Solution Approach 1:
The patent divides the database replication process into incremental updates rather than requiring complete snapshots. Data is replicated in smaller chunks over time, allowing large databases to be synchronized without taking days to complete, thus resolving the contradiction between maintaining data consistency and reducing snapshot generation time.
Solution Approach 2:
The system performs preliminary actions by establishing replication relationships and setting up data capture mechanisms before full data synchronization is needed. This allows the replication infrastructure to be ready and waiting, so when data needs to be replicated, the process can begin immediately without lengthy snapshot generation.
2Reliability
If transactional replication polls regional server databases at regular intervals to maintain data consistency, then data synchronization is achieved, but database performance degrades to the point where replication must be disabled
Solution Approach 1:
The patent implements change data capture (CDC) technology that allows the database to self-report changes as they occur through triggers or log-based mechanisms, rather than requiring external polling. This eliminates the performance degradation caused by regular interval polling while maintaining real-time data synchronization, as the database only processes actual change events.
Solution Approach 2:
The system uses feedback mechanisms where the central server receives notifications from regional servers when data changes occur. This event-driven approach allows synchronization to happen only when necessary, avoiding the continuous performance impact of periodic polling while maintaining data consistency.
3Ease of operation
If data pulling schedule is common for all regions to simplify coordination, then system management is easier, but table level locks are created when multiple regions attempt to write to the same table simultaneously
Solution Approach 1:
The patent assigns different data pulling schedules to different regions based on their specific needs and characteristics. Instead of a uniform schedule for all regions, each region can have optimized timing that avoids conflicts with other regions, thereby preventing table level locks while maintaining ease of management through localized configuration.
Solution Approach 2:
The system implements dynamic data pulling schedules that can adjust timing based on current system conditions, region activity levels, and conflict detection. This allows the coordination mechanism to remain simple in structure while adapting in real-time to prevent table level locks, resolving the contradiction between operational simplicity and system stability.
4Reliability
If central server database replica must be perfect match with regional server database to ensure consistency, then data integrity is maintained, but database upgrades require complete re-installation and cause significant downtime
Solution Approach 1:
The patent performs preliminary actions by creating a test environment that mirrors the production database schema before upgrades are attempted. This allows validation of upgrade compatibility and automated schema migration planning without affecting the production system, enabling upgrades to proceed with minimal downtime while maintaining data integrity through controlled synchronization.
Solution Approach 2:
The system allows database version parameters to differ between central and regional servers during upgrade transitions. Instead of requiring perfect version matching, the system manages schema evolution through controlled parameter changes and automated synchronization, enabling incremental upgrades without complete re-installation and significantly reducing downtime.
5Reliability
If multiple copies of regional server databases are created and copied to central server using replication, then data redundancy is achieved, but processing times slow down and memory consumption increases
Solution Approach 1:
The patent extracts only the essential data changes from regional servers and transmits only those increments to the central server, rather than copying entire database copies. This selective data extraction maintains data redundancy for critical information while dramatically reducing processing overhead and memory consumption associated with transmitting and storing complete database replicas.
Data Source
AI summary
Systems and methods for processing data using parallelism, stateless queries, data slicing, and/or asynchronous pull are provided. Some methods can include each of a plurality of data pull devices of a central server retrieving data from a respective one of a plurality of regional server databases, wherein a number of the plurality of data pull devices equals a number of the plurality of regional server databases, and wherein the plurality of data pull devices function asynchronously. Additionally or alternatively, some methods can include each of a plurality of threads of a central server retrieving a predetermined size of data from a plurality of regional server databases, wherein the plurality of threads are stateless so that any of the plurality of threads can retrieve the data from any of the plurality of regional server databases.