Distributed Database Event Identifier Generation via Supplementary File System
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Distributed database systems face issues with event loss due to unavailability during high-volume write activities, as they cannot generate identifiers for events stored in memory when the database is down, leading to memory overflow and event dropping.
Innovation Solution
A supplementary low-latency file system is used to temporarily store events without identifiers, with an indirect-identifier-generating component assigning identifiers once the database becomes available, ensuring reliable identifier generation and event writing.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Productivity
If events are stored in memory during database unavailability, then the system can maintain write capability, but memory capacity is exceeded and events are dropped
Solution Approach 1:
The patent introduces a supplementary file system as an intermediary storage layer between memory and the main database. When the database is unavailable, events are written to this intermediate file system instead of being dropped or overflowing memory. This mediator absorbs the excess write load and preserves events for later processing when the database becomes available again.
2Reliability
If the database is unavailable during high-volume write activity, then system reliability is maintained, but identifier generation fails and events are lost
Solution Approach 1:
The system performs preliminary actions by writing events to the supplementary file system in advance, before the database becomes available. This ensures events are preserved and ready for processing. Additionally, unique identifiers are generated in advance using an indirect generation mechanism that doesn't require direct database access, so when the database recovers, events can be written immediately without loss.
3Measurement precision
If unique identifiers are generated directly from the database, then identifier uniqueness is ensured, but the system becomes unavailable during database downtime
Solution Approach 1:
The identifier generation process is segmented into two independent parts: (1) indirect identifier generation that can operate without direct database access, and (2) direct database writing that occurs when the database is available. This segmentation allows the system to maintain identifier generation capability during database downtime while ensuring uniqueness through the indirect generation mechanism, eliminating the all-or-noth er dependency on database availability.
Data Source
Figure 1
Figure 2
Figure 3
AI summary
A database system may determine that a database is unavailable to generate an identifier for an event or that events may not be written to the database. As a result, the database system may write the event to a low-latency, supplementary file system. The database system may determine that the database is available at a later time, and read the event from the supplementary file system, store the event in a memory component, and then delete the event from the supplementary file system. The database system may then access the database to generate an identifier for the event, write the event to the database, and then delete the event from the memory component.