State-Machine Database Updates for In-Flight Task Records
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Conventional databases, particularly Cloud-based Platform as a Service (PaaS) databases, face challenges in transactional processing due to prolonged in-flight periods, leading to unreliable and static updates, which can result in data infidelity and inefficient processing, especially in systems handling large volumes of data.
Innovation Solution
A state machine framework for multi-threaded, multi-step task extraction is implemented, allowing updates to in-flight records without actual data copying, enabling efficient processing and error handling, and ensuring data integrity through a suite of tables and views that support disparate processing regimes.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Speed
If conventional databases store data in in-flight periods for efficient retrieval, then data access speed is improved, but data reliability and update capability deteriorate
Solution Approach 1:
The database system segments data into two distinct storage states: in-flight data stored in temporary storage for fast access, and committed data stored in permanent storage for reliability. This segmentation allows the system to maintain high-speed access for active data while ensuring data integrity through proper commit mechanisms, resolving the contradiction between speed and reliability.
Solution Approach 2:
The patent introduces a commit mechanism as an intermediary between data insertion and permanent storage. The commit operation acts as a mediator that ensures data integrity by providing atomicity, consistency, isolation, and durability (ACID properties). This intermediary layer allows the system to maintain fast in-flight data access while ensuring reliable updates through controlled commit operations.
2Productivity
If databases allow rapid insertion of large volumes of data, then productivity is improved, but update capability and transactional processing deteriorate
Solution Approach 1:
The database system dynamically manages data states, allowing data to transition from in-flight temporary storage to committed permanent storage based on operational needs. This dynamic state management enables the system to handle rapid data insertion efficiently while providing update capabilities through the commit mechanism, resolving the contradiction between productivity and ease of operation.
Solution Approach 2:
The system changes the status parameter of data from uncommitted to committed through the commit operation. This parameter change enables the data to transition from a read-optimized state to an update-capable state, allowing the database to maintain high insertion productivity while providing full update capability when needed through controlled parameter transitions.
3Stability of the object's composition
If databases maintain static updates during in-flight periods, then data consistency is improved, but data fidelity and transactional processing deteriorate
Solution Approach 1:
The system performs preliminary actions by maintaining data in an in-flight state in temporary storage before final commitment. This preliminary state allows for fast retrieval and processing while preserving the ability to make updates before the final commit. The preliminary action ensures data consistency during the in-flight period while maintaining data fidelity through the controlled commit process that follows.
Data Source
AI summary
Disclosed are systems and methods for a computerized framework that involves a state machine for multi-threaded, multi-step task extraction that is configured to operate by implementing a suite of tables and corresponding views that do not require any updates and/or data copying, which allows for additional states being adopted for disparate processing regimes. The state machine enables insertions of transactional usage records into several tables in a data warehousing environment, where jobs in the table can be configured to identify the job name along with other key information for how to process such job, as well as information related to what kind of identifier (ID) to use and rules for retry and/or error handling. The framework allows multiple threads to work on subsets of data in a single table and update the status of many records while they are in-flight.