NoSQL Consistency via Asynchronous Transaction File Scanning
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
NoSQL databases lack transaction management tools, forcing current implementations to compromise between single table updates and providing a correction transaction mechanism, which hampers maintaining consistency between NoSQL databases and non-transactional content associated with files in Big Data environments.
Innovation Solution
A system that identifies non-transactional data, stores it in a transaction file with a corresponding transaction ID, scans the transaction file using an asynchronous process to locate successful transactions, and distributes artifacts to their final locations, ensuring consistency between the NoSQL database and non-transactional content.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Adaptability or versatility
If NoSQL database is used for storing data in Big Data environment, then data storage and retrieval flexibility is improved, but transaction management capability deteriorates
Solution Approach 1:
The patent introduces an intermediary transaction management layer that sits between the NoSQL database and the application layer. This intermediary component intercepts write operations, manages transaction states, and coordinates multi-table operations without requiring the NoSQL database itself to provide native transaction support. The intermediary maintains a transaction log and coordinates commits across multiple tables, effectively bridging the gap between NoSQL flexibility and ACID transaction requirements.
2Productivity
If multiple file outputs are generated for analytical calculations, then data processing capability is improved, but data consistency between files and database deteriorates
Solution Approach 1:
The patent implements a feedback mechanism where the system continuously monitors and verifies consistency between the NoSQL database and generated file outputs. After data is written to the database and corresponding files are generated, the system performs verification checks to ensure data coherence. If inconsistencies are detected, corrective actions are triggered to restore consistency, providing continuous feedback loops that maintain data integrity across multiple output channels.
Solution Approach 2:
The system performs preliminary consistency planning before generating multiple file outputs. Transaction IDs and consistency markers are established in advance, and a coordination framework is set up before the actual data processing and file generation occur. This preliminary setup enables the system to track and verify consistency across all output files systematically, rather than attempting to verify consistency after the fact.
3Reliability
If correction transaction mechanism is implemented, then data consistency is improved, but system complexity deteriorates
Solution Approach 1:
The patent extracts the complex transaction management and correction logic into a separate, dedicated component rather than embedding it throughout the entire system. This extracted transaction management module handles all consistency corrections in one localized place, making the complexity manageable and isolated. The rest of the system can interact with this module through simple interfaces without needing to understand or implement complex correction logic themselves.
Data Source
AI summary
A system, method, and computer program product are provided for maintaining consistency between a NoSQL database and non-transactional content associated with one or more files. In operation, the system identifies one or more files including non-transactional data for maintaining consistency between the non-transactional data and a NoSQL database associated with the system. The system stores the non-transactional data associated with the one or more files in a transaction file. The system stores artifacts associated with the non-transactional data in a file with an ending marker and stores the non-transactional data with a corresponding transaction ID. The system stores data of the transaction file in the NoSQL database associated with the system. The system stores the transactional data with the corresponding transaction ID. The system scans the transaction file utilizing an asynchronous process to locate successful transactions. Moreover, the system distributes the artifacts associated with the successful transactions to their corresponding final locations.