Database Application Integration with Big Data Infrastructure
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Traditional data processing applications are inadequate for handling exceptionally voluminous and complex big data sets, requiring advanced big data infrastructure for ingestion, cleansing, storage, analysis, sharing, transformation, and visualization.
Innovation Solution
A system and method for integrating database applications with big data infrastructure, using a data processor and memory to execute scripts in a data engine for retrieving and manipulating data from big data stores, with secure connections and error monitoring, allowing for minimal knowledge of the underlying big data store and supporting development and execution of applications with abstraction layers.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Productivity
If traditional data processing applications are used, then ease of operation is maintained, but productivity deteriorates due to inability to handle voluminous and complex big data sets
Solution Approach 1:
The patent introduces a data engine as an intermediary component between the application and the big data store. The data engine handles the complexity of data retrieval and manipulation by executing scripts, while the application simply sends requests and receives results. This mediator resolves the contradiction by absorbing the system complexity in the data engine layer while maintaining simple application operation interfaces.
Solution Approach 2:
The system is segmented into distinct functional layers: the application layer for business logic, the data engine layer for data processing, and the big data store layer for storage. This segmentation allows each component to be optimized independently - the application remains simple while the data engine handles the complexity of big data processing, resolving the productivity-complexity contradiction.
2Productivity
If big data infrastructure is implemented, then productivity is improved for handling big data, but ease of operation deteriorates due to requiring extensive knowledge of underlying systems
Solution Approach 1:
The data engine serves as a mediator that translates simple application requests into complex data retrieval operations. Applications don't need to understand the underlying big data store structure - they simply send requests to the data engine, which handles the complexity and returns results. This maintains ease of operation while enabling big data processing capability.
Solution Approach 2:
The data engine performs self-service by automatically executing scripts to retrieve and manipulate data from the big data store. The system handles its own complexity internally without requiring users to understand or manage the underlying infrastructure, maintaining operational simplicity while providing advanced big data handling capabilities.
3Reliability
If data retrieval scripts are executed directly in the application, then ease of operation is maintained, but reliability deteriorates due to lack of error monitoring and secure connection management
Solution Approach 1:
The data engine acts as an intermediary that manages secure connections and error monitoring between the application and the big data store. It handles connection establishment, data transfer security, and error detection automatically, improving reliability while the application simply sends requests and receives results without managing connection complexity.
Solution Approach 2:
The system implements feedback mechanisms where the data engine monitors data transfer and detects errors, then provides status information back to the application. This feedback loop improves reliability by enabling error detection and handling, while the application receives simplified status information without needing to manage the complexity of connection monitoring itself.
Data Source
AI summary
A method may include receiving, from a client, an indication to include, in an application, a data processing pipeline implementing a data processing procedure that requires interaction with a data engine to retrieve and/or manipulate data in a data store coupled with the data engine. The application may be executed by exporting, to the data engine, a script corresponding to the data processing pipeline. Exporting the script may enable the data engine to execute the script to retrieve and/or manipulate data in the data store. The transfer of data resulting from the data engine executing the script may be monitored, for example, based on messages from the data engine. The messages may be exchanged via a message broker. Related systems and articles of manufacture, including computer program products, are also provided.