Self-service Data Provisioning System with Layered Architecture
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing data provisioning systems face challenges in reducing complexity, increasing speed, and improving efficiency due to the large amount of data being integrated in centralized integrated data platforms, particularly in processing and formatting raw data records from various systems of record (SORs) to conform to data exchange specifications within tight timeframes.
Innovation Solution
The system employs a data exchange hosted on an Oracle Exadata box, which includes a raw data layer for receiving raw data records, a model data layer for processing and formatting data to conform to Oracle or data exchange specifications, and a self-service provisioning mechanism with workspaces for consumers to query and execute scripts, utilizing bulk data extract hybrid job processing to maintain dependencies and reduce processing time, and enforces resource and complexity constraints through testing environments.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Loss of time
If traditional data provisioning methods are used to process and format raw data records from various systems of record, then data integration can be achieved, but processing time becomes excessively long and system complexity increases
Solution Approach 1:
The system segments the data provisioning process into distinct layers: a raw data layer for receiving raw data records and a model data layer for processed data. This segmentation allows parallel processing of multiple data records simultaneously, reducing overall processing time while maintaining manageable system complexity through clear separation of concerns.
Solution Approach 2:
The system performs preliminary formatting and modeling of raw data records before they reach the model data layer. By pre-processing data in the raw data layer using templates and data models, the system reduces the complexity of subsequent processing steps and accelerates the overall data provisioning workflow.
2Manufacturing precision
If data records are reformatted to conform to data exchange specifications with predetermined data order and format, then data quality and consistency are improved, but processing complexity and time increase
Solution Approach 1:
The system employs self-service data provisioning where consumers can independently query and execute scripts against the data exchange. This self-service mechanism automatically handles data formatting and conversion to predetermined specifications, ensuring data consistency while reducing the complexity burden on system operators and administrators.
Solution Approach 2:
The system uses configurable templates and data models that define predetermined data formats, orders, and structures. By parameterizing the formatting rules rather than hardcoding them, the system maintains high data format consistency while allowing flexible adjustment without increasing processing complexity.
3Productivity
If bulk data extract hybrid job processing is used to process close to a billion records, then processing speed increases to under 5 hours, but resource consumption and coding complexity must be strictly controlled
Solution Approach 1:
The system implements feedback mechanisms through testing environments that evaluate resource consumption and coding complexity before scripts are executed. This feedback loop allows the system to maintain high processing speeds by automatically identifying and preventing resource-intensive operations, thereby controlling resource management complexity without sacrificing productivity.
Solution Approach 2:
The system processes data in controlled batches rather than attempting to process all billion records simultaneously. By using partial action on subsets of data with strict resource constraints, the system achieves high overall productivity while keeping individual processing tasks manageable in terms of resource consumption and complexity.
4Adaptability or versatility
If multiple consumers query and execute scripts independently in separate workspaces, then system versatility and ease of operation improve, but resource management and coordination become more complex
Solution Approach 1:
The system provides a universal data exchange platform that serves multiple consumers with different requirements. Each consumer can independently query and execute scripts in their own workspace, yet all consumers share the same underlying data models and formatting rules. This universality achieves high adaptability and consumer independence while the shared infrastructure reduces resource coordination complexity compared to fully separate systems.
Data Source
AI summary
A data exchange that provides self-service data provisioning is provided. The data exchange may include a raw data layer, a model data layer, a plurality of workspaces and a testing environment. The raw data layer may be a landing zone for raw data records received from systems of record. The raw data layer may receive a plurality of raw data records, model and process the data records and transfer the data records to the model data layer. The model data layer may be a data layer that includes data modeled to data exchange specifications and enables queries to be executed on the data included in the model data layer. Each workspace may be allocated to a consumer. The consumer may query the plurality of data records within the model data layer. The testing environment may test scripts to ensure that the scripts conform to a predetermined set of testing specifications.