Hierarchical Data Loader Framework for Error-Free Loading
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing data loading methods, such as SQL scripts, are prone to errors and require extensive editing and recompilation, leading to inefficient data loading and testing processes, especially in complex hierarchical frameworks like those used in financial transactions.
Innovation Solution
The implementation of a hierarchical framework using domain objects and data loaders that evaluate and execute dependencies to efficiently load and clean data, providing compile-time feedback and error resolution, and allowing for reuse across different frameworks.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If SQL scripts are used to load data, then data can be loaded into the data structure, but the process is prone to errors and requires extensive editing and recompilation
Solution Approach 1:
The patent segments the data loading process into multiple independent data loaders, each responsible for a specific data object type (e.g., issuer data loader, company data loader, account data loader). This segmentation allows individual loaders to be developed, tested, and maintained independently, reducing errors and eliminating the need for extensive recompilation when changes are made.
Solution Approach 2:
The framework implements compile-time feedback by evaluating data loader dependencies and detecting errors before execution. The system provides feedback about missing dependencies, circular references, and other issues during the compilation phase, allowing developers to correct errors before running the actual data loading process.
2Adaptability or versatility
If SQL scripts are written to be specific to particular data loads, then the data can be loaded as desired, but the scripts cannot be easily reused across different frameworks
Solution Approach 1:
The patent creates a universal data loader framework that can be applied across different testing frameworks and environments. Each data loader is designed as a standalone component with standardized interfaces, allowing the same data loader to be reused in different contexts without modification. The framework handles framework-specific details, enabling the data loaders themselves to remain framework-agnostic.
3Productivity
If data loaders with dependencies are implemented in a hierarchical framework, then efficient data loading and cleaning is achieved, but the system complexity increases
Solution Approach 1:
The patent introduces a central evaluation mechanism that acts as an intermediary between data loaders and their dependencies. This evaluator automatically determines the execution order of data loaders based on their dependency relationships, managing the hierarchical complexity centrally rather than requiring complex coordination between individual loaders. This mediator handles the complexity of dependency resolution, allowing individual loaders to remain simple.
Data Source
AI summary
Systems and methods are provided for loading data based on a data hierarchical framework for an entity. An exemplary method includes, for a data load related to the entity, determining whether a data loader associated with a root node requires action. And, when the data loader associated with the root node requires action, the method includes identifying a first dependency of the data loader associated with the root node of the hierarchical framework on another data loader associated with another node of the hierarchical framework, and executing the another data loader and then executing the data loader associated with the root node, whereby data is loaded to the data structure consistent with the data hierarchical framework.