Asynchronous Data Unit Test Framework for Integrity Validation
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing unit test frameworks primarily validate functions rather than underlying data, which can lead to issues in data integrity and downstream system compatibility, especially in data management systems where transformation software changes over time.
Innovation Solution
An improved unit test framework that validates large datasets by executing data unit tests asynchronously with the data transformation process, using a 'given, when, then' syntax to load data into memory, check preconditions, and apply unit test logic, generating notifications upon failures without interrupting the transformation process.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If unit test frameworks validate functions, then function correctness is ensured, but data integrity and downstream system compatibility cannot be guaranteed
Solution Approach 1:
The patent segments the validation process into distinct data unit tests that can be independently configured and executed. Each test targets specific data aspects (format, content, relationships) separately, allowing comprehensive data validation without requiring a monolithic complex validation system. This segmentation enables targeted validation of data integrity while maintaining manageable test complexity.
Solution Approach 2:
The patent introduces data unit tests as an intermediary layer between data transformation processes and downstream systems. These tests act as mediators that validate data without directly interfering with the transformation process, ensuring data integrity while maintaining system compatibility. The intermediary tests provide a buffer that protects downstream systems from data issues.
2Reliability
If data unit tests are executed synchronously with data transformation process, then validation thoroughness is improved, but data transformation productivity decreases
Solution Approach 1:
The patent implements preliminary action by executing data unit tests after data transformation steps are completed but before downstream systems process the data. This timing allows thorough validation to occur in advance, ensuring data integrity without blocking the transformation process. The tests run in the background, providing validation thoroughness while maintaining transformation productivity.
Solution Approach 2:
The patent applies dynamics by making the validation process adaptable and flexible. Data unit tests can be configured to run at different stages and with different intensities based on data criticality. This dynamic approach allows the system to adjust validation thoroughness according to specific data requirements while maintaining overall transformation throughput, balancing reliability and productivity.
3Productivity
If data unit tests are executed asynchronously, then data transformation process continues uninterrupted, but detection of data issues is delayed
Solution Approach 1:
The patent implements feedback mechanisms where data unit tests continuously monitor and report on data quality issues. When async tests detect problems, feedback is provided to the transformation process, enabling corrective actions. This feedback loop ensures that while detection may be slightly delayed, issues are identified and addressed promptly, maintaining transformation continuity while minimizing the impact of detection delay.
Solution Approach 2:
By executing tests asynchronously in the background, the system performs validation as a preliminary action that does not block the main transformation process. The tests prepare validation results and notifications in advance, so when issues are detected, the information is ready for immediate action. This approach maintains productivity while managing detection timing through proactive background validation.
4Reliability
If comprehensive data validation is performed on entire datasets, then data integrity is ensured, but processing time and resources increase
Solution Approach 1:
The patent applies partial action by allowing users to configure data unit tests to validate specific portions or aspects of datasets rather than requiring complete validation of entire datasets. Tests can target particular data elements, formats, or relationships based on criticality. This selective validation ensures data integrity for critical elements while reducing overall processing time and resource consumption.
Solution Approach 2:
The patent segments data validation into manageable units that can be executed independently and in parallel. By dividing comprehensive validation into smaller data unit tests, the system can process validation tasks more efficiently, reducing total processing time while maintaining thoroughness. Segmentation also enables parallel execution, further optimizing the balance between integrity assurance and processing time.
Data Source
AI summary
An improved unit test framework that validates large datasets generated by a data management system is described herein. Typical unit test frameworks validate functions. However, the improved unit test framework validates the underlying data. For example, after each step of a data transformation process implemented by the data management system, the data management system can execute a data unit test that loads data sets into memory, checks a set of preconditions, and applies unit test logic to the loaded data sets. In some embodiments, the data management system executes the data unit tests asynchronously with the data transformation processes and therefore do not interfere with the data transformation processes. Rather, the data management system generates and transmits a notification when any step of the data transformation process fails a particular data unit test.