Data Access Pipeline With Detection Rules for Stable Data Quality
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing data middle platforms struggle to ensure data quality (completeness, timeliness, and accuracy) under complex network conditions and multiple independently evolving data source systems, leading to high operation and maintenance costs.
Innovation Solution
A data access method and apparatus that includes sending a data connection request with configuration information, loading and parsing data based on connection configuration, and accessing it to a target database, ensuring data quality through multi-dimensional detection and standardized processes.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Adaptability or versatility
If data is accessed from multiple independently evolving data source systems under complex network conditions, then data accessibility and versatility are improved, but data quality (completeness, timeliness, accuracy) deteriorates and operation and maintenance costs increase
Solution Approach 1:
The patent introduces a data detection mechanism as an intermediary layer between data source systems and the data middle platform. This detector receives data from various data source systems, validates it against predefined detection conditions (completeness, timeliness, accuracy), and only passes detected data to the platform. This intermediary approach allows the system to access multiple data sources while maintaining data quality through systematic validation.
Solution Approach 2:
The patent implements a feedback mechanism where the data detection result is fed back into the data access process. When data fails detection (e.g., incomplete, late, or inaccurate), the system can reject the data or trigger alerts, ensuring that only quality-assured data enters the platform. This feedback loop continuously monitors and maintains data quality across multiple heterogeneous sources.
2Reliability
If data detection and validation processes are implemented to ensure data quality, then data accuracy and reliability are improved, but processing time and operational complexity increase
Solution Approach 1:
The patent performs data detection and validation as a preliminary action before data enters the middle platform. By establishing detection conditions and validation rules in advance, the system can automatically filter and verify data upfront, reducing the need for complex post-processing and ensuring quality before integration. This preliminary validation simplifies subsequent operational processes.
Solution Approach 2:
The patent segments the data access process into distinct modules: data loading, data detection, and data access to the platform. This segmentation allows each component to focus on specific tasks (loading from sources, validating against conditions, transferring to platform), making the overall complex process more manageable and easier to maintain through modular architecture.
3Reliability
If comprehensive data detection is performed to ensure completeness and accuracy, then data quality is improved, but data processing speed and timeliness may deteriorate
Solution Approach 1:
The patent applies partial detection by implementing detection conditions that are sufficient to ensure data quality without requiring exhaustive validation of every possible data attribute. The detection mechanism focuses on critical quality dimensions (completeness, timeliness, accuracy) rather than performing complete data analysis, thereby maintaining acceptable processing speeds while ensuring necessary data quality standards.
Data Source
AI summary
The present disclosure relates to a data access method and apparatus, an electronic device, and a storage medium. The method includes: sending a data connection request to a data source system, where the data connection request carries connection configuration information, and the connection configuration information is used to represent configuration information required for completing data connection with the data source system; loading data to be accessed, as determined by the data source system in response to the data connection request; parsing the successfully-loaded data to be accessed, and detecting the parsed data to be accessed based on the connection configuration information; and accessing the successfully-detected data to be accessed to a target database.