Data Deduplication via Hash Values and Access Control Identifiers
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Conventional data integration pipelines face inefficiencies in managing duplicate data fragments from disparate sources, leading to increased computational overhead and slower processing times, which degrade user experience.
Innovation Solution
A system and method for de-duplicating data fragments based on hash values and access control identifiers, representing them as single object data source records within a data platform system, and enforcing invariants to manage relationships and generate error logs.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Quantity of substance
If data integration pipelines ingest all data fragments from disparate sources, then data completeness is improved, but computational overhead increases due to duplicate data processing
Solution Approach 1:
The system performs preliminary de-duplication by computing hash values for data fragments during the ingestion phase. This preliminary action identifies duplicate content before full processing occurs, allowing the system to eliminate redundant computational work downstream while maintaining complete data representation through object DSRs.
Solution Approach 2:
The system creates simplified copies (hash values) of data fragments to identify duplicates without processing the full data content. These hash copies enable efficient duplicate detection while the actual data is represented once per unique object through object DSRs, reducing computational overhead while preserving data completeness.
2Manufacturing precision
If data integration pipelines process all ingested data fragments, then data accuracy is improved, but processing time increases due to duplicate fragments
Solution Approach 1:
The system performs preliminary de-duplication using hash value comparison before main processing occurs. This preliminary action groups duplicate fragments and selects representative object DSRs, ensuring data accuracy is maintained through proper representation while eliminating redundant processing time for duplicate content.
Solution Approach 2:
The system extracts the essential identifying characteristic (hash value) from each data fragment to create object DSRs. By taking out this key identifying feature and using it to represent entire groups of duplicate fragments, the system maintains data accuracy through proper object representation while significantly reducing processing time by handling each unique object only once.
3Loss of information
If the system maintains separate records for duplicate data fragments, then data provenance is improved, but system complexity increases
Solution Approach 1:
The system merges multiple duplicate data fragments into single object DSRs that represent entire groups of duplicates. This merging maintains data provenance by preserving the relationship between fragments and their source objects through the object DSR structure, while reducing system complexity by eliminating redundant separate records for duplicate content.
Solution Approach 2:
The object DSR structure serves multiple functions simultaneously: it represents duplicate fragments, maintains provenance information, enables efficient querying, and reduces storage requirements. This multi-functionality allows the system to preserve data provenance without increasing complexity, as the same object DSR structure handles multiple concerns at once.
Data Source
AI summary
Systems and methods are provided for determining multiple fragments of data to be imported, the multiple fragments of data corresponding to different instances of data obtained from one or more external data sources, the different instances of data each corresponding to duplicate content. The multiple fragments of data that each correspond to different instances of duplicate content can be ingested. The multiple fragments of data can be de-duplicated to determine one or more corresponding object data source records (DSRs). The one or more object DSRs can be imported within a data platform system.