On-demand Data Profile Integration via Lookup File Segmentation
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing data processing systems face inefficiencies in updating data profiles with new fields, as they often require updating thousands of profiles even if only a few are requested, leading to wastage of computational and memory resources.
Innovation Solution
The system allows users to browse a data catalog, select fields from multiple data sources to integrate into a data profile, and generate a lookup file with values for specified fields. This lookup file is stored and updated only when a profile is requested, ensuring that only used data is integrated into the profile.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If the system updates all data profiles with new fields from the data catalog, then the data profiles are kept up-to-date with available data, but computational and memory resources are wasted by updating thousands of profiles even when only a few are requested
Solution Approach 1:
The patent segments the data profile update process by creating a lookup file that contains only the specific fields requested by the user, rather than including all available fields from the data catalog. This segmentation allows the system to update only the necessary profiles with only the necessary fields, eliminating computational waste while maintaining data completeness for requested profiles.
Solution Approach 2:
The patent applies partial action by generating a lookup file with a subset of fields from the data catalog based on user selection, rather than including all fields. The system then performs partial updates on data profiles using only this selected subset, avoiding the excessive computation that would result from updating all profiles with all available fields.
2Quantity of substance
If the system integrates fields from multiple data sources into data profiles, then the data profiles contain comprehensive information, but the processing time and computational complexity increase
Solution Approach 1:
The patent performs preliminary action by pre-generating a lookup file that contains only the specific fields selected from the data catalog before the actual profile update process. This preliminary filtering and organization of data reduces the complexity of the subsequent integration process, allowing the system to efficiently combine fields from multiple data sources without excessive processing time.
Solution Approach 2:
The patent introduces a lookup file as an intermediary data structure that mediates between the data catalog and the data profiles. This intermediary contains pre-selected fields from the data catalog and serves as an efficient source for updating profiles, reducing the direct computational complexity of integrating fields from multiple data sources while maintaining comprehensive data integration.
3Speed
If the system stores all available fields from the data catalog in memory, then quick access to any field is possible, but memory resources are consumed by storing data that may never be used
Solution Approach 1:
The patent applies local quality by creating a lookup file that contains only the specific fields selected by the user from the data catalog, rather than storing all available fields in memory. This localized data structure optimizes memory usage by storing only the relevant fields needed for the current operation, while still enabling quick access to those specific fields when needed.
Data Source
AI summary
A method implemented by a data processing system for enabling a user to browse a data catalog and select fields from multiple data sources to be integrated into a data profile so that, when a request is received for the data profile, data from those fields can be made available efficiently and immediately.