De-normalized Data Structure File Generation for Intelligence Reports
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing data processing systems face challenges in handling large volumes of data from multiple sources, particularly in converting structured and unstructured data into a standard format, validating data integrity, and generating actionable intelligence, as they often operate in silos without parallel analytics processing and fail to provide proactive guidance on prioritizing issues.
Innovation Solution
A processor-implemented method and system that obtains files from various sources, validates and consolidates records, performs logic validation, and generates de-normalized data structure files, enabling the creation of intelligence reports by merging validated records and querying for key performance indicators.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Productivity
If existing systems retrieve data from databases in silos without parallel analytics processing, then data processing operations can be performed, but productivity is low and time-consuming
Solution Approach 1:
The patent merges data retrieval operations with analytics processing operations into a single integrated system. The data analytics system simultaneously performs data retrieval from multiple databases and executes analytics processing in parallel, eliminating the siloed approach where these operations were performed separately. This integration enables efficient processing of large volumes of data from multiple sources without time loss.
Solution Approach 2:
The system maintains continuous data processing by performing analytics operations concurrently with data retrieval. The parallel processing architecture ensures that data is continuously analyzed as it is retrieved, rather than waiting for complete data collection before analysis begins. This continuous action significantly reduces processing time and improves productivity.
2Reliability
If existing systems retrieve data from multiple transactional systems, then accurate information can be obtained, but device complexity increases due to system integration challenges
Solution Approach 1:
The patent introduces a centralized data analytics system that acts as an intermediary between multiple transactional systems and downstream applications. This intermediary system retrieves data from various transactional systems, validates and processes the information, and then provides accurate results to consumers. The intermediary approach simplifies the overall architecture by centralizing data retrieval and processing logic, reducing the complexity of direct integrations between multiple systems.
Solution Approach 2:
The data analytics system is designed with multi-functionality to handle diverse data types and sources. It can retrieve, validate, and process data from multiple transactional systems using unified processing logic, eliminating the need for separate specialized systems for each function. This universal approach maintains information accuracy while reducing integration complexity through standardized processing mechanisms.
3Productivity
If existing systems process data without parallel analytics processing, then operations can be completed sequentially, but productivity remains inefficient
Solution Approach 1:
The patent segments the data processing workflow into independent parallel tasks that can be executed simultaneously. The system divides data retrieval operations for different databases and analytics processing operations into separate concurrent threads or processes. This segmentation enables parallel execution of multiple operations without creating a monolithic complex system, as each segment can be managed independently while contributing to overall productivity improvement.
Data Source
AI summary
De-normalized data structure files generation systems and methods are provided. The system obtains files from sources wherein each file include records, parses files to validate records and attributes in the records, identifies a set of similar files from the validated files, and append two or more files from the set of similar files to obtain one or more consolidated files. Each of the one or more consolidate files corresponds to a specific category. The system further a predefined logic validation on each of the one or more consolidated files to obtain a logic validated file for each of the one or more consolidated files. Each logic validated file obtained for the one or more consolidated files include validated records. The system further generates a de-normalized data structure file including de-normalized records by merging each of the logic validated files to be used for generating intelligence reports.