Big Data Platform Real-Time Insight Generation
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Users, especially non-technical individuals and organizations, face difficulties in accessing and interpreting large volumes of big data due to access control and technical complexities, leading to delayed and inaccurate insights, and lack of comprehensive tools for comparative analysis.
Innovation Solution
A system that processes transaction data from a big data management system to generate real-time insights by modeling corporate accounts, identifying demographic patterns, and comparing user data to characteristic data, enabling real-time views and flagging out-of-pattern transactions, while storing data in a format suitable for rapid retrieval and processing.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Ease of operation
If users access big data sets directly through queries or mapping, then data access capability is improved, but technical complexity and access control barriers increase
Solution Approach 1:
The patent introduces an intermediary layer that sits between users and the big data management system. This intermediary automatically generates and executes queries, mapping user-friendly requests to the underlying complex data infrastructure without requiring users to understand SQL or data schema details, thus resolving the contradiction between ease of access and technical complexity
Solution Approach 2:
The system implements self-service capabilities where the data access layer automatically handles query generation, optimization, and execution based on user requests. The system autonomously manages the complexity of data retrieval operations, allowing users to access data without needing technical expertise in big data technologies
2Measurement precision
If comprehensive data analysis is performed on large data sets, then insight quality is improved, but processing time increases
Solution Approach 1:
The patent implements preliminary action by pre-processing and pre-aggregating data in the big data management system before user requests arrive. Data is organized, cleaned, and partially analyzed in advance, so when users request insights, the system can quickly retrieve and combine pre-prepared data structures rather than performing full analysis from raw data, thus improving insight quality while reducing processing time
Solution Approach 2:
The system segments the large data sets into smaller, manageable partitions that can be processed in parallel. By dividing the data analysis task into independent segments that can be handled simultaneously across multiple processing nodes, the system maintains comprehensive analysis quality while significantly reducing overall processing time through parallel computation
3Reliability
If sensitive financial data is restricted for security reasons, then data security is improved, but availability of comparative insights deteriorates
Solution Approach 1:
The patent applies local quality by implementing different levels of data access and anonymization for different user groups and data types. Sensitive financial data is protected through selective masking and aggregation at the point of access, allowing comparative insights to be generated from anonymized data while maintaining security controls on raw sensitive information, thus resolving the contradiction between security and information availability
Data Source
AI summary
A system, method, and computer readable medium (collectively, the “system”) is disclosed for generating real-time insights from a big data storage platform. The system may process transaction data stored in a big data management system (BDMS) to generate characteristic data associated with a plurality of demographics. The system may also model an account associated with a corporate ID to identify a demographic from the plurality of demographics associated with the corporate ID. The system may further compare the user data associated with the corporate ID to the characteristic data of the demographic associated with the corporate ID to generate a view responsive to a request. The system may also transmit the view in response to the request, for example, to an interface for use by a program administrator.