Data Processing System for Unstructured Source Data
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Large-scale processing of unstructured data from various sources, such as aircraft systems, is challenging due to the limitations of standard database management techniques, which struggle with handling and analyzing vast amounts of data efficiently, particularly in identifying and retrieving vital information in a timely manner.
Innovation Solution
A custom configured network tool that automatically receives, processes, categorizes, and manages data from multiple sources by translating and formatting incoming data, detecting vital information associated with out-of-tolerance conditions, and performing analytics to identify trends and predictive maintenance needs.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Productivity
If standard database management techniques are used to process source data, then data storage and retrieval functions are provided, but processing speed and efficiency deteriorate when handling large volumes of unstructured data
Solution Approach 1:
The system segments the data processing task into distinct functional modules: a translation processor for data format conversion, a vital data processor for extracting critical information, and an analytics processor for advanced analysis. This segmentation allows each component to specialize in specific operations, improving overall processing efficiency while maintaining manageable system complexity through modular architecture.
Solution Approach 2:
The translation processor performs preliminary data translation and formatting before the data reaches the vital data processor. By pre-processing the data to convert it into a standardized format and identify potential vital data points early in the pipeline, the system reduces the computational burden on subsequent processing stages, thereby increasing overall productivity without proportionally increasing complexity.
2Loss of time
If standard computing networks are used for data processing, then basic data storage is achieved, but the ability to identify and retrieve vital information in a timely manner deteriorates
Solution Approach 1:
The vital data processor extracts only the critical information from the large volumes of processed data. By implementing a dedicated extraction mechanism that identifies and isolates vital data points based on predefined criteria and patterns, the system minimizes the time required to retrieve important information while ensuring high accuracy in identifying what constitutes vital data, thus addressing both time loss and information loss concerns.
3Productivity
If relational database management systems are used, then data storage is provided, but processing efficiency deteriorates when handling petabytes of data from multiple sources
Solution Approach 1:
The translation processor acts as an intermediary between the raw data sources and the vital data processor. It converts data from various formats and sources into a standardized intermediate representation, enabling efficient processing of large volumes of data without overwhelming the subsequent processing stages. This intermediary layer maintains processing efficiency even as data volume increases to petabyte scales.
Data Source
AI summary
A system and method for processing source data, which includes a translation processor configured on a server to translate a stream of source data received at the server into a stream of formatted data types based on one or more of parameter definitions for the source data and new parameter information for the source data having an unknown data type. The source data can be received at the server from multiple sources and the data format may not be consistent from one source to another. An analytics processor on the server can be configured to operate on the out-of-tolerance data type to perform one or more of trend data analytics, associated data analytics and preventative action analytics.