Data Processing Apparatus for Early Start Timing
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing ETL systems wait for all record data to be collected before starting the processing, leading to delayed start times and reduced processing efficiency due to random data arrival times from link destination systems.
Innovation Solution
An information processing apparatus that collects record data from multiple sources and starts processing without waiting for all data to arrive, using definition information to determine when to initiate the process, and segments data conversion processes into individual operations to advance the start timing and increase efficiency.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If the system waits for all record data to be collected before starting processing, then data completeness is ensured, but processing start time is delayed and efficiency is reduced
Solution Approach 1:
The system performs preliminary actions by starting the processing function as soon as a predetermined piece of record data is collected, without waiting for all record data to arrive. This allows the processing to begin in advance while maintaining data completeness through subsequent updates when additional data arrives.
Solution Approach 2:
The system dynamically adjusts the processing based on data arrival states. The processing function can start with partial data and continue to update results as more data becomes available, making the system flexible rather than rigid in its data collection requirements.
2Reliability
If the system waits for all record data to be collected before starting processing, then accurate result data is generated, but processing efficiency is reduced
Solution Approach 1:
The system performs preliminary processing actions as soon as sufficient data is available, generating initial result data that can be refined later. This preliminary action improves efficiency while maintaining accuracy through subsequent updates.
Solution Approach 2:
The processing function operates continuously rather than in discrete batches. It starts when a predetermined piece of data arrives and continues to process and update results as additional data becomes available, eliminating idle waiting time while maintaining accuracy.
3Productivity
If the system processes data as it arrives without waiting for complete collection, then processing efficiency is improved, but data completeness may be compromised
Solution Approach 1:
The system takes preliminary action by starting processing when a predetermined piece of record data is collected, which serves as a threshold for initiating useful work while maintaining the option to update results as more complete data arrives.
Solution Approach 2:
The system incorporates feedback mechanisms where the processing function can be updated with additional record data as it arrives. This allows the system to refine and correct results based on complete data while having already produced preliminary results from partial data.
Data Source
AI summary
An apparatus collects record data from a plurality of information sources. When a process to be performed on the collected record data is determined, with reference to definition information on the process, to be able to generate result data from a single piece of the record data, the apparatus starts the process on the collected record data without waiting for an arrival of all the record data as a collection target from the plurality of information sources. When pieces of the record data are required to be referenced in order for the process to generate the result data and the pieces of the record data includes an uncollected piece of the record data, the apparatus starts the process in response to completion of collecting all the record data as the collection target from the plurality of information sources.