Photovoltaic Data Processing via Row-to-Column Storage Segmentation
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Traditional big data processing systems for photovoltaic data require manual merging due to timing errors and data updating issues, leading to inefficiencies and a lack of automatic data update perception.
Innovation Solution
A method and apparatus for processing photovoltaic data that automatically merges updated data by writing it in a row storage format and then converting it to a column storage format when a data threshold is reached, using request identification metadata and resilient distributed dataset objects for deduplication and query optimization.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If manual data merging operation is used to handle timing errors and data updating, then data accuracy can be maintained, but processing time increases and automation is reduced
Solution Approach 1:
The system implements self-service through automatic data merging based on timestamps. The merging module automatically identifies and merges updated data without manual intervention, using timestamp comparison to determine which data to retain. This eliminates the need for manual data merging operations while maintaining data accuracy through automated timestamp-based validation.
Solution Approach 2:
The system performs preliminary action by pre-processing data with timestamps before merging operations. Data is pre-tagged with timing information and organized in a structure that enables automatic identification of updates. This preliminary organization allows the merging module to quickly identify and process only the data that needs updating, reducing overall processing time.
2Quantity of substance
If traditional big data database is used, then data storage capability is sufficient, but automatic perception of data update needs is lost
Solution Approach 1:
The system implements feedback mechanisms where the merging module continuously monitors incoming data timestamps and automatically determines when merging is needed. The module compares timestamps of incoming data with existing data, providing feedback that triggers automatic merging operations when updates are detected. This creates an automated loop that perceives and responds to data update needs without manual intervention.
Solution Approach 2:
The merging module acts as an intermediary between data ingestion and data storage. It sits between the data input stream and the database, automatically analyzing timestamps and coordinating merging operations based on update detection. This intermediary layer enables automatic perception of update needs while maintaining the database's storage capacity.
3Ease of manufacture
If data is written in row storage format, then writing cost is reduced, but query efficiency decreases
Solution Approach 1:
The system segments the storage process into two distinct phases: a writing phase using row storage format and a query phase using column storage format. The merging module performs segmentation by converting row-formatted data to column-formatted data in batches. This segmentation allows inexpensive row-based writing for data ingestion while enabling efficient column-based querying for data retrieval, optimizing both writing cost and query efficiency for different operational stages.
Data Source
Figure 1
Figure 2
Figure 3~5
AI summary
A method and an apparatus for processing photovoltaic data, and a system for managing photovoltaic data are provided, which relates to the field of data processing technology. The method includes: generating request identification metadata for starting a current data write transaction based on a current time when it is determined that a previous data write transaction is successfully executed; writing to-be-written photovoltaic data in the current data write transaction into a row storage format file in a row storage format based on the request identification metadata; merging the photovoltaic data in the row storage format file into a column storage format file in a column storage mode when it is determined that an amount of data written in the row storage format exceeds a target data amount threshold, where the row storage format file and the column storage format file are used for data query.