3 results about "Big data architecture" patented technology

This invention discloses a multimodal data fusion coding method for constructing a unified underlying language for national big data, belonging to the fields of national big data architecture, multimodal governance, and domestically produced unified coding. Based on the mathematical fundamental feature theory and the STE domestic coding system, coupled with the GZ-BigData-RISC-V domestic dedicated chip and a multimodal AI fusion model, it achieves normalized access to multimodal data, unified extraction of fundamental features, unified underlying language coding, lossless fusion, and secure storage. Quantitative thresholds are set for cross-modal fusion efficiency ≥85%, data consistency ≥99.5%, coding latency ≤1ms, and lossless fusion rate 100%, completing the unified coding and semantic association of text, image, audio, video, time-series, and geospatial data. This method breaks down data heterogeneity barriers, constructs the only nationally interoperable unified underlying language for big data, and is 100% domestically produced and controllable throughout the entire process, supporting the construction of a national integrated big data center and promoting the market-oriented circulation of data elements.

The application provides a lightweight real-time data synchronization and processing method and system for a virtual wafer factory, the method comprising: performing full-amount extraction and synchronization of historical data of a source-end database through a DataX tool; performing real-time monitoring and capturing of incremental change data of the source-end database through a CDC tool, and sending the incremental change data to a Kafka message queue in the form of a message; consuming the incremental change data from the Kafka message queue through a NiFi data flow tool; writing data into a TimeScaleDB time series database; using an aggregation engine built in the TimeScaleDB to perform continuous aggregation operation on the written data, generate and maintain a materialized view, and the materialized view is used for direct query and calling by a business system, so as to realize storage-computing fusion of data storage and aggregation calculation. The application can solve the problems of long existing big data ETL architecture tool chain, large resource consumption and high operation and maintenance cost, and realize lightweight operation of data cleaning and aggregation calculation.

The application discloses a memory resident data management method based on a big data architecture, relates to the technical field of data management, and realizes dynamic management of cache data by establishing metadata for cache data blocks and maintaining a global cache list in the memory of a big data computing framework in combination with a timing cleaning and a forced cleaning mechanism based on memory occupancy; when the memory occupancy is lower than a preset safety threshold, a judgment index is further calculated and compared with a judgment threshold to judge whether a safety state is truly entered or an alarm is triggered. The method can dynamically adapt system load when task quantity suddenly increases or memory pressure fluctuates, avoids resource idling or memory overflow, and improves cache retention efficiency and reduces the risk of mistaken deletion through a multi-level cleaning strategy. Meanwhile, the intelligent alarm mechanism is introduced to guarantee the operation stability of the system in a high-pressure environment and improve the safety and intelligent level of overall memory management.