572 results about "Data stream processing" patented technology

The invention discloses an experimental calibration method based on multi-sensor signal fusion processing, particularly relates to the field of data analysis, and comprises the steps of a stepped confrontation disturbance protocol, data stream processing and real-time analysis, and multi-stage logic judgment and calibration execution. A stepped confrontation disturbance protocol is created for the first time, potential errors of the sensor are excited in real time and corrected in a closed loop mode under the condition that system operation is not interrupted, and a traditional shutdown calibration mode is thoroughly replaced; a multi-stage disturbance defense system is constructed, and composite interferences such as spectrum aliasing, overload distortion and signal-to-noise ratio attenuation caused by electromechanical thermal multi-physical field strong coupling are effectively suppressed; laser space calibration and dynamic rigidity modulation technologies are fused, and space drift errors caused by deformation of the mounting base and mechanical loosening are accurately diagnosed and compensated. Finally, long-term precision stability and data fusion reliability of the multi-source sensor under complex working conditions are achieved.

The invention relates to the technical field of real-time data processing, in particular to a distributed real-time data stream processing system and method, and the system comprises a state mapping modeling module, a load dynamic calculation module, a node task adjustment module and a resource scheduling execution module. According to the method, the dynamic identification and classification of the flow state are realized by analyzing the data flow change sequence and identifying the frequency and trend of state switching, the monitoring and understanding capability of real-time data is improved, the node resource pressure is accurately quantified, the predictability of load balancing is enhanced in the resource occupation and load range analysis, and the load balancing efficiency is improved. A scientific basis is provided for resource scheduling, task priority ranking and dynamic task adjustment are combined, the node computing resource utilization rate and the task processing response speed are optimized, flexible configuration of internal storage and computing resources is achieved, peak load pressure is relieved, and data processing efficiency and system stability are improved. And the adaptability and the reliability of processing large-scale data streams by the system are improved.

The invention discloses an urban greening system based on big data and an optimization method thereof, relates to the technical field of artificial intelligence, and solves the problems of difficulty in heterogeneous data integration, insufficient real-time analysis capability of dynamic data and low management efficiency in an existing urban greening system. Multi-source heterogeneous data are collected through an API interface and Internet of Things communication, and are cleaned, aligned, converted and fused through a distributed real-time data stream processing platform; image recognition and a machine learning algorithm are used for monitoring diseases and insect pests in real time, and a growth prediction model is trained in combination with an LSTM network, so that the quick response capability of the system to environmental changes is enhanced; according to the method, target weights are dynamically adjusted and optimized through the deep Q network, a multi-target optimization model is constructed in combination with mixed integer linear programming, context awareness and a genetic algorithm, intelligent optimization of irrigation, fertilization, pest control and greening layout is achieved, and the management efficiency and the resource utilization rate of an urban greening system are remarkably improved.

The invention discloses a medical history recognition method and system based on a large model and an expert strategy, and the method comprises the steps: collecting data in a medical network, and carrying out the cleaning and standardization processing of the data; extracting medical entities and relationships thereof from the text, and optimizing the rule set; the extracted entities and relations are fused into a knowledge graph, a multi-relation graph is constructed, and attributes are defined; storing the knowledge graph in a graph database, and receiving new data through a real-time data stream processing system; basic information and symptom description of a patient are collected, and key information is extracted by using a large language model; an inquiry path is dynamically adjusted, and specific symptoms are clearly and fuzzily answered or deeply discussed by questioning; and generating a structured electronic medical record by using the large language model. Through an intelligent pre-inquiry process, the system can collect detailed medical history information and generate a standardized electronic medical record before a patient officially sees a doctor, so that the information verification and supplement time when a doctor receives the doctor is greatly shortened.

The invention discloses an engineering project supervision real-time visual decision-making platform based on multi-source heterogeneous data fusion, and relates to the technical field of engineering supervision. The visual decision platform is in communication connection with a data acquisition module, a project risk feature module, a risk identification model construction module, a real-time data stream processing module, a risk early warning module and a decision support module. According to the invention, the risk in the engineering project is accurately identified in combination with historical multi-source heterogeneous data and project risk characteristics, and meanwhile, the real-time data stream processing module can analyze data in real time, identify potential risks and problems and generate early warning information, so that the accuracy and timeliness of risk identification are greatly improved, and the risk identification efficiency is improved. According to the invention, powerful decision support is provided for supervisors, and through the preset early warning level and the early warning signal, managers can rapidly understand the risk condition and take corresponding measures, thereby effectively reducing the project risk.

The invention discloses an inspection well overflow point forecasting and early warning method and system based on digital twinborn. The system comprises a city sensing module, a digital twinborn body module, an inspection well forecasting module and a flood risk early warning module. The method comprises the following steps: firstly, perfecting a city knowledge graph to construct a city perception module; constructing a digital twin city; constructing an inspection well forecasting module; and constructing a flood risk early warning module. Based on a digital twin platform, through an Internet of Things network and a data flow processing framework, an urban hydrological hydrodynamic model is coupled, parameters of the coupled model are dynamically calibrated, and an inspection well is finely simulated; the method comprehensively displays the multi-temporal-spatial-scale urban development state flood inundation condition, tracks and reflects the actual operation state of an urban drainage system, dynamically evaluates the risk level and carries out real-time early warning, improves the urban emergency management response efficiency, and provides support for smooth development of urban flood control and drainage work.

The invention provides a thermal power plant environment monitoring and early warning method and system based on the Internet of Things, and relates to the technical field of the Internet of Things, and the method comprises the steps: forming a preprocessing data stream for collected environment monitoring data through an edge calculation node; and carrying out real-time data stream processing by utilizing a distributed message queue, constructing a hierarchical distributed data storage structure, and carrying out homomorphic encryption and desensitization processing on data to form an encrypted data set. Based on an encrypted data set, a long-short term memory network, a random forest algorithm and a limit gradient boosting tree are adopted to construct a layered anomaly detection engine, environmental parameter spatial-temporal characteristics are extracted through a multi-head attention mechanism and residual connection, an early warning threshold value is iteratively calculated based on a Bayesian optimization algorithm, and an early warning level judgment standard is dynamically adjusted. And generating early warning information. And finally, through a three-dimensional visualization module and a knowledge inference engine based on a graph neural network and a deep reinforcement learning algorithm, visualization of early warning information and generation of a self-adaptive emergency processing strategy are realized respectively.

The invention relates to an Internet of Things equipment intelligent management platform based on edge computing, and aims to realize efficient and intelligent management of Internet of Things equipment through an edge computing technology. According to the platform, the edge nodes are deployed near the data source, so that local processing and storage of the data are realized, the data transmission delay is remarkably reduced, and the response speed of the system is improved. And the platform is provided with an intelligent management module, so that the equipment state can be monitored in real time, faults can be predicted, repair operation can be automatically executed, and the stability and reliability of the system are enhanced. And a self-adaptive data stream processing algorithm is adopted, and the data stream is dynamically adjusted according to a real-time network state and equipment requirements, so that the high efficiency of data processing is ensured. The secure transmission module protects the data transmission process through an end-to-end encryption technology, and prevents data leakage and tampering. The cloud-side collaborative resource scheduling system realizes optimal allocation of resources, and a user-defined rule engine provides high flexibility and personalized management strategies to meet different user requirements.

The invention discloses a rural water supply pipe network pressure monitoring analysis method and device based on digital twinning. The method comprises the following steps: collecting historical basic data of a rural prediction area; flow and pressure sensors are arranged at nodes of a rural water supply network to obtain monitoring data; uploading the data to a server through a data stream processing framework, and preprocessing the data to obtain preprocessed data; constructing a relational database management system to perform batch processing on the preprocessed data to obtain historical basic data and monitoring data after batch processing; constructing a rural pipe network water pressure prediction model; constructing a pipe explosion detection and water pressure control model; according to the physical data, constructing a rural water supply pipe network geometric model based on ArcGIS; and constructing a rural water supply pipe network system data model and realizing visualization of the operation state of the water supply pipe network. Rural water supply characteristics and water consumption characteristics are fully considered, intelligent management and control of the rural water supply system are achieved, and reliability and safety of rural water supply are effectively guaranteed.

The invention relates to the technical field of data processing, and discloses a real-time data stream processing method and device, electronic equipment and a storage medium. The method comprises the following steps: distributing a received to-be-processed data stream to a first distributed node through a preset load balancer; fragmenting the received distribution data stream at the first distributed node according to a preset fragmentation rule, then writing the fragmented data stream into a message queue, and performing parallel processing on fragmented data in the message queue through multiple threads; storing the intermediate state data in the fragmented data processing process, generating snapshot data according to the fragmented data, and synchronizing the snapshot data to the distributed storage system; acquiring load data of the first distributed node, and dynamically updating a data flow distribution strategy of the load balancer according to the load data through the resource use prediction model; and writing a processing result of the to-be-processed data stream into persistent storage through an asynchronous batch mechanism, and returning the processing result through a preset real-time communication protocol. The method improves the data stream processing efficiency and stability.

The invention discloses an intelligent data stream processing system based on Flink and an implementation method thereof, and belongs to the technical field of distributed streaming data processing, and the implementation method comprises the following steps: a data source access layer uniformly accesses multi-source data; the entity monitoring layer captures service entity change in a non-intrusive manner based on JPA, generates a standardized message body and asynchronously delivers the standardized message body to double channels; the unified event model layer maps the data into a standardized event and expands the standardized event; the Flink stream processing engine layer is used for complex event processing, state management and window calculation, and supports declarative pipeline definition and dynamic construction; the anomaly detection layer performs multi-dimensional anomaly detection; the distributed task scheduling layer schedules tasks according to a DAG model, and realizes load balancing through hotspot splitting and state transition; and the result storage and visualization layer adopts a cold and hot separation strategy to store data, and provides a dynamic visualization interface. The real-time performance, the consistency and the intelligent level of data processing are remarkably improved, and the development and maintenance cost is reduced.

The invention provides a financial data intelligent quality inspection method and system, and the method comprises the steps: S1, accessing a real-time transaction data flow through a dynamic rule engine, and enabling the dynamic rule engine to dynamically adjust the rule weight through a Bayesian network and reinforcement learning hybrid model; s2, calling a multi-modal LLM verification framework, performing joint semantic analysis on the text, the image and the time series data, and generating a risk early warning signal; s3, identifying a cross-entity risk path based on the financial knowledge graph, and converting the identified risk path into a structured risk report; and S4, a closed loop iteration system is formed according to the weight of the early warning feedback optimization rule. According to the method, full-life-cycle quality management and control of financial transactions can be realized through technical collaboration of real-time data stream processing, multi-dimensional semantic verification and cross-entity risk tracking.

The invention discloses a full-link monitoring method and system for state application in an edge distribution scene, and the method comprises the steps: S1, deploying a monitoring agent at an edge calculation node, and collecting the state data and performance indexes of edge equipment; s2, sending the collected data to a central monitoring system in real time through an efficient data transmission protocol; s3, storing and managing synchronous data between the edge nodes and the center node by using a distributed database; s4, preprocessing the collected data by applying a real-time data stream processing technology; s5, performing anomaly detection and mode recognition on the preprocessed data, and generating intelligent early warning; s6, adopting a full-link tracking tool to monitor and record a full-link path requested in the system, and identifying and analyzing a performance bottleneck and a fault point; and S7, optimizing a data transmission process by using a message queue system, and implementing a data encryption and access control strategy. The method has the advantages of low delay, high data security and high system response speed.

The invention provides a risk early warning method and platform for power distribution network protection area monitoring, and relates to the technical field of power systems, and the method comprises the steps: traversing the safety demand information of a power distribution network protection area, and carrying out the operation risk recognition; monitoring the protection area of the power distribution network in real time; constructing a risk prediction model, and obtaining a risk prediction data set; introducing a risk rule to carry out risk assessment; and triggering an early warning mechanism according to a risk assessment result, generating early warning feedback information, and performing intelligent early warning on risk monitoring of the power distribution network protection area. The technical problem that potential risks cannot be found in time due to the fact that an existing power distribution network protection area monitoring method lacks real-time performance and comprehensiveness in the prior art can be solved, through real-time monitoring and data flow processing, risk rules are introduced, risk assessment is carried out, intelligent early warning is triggered, potential risk factors are comprehensively recognized, and the safety of the power distribution network protection area is improved. And the risk identification accuracy is improved.

The invention provides an urban space intelligent processing method based on multi-modal fusion, and the method comprises the steps: taking multi-modal data as input, and constructing a unified data stream processing and feature alignment mechanism; a physical space is used as a core framework, and the multi-modal data is converted into a space behavior graph with space-time position semantics; constructing an entity attribute-relation type-influence weight ternary interaction model on the basis of an interaction layer of the spatial behavior map, and analyzing a human, object and environment ternary interaction relation in the city and the park based on the ternary interaction model; designing a space intelligent engine with time sequence modeling and dynamic prediction capabilities; and constructing a task processing system. According to the invention, through deep combination of multi-modal fusion and a space intelligent technology, full-link upgrading of urban space from data perception to intelligent decision is realized, and powerful technical support is provided for fine management, efficient operation and safety guarantee of complex space scenes.

The invention discloses an integrated learning classification method and system with a concept drift detection function, and relates to the technical field of machine learning and data stream processing, and the method comprises the steps: obtaining high-dimensional data of an extensible sliding window, carrying out the dimension reduction of the high-dimensional data of the extensible sliding window through PCA, and carrying out the extraction to obtain key feature data; performing real-time monitoring on the key feature data based on an adaptive window early warning detection method, automatically expanding an adaptive window during drifting early warning, and obtaining original data of the adaptive window when concept drifting occurs; and inputting the original data of the adaptive window when the concept drift occurs into a pre-established learning model based on an integrated Hough tree to obtain a trained learning model based on the integrated Hough tree, and performing real-time classification of data streams based on the trained learning model based on the integrated Hough tree. The accuracy of stream data classification can be improved, and the adaptive ability of the model to concept drift is enhanced.

The invention relates to the technical field of enterprise project management, and discloses an enterprise project management method and system based on multi-source heterogeneous data integration, and the method comprises the steps: receiving heterogeneous data from a plurality of data sources; classifying the heterogeneous data into different-speed data streams of a high-speed stream, a medium-speed stream and a low-speed stream according to the updating frequency; allocating a logic timestamp for the different-speed data stream; the different-speed data flow is processed based on the logic timestamp, and the logic consistency of data processing is guaranteed; converting, aggregating and analyzing the processed data according to a preset data stream processing logic; the analysis result is applied to enterprise project management decision support; real-time performance, consistency and integrity of data in enterprise project management are achieved, efficiency and decision quality of enterprise project management are improved, and the problems of data islands, data inconsistency and data processing lag in traditional enterprise project management are solved.

A method for managing a data stream processing pipeline includes: monitoring, by an orchestrator, data stream ingestion of a streaming storage system (SSS) to obtain data stream metrics; analyzing, by the orchestrator, the data stream metrics based on a user-defined scaling policy; making, based on the analyzing and by the orchestrator, a first determination that task manager scaling is required; making, by the orchestrator, a second determination that the data stream ingestion is increased, in which the second determination indicates an increase in a number of parallel stream segments associated with a data stream, in which a segment store hosted by the SSS manages the parallel stream segments; and to increase a stream processing system's compute capability and by the orchestrator, initiating scaling of a number of the plurality of task managers to support the increase in the number of the parallel stream segments.

The invention relates to the technical field of building construction monitoring, and discloses a construction progress real-time monitoring system based on BIM and Internet of Things, and the system comprises an Internet of Things data collection module which obtains real-time construction data through a sensor network and divides the real-time construction data into a spatial position flow and an equipment state flow; the BIM data integration module processes the two types of data streams to generate a first progress index and a second progress index; the progress deviation analysis module is combined with a preset progress reference model, and the progress indexes are fused and then mapped into a progress deviation value; the response decision module calls a preset scheme based on the deviation value to generate a target response strategy; the central processing unit is responsible for data transmission and logic verification. In addition, the system is further provided with a construction linkage module, an instruction output module and a progress storage module which are respectively used for optimizing resource scheduling, sending control commands and archiving data. According to the system, real-time monitoring, deviation analysis and dynamic adjustment of the construction progress are realized, and the intelligence and accuracy of construction management are improved.

The invention discloses a fault analysis method and system applied to a 5G base station transmission link, and the method comprises the following steps: obtaining alarm information of the 5G base station transmission link through integrating a real-time data stream processing mechanism and an incremental learning model; the alarm information is preprocessed; based on a time dimension, a topology dimension and a text dimension, converting the pre-processed alarm information into multi-dimensional vector representation; grouping aggregation is carried out by combining a DBSCAN algorithm; through an Incident root cause analysis algorithm, performing deep suspected root cause analysis on the grouped and aggregated alarm information, and determining suspected root cause alarms; and performing fault diagnosis through the knowledge graph and the diagnosis rule, generating a repair process and recording the whole process. According to the method, the accuracy and speed of fault positioning can be remarkably improved, the intelligence and automation level of fault analysis is enhanced, and meanwhile, the real-time performance and flexibility of fault processing are ensured, so that the operation and maintenance management of a 5G network is optimized, and a solid guarantee is provided for the stability and reliability of communication service.

The invention relates to the technical field of storage systems, in particular to an electrolytic cell test data cloud storage method, which comprises the following steps: performing timestamp marking and partition size calculation on electrolytic cell test data, aggregating the data into partitions with fixed sizes according to timestamps, calculating a data density value and a write-in rate value of each partition, obtaining a partition data set, and storing the partition data set into a database; and comparing the write-in rate value in the partition data set with a preset partition threshold value. According to the method, through a dynamic partition management and multi-path data flow processing mechanism, in the storage process of the test data of the electrolytic cell, the data is subjected to partition aggregation through timestamp marking, a data density value and a write-in rate value reflect the dynamic change of the partition, and the size of the partition is adjusted in real time to cope with the fluctuation of a data write-in load. Multi-channel data stream distribution and merge sorting improve concurrent processing performance and eliminate single-channel bottleneck restriction. A double-buffer-area framework is matched with data access frequency analysis to support intelligent migration of cold and hot data, and IO resource configuration is optimized.

The invention provides a streaming data processing method, system and device based on Flink and a medium, and belongs to the technical field of data processing.The method comprises the following steps that data streams are received from different data sources, a connection pool is established according to the types of the data sources, and data in the data streams are processed; dividing the data stream into a plurality of windows for processing based on event time, storing the processing state of the data stream in an increment form, and performing partition management; task faults in the data processing process are detected, task restarting and load balancing are automatically carried out, meanwhile, data stream processing delay is monitored, and computing resources are dynamically adjusted when the delay exceeds the standard; and monitoring the state of the data stream in real time, carrying out visual display, and giving an alarm when the data stream is abnormal. According to the method, mass real-time data streams from different data sources are received and preprocessed, so that efficient and stable processing is realized; and the stability and the real-time performance of the system are guaranteed by means of mechanisms such as fault detection and load balancing.

The invention discloses a shared cache system applied to a real-time network terminal chip. The shared cache system comprises queue management, enqueue scheduling control, dequeue scheduling control, write bus control, read bus control, a data cache, a register module, an RXFIFO module and a TXFIFO module. According to the cache allocation strategy, the variable-length data is stored by using the fixed-length unit. The user can customize and configure the private space, the residual space is the shared space, and the storage utilization rate is high. The shared cache is mainly used for storing data packets sent by an external application layer through a host interface. And storing the received data packet, extracting frame information according to a network flow protocol, and directly discarding frames which do not meet the protocol. The shared cache is provided with a dynamic cache structure, the state of each queue can be updated in real time, frame loss is prevented, and the data stream processing capacity is improved. And meanwhile, complex hybrid scheduling of a real-time network is met. The whole shared cache is subjected to anti-radiation reinforcement, the scene requirement of space navigation application is met, and high reliability is achieved.

The invention provides a communication network optimization system and method based on spatio-temporal data fusion, and relates to the technical field of real-time data stream processing, and the method comprises the steps: carrying out the base station coverage analysis of a predicted driving track, and outputting K potential blind area sequences; performing driving characteristic analysis on the K potential blind area sequences, and outputting K channel switching request sequences to construct a communication network optimization library; and according to the real-time spatio-temporal data of the target vehicle, scheduling a switching instruction from the communication network optimization library to dynamically update the communication link of the target vehicle. The technical problems that in the prior art, due to the fact that data processing response of communication link switching analysis in a high-speed moving scene is delayed, the communication network coverage stability and reliability in the high-speed moving scene are poor, and the communication interruption risk is high are solved. The technical effects of improving the data analysis efficiency of communication link dynamic switching in a high-speed mobile communication scene, enhancing the reliability and coverage performance of a communication network and effectively reducing the risk of communication interruption are achieved.

The invention discloses a monitoring video information intelligent processing system based on cloud computing, which relates to the technical field of monitoring video information processing and comprises a monitoring information processing platform. The monitoring information processing platform is in communication connection with a video data acquisition module, a data transmission module, a cloud data storage module, a data stream processing module, a target analysis module, an anomaly detection module and an intelligent alarm management module. Through training the deep neural network model and learning the features of normal and abnormal behaviors, the method achieves the precise recognition and tracking of a target object in a video, detects a dynamic target in the video, extracts the motion and appearance features of the dynamic target, recognizes the abnormal behaviors through behavior analysis, calculates an abnormal score, and generates alarm information in time. The accuracy of the monitoring system is improved, the situations of false alarm and missing alarm are reduced, and safety monitoring is more efficient and reliable.

The invention discloses a big data integration system based on artificial intelligence, and particularly relates to the technical field of data integration, by deploying an edge data analysis model at a big data acquisition end and combining cloud computing resources for collaborative optimization, efficiency abnormity in the data integration process is timely warned, computing resource allocation is dynamically adjusted, and the efficiency of data integration is improved. A data processing strategy is optimized, so that delay and resource waste in an integration process are effectively reduced; a data stream processing path is optimized cooperatively through edge end preprocessing and cloud deep analysis; through combined application of key node monitoring, key node performance association rule mining, adaptive performance bottleneck diagnosis and dynamic scheduling, real-time optimization and closed-loop control of a big data integration process are realized, and abnormity of an actual value of integration efficiency is avoided. The problems of low integration efficiency, unbalanced resources, integration efficiency prediction lag and insufficient resource optimization are solved.

The invention particularly relates to a method for realizing data watermark processing based on NiFi. According to the method for realizing data watermark processing based on NiFi, a data source connector is configured in a NiFi server, and a to-be-processed data stream is received in real time; a watermark embedding strategy is formulated, a watermark embedding algorithm is selected and optimized, watermark information is embedded into a data stream, and the data stream is transmitted to a target system or a storage medium by adopting an encryption means; extracting watermark information from the data stream with the watermark, verifying the extracted watermark information, and generating a verification result report; and a watermark verification result is fed back to a system administrator or a related user, and a watermark embedding and extracting report is generated. According to the method for realizing data watermark processing based on NiFi, the efficiency and the flexibility of data watermark processing are improved, the security and the integrity of the data are guaranteed, and powerful support is provided for secure transmission and copyright protection of the data and application of a data stream processing technology in more fields.

The invention relates to the technical field of data processing, in particular to a data stream processing system and method in a big data environment. The method comprises the following steps: collecting data in real time to obtain an original multi-source heterogeneous data stream; introducing a data adaptation mechanism to carry out standardization processing on the original multi-source heterogeneous data stream to obtain a standardized data stream; performing intelligent scheduling on the standardized data stream to obtain a preliminary scheduling strategy; based on feedback data of a data stream processing system, parallel real-time state feedback and edge real-time state feedback are obtained, a preliminary scheduling strategy is optimized, and based on the optimized scheduling strategy, normalized data streams are divided into primary normalized data streams and secondary normalized data streams. And processing the primary normalized data stream and the secondary normalized data stream. The problems that when a traditional data processing system processes data streams in a big data environment, multi-source heterogeneous data cannot be accurately adapted, and distribution and scheduling are unreasonable during processing are solved.

The invention relates to the technical field of distributed data stream processing, in particular to a distributed data stream asynchronous snapshot method and system based on flink, and the method comprises the following steps: designing and executing a snapshot algorithm which is completed in limited time under the condition that all processes are in an active state, thereby meeting the termination requirement; when the snapshots are executed, the causal sequence of the data streams is maintained, and it is ensured that the delivery sequence recorded in the snapshots is consistent with the sequence in actual task execution, so that the feasibility requirement is met; the flink-based distributed data stream asynchronous snapshot method and system provided by the invention have the beneficial effects that an asynchronous obstacle snapshot algorithm is introduced, and the flink-based distributed data stream asynchronous snapshot method and system are a new snapshot technology for realizing good throughput. The asynchronous obstacle snapshot algorithm is the first algorithm which considers the minimum state required by the loop-free execution topology. In addition, an asynchronous barrier snapshot algorithm is extended to accommodate a loop execution graph by storing only records that need to be reprocessed when recovery is required.