444 results about "Data stream processing" patented technology

The invention discloses an experimental calibration method based on multi-sensor signal fusion processing, particularly relates to the field of data analysis, and comprises the steps of a stepped confrontation disturbance protocol, data stream processing and real-time analysis, and multi-stage logic judgment and calibration execution. A stepped confrontation disturbance protocol is created for the first time, potential errors of the sensor are excited in real time and corrected in a closed loop mode under the condition that system operation is not interrupted, and a traditional shutdown calibration mode is thoroughly replaced; a multi-stage disturbance defense system is constructed, and composite interferences such as spectrum aliasing, overload distortion and signal-to-noise ratio attenuation caused by electromechanical thermal multi-physical field strong coupling are effectively suppressed; laser space calibration and dynamic rigidity modulation technologies are fused, and space drift errors caused by deformation of the mounting base and mechanical loosening are accurately diagnosed and compensated. Finally, long-term precision stability and data fusion reliability of the multi-source sensor under complex working conditions are achieved.

The invention discloses an urban greening system based on big data and an optimization method thereof, relates to the technical field of artificial intelligence, and solves the problems of difficulty in heterogeneous data integration, insufficient real-time analysis capability of dynamic data and low management efficiency in an existing urban greening system. Multi-source heterogeneous data are collected through an API interface and Internet of Things communication, and are cleaned, aligned, converted and fused through a distributed real-time data stream processing platform; image recognition and a machine learning algorithm are used for monitoring diseases and insect pests in real time, and a growth prediction model is trained in combination with an LSTM network, so that the quick response capability of the system to environmental changes is enhanced; according to the method, target weights are dynamically adjusted and optimized through the deep Q network, a multi-target optimization model is constructed in combination with mixed integer linear programming, context awareness and a genetic algorithm, intelligent optimization of irrigation, fertilization, pest control and greening layout is achieved, and the management efficiency and the resource utilization rate of an urban greening system are remarkably improved.

The invention discloses a medical history recognition method and system based on a large model and an expert strategy, and the method comprises the steps: collecting data in a medical network, and carrying out the cleaning and standardization processing of the data; extracting medical entities and relationships thereof from the text, and optimizing the rule set; the extracted entities and relations are fused into a knowledge graph, a multi-relation graph is constructed, and attributes are defined; storing the knowledge graph in a graph database, and receiving new data through a real-time data stream processing system; basic information and symptom description of a patient are collected, and key information is extracted by using a large language model; an inquiry path is dynamically adjusted, and specific symptoms are clearly and fuzzily answered or deeply discussed by questioning; and generating a structured electronic medical record by using the large language model. Through an intelligent pre-inquiry process, the system can collect detailed medical history information and generate a standardized electronic medical record before a patient officially sees a doctor, so that the information verification and supplement time when a doctor receives the doctor is greatly shortened.

The invention discloses an engineering project supervision real-time visual decision-making platform based on multi-source heterogeneous data fusion, and relates to the technical field of engineering supervision. The visual decision platform is in communication connection with a data acquisition module, a project risk feature module, a risk identification model construction module, a real-time data stream processing module, a risk early warning module and a decision support module. According to the invention, the risk in the engineering project is accurately identified in combination with historical multi-source heterogeneous data and project risk characteristics, and meanwhile, the real-time data stream processing module can analyze data in real time, identify potential risks and problems and generate early warning information, so that the accuracy and timeliness of risk identification are greatly improved, and the risk identification efficiency is improved. According to the invention, powerful decision support is provided for supervisors, and through the preset early warning level and the early warning signal, managers can rapidly understand the risk condition and take corresponding measures, thereby effectively reducing the project risk.

The invention provides a thermal power plant environment monitoring and early warning method and system based on the Internet of Things, and relates to the technical field of the Internet of Things, and the method comprises the steps: forming a preprocessing data stream for collected environment monitoring data through an edge calculation node; and carrying out real-time data stream processing by utilizing a distributed message queue, constructing a hierarchical distributed data storage structure, and carrying out homomorphic encryption and desensitization processing on data to form an encrypted data set. Based on an encrypted data set, a long-short term memory network, a random forest algorithm and a limit gradient boosting tree are adopted to construct a layered anomaly detection engine, environmental parameter spatial-temporal characteristics are extracted through a multi-head attention mechanism and residual connection, an early warning threshold value is iteratively calculated based on a Bayesian optimization algorithm, and an early warning level judgment standard is dynamically adjusted. And generating early warning information. And finally, through a three-dimensional visualization module and a knowledge inference engine based on a graph neural network and a deep reinforcement learning algorithm, visualization of early warning information and generation of a self-adaptive emergency processing strategy are realized respectively.

The invention relates to an Internet of Things equipment intelligent management platform based on edge computing, and aims to realize efficient and intelligent management of Internet of Things equipment through an edge computing technology. According to the platform, the edge nodes are deployed near the data source, so that local processing and storage of the data are realized, the data transmission delay is remarkably reduced, and the response speed of the system is improved. And the platform is provided with an intelligent management module, so that the equipment state can be monitored in real time, faults can be predicted, repair operation can be automatically executed, and the stability and reliability of the system are enhanced. And a self-adaptive data stream processing algorithm is adopted, and the data stream is dynamically adjusted according to a real-time network state and equipment requirements, so that the high efficiency of data processing is ensured. The secure transmission module protects the data transmission process through an end-to-end encryption technology, and prevents data leakage and tampering. The cloud-side collaborative resource scheduling system realizes optimal allocation of resources, and a user-defined rule engine provides high flexibility and personalized management strategies to meet different user requirements.

The invention discloses a rural water supply pipe network pressure monitoring analysis method and device based on digital twinning. The method comprises the following steps: collecting historical basic data of a rural prediction area; flow and pressure sensors are arranged at nodes of a rural water supply network to obtain monitoring data; uploading the data to a server through a data stream processing framework, and preprocessing the data to obtain preprocessed data; constructing a relational database management system to perform batch processing on the preprocessed data to obtain historical basic data and monitoring data after batch processing; constructing a rural pipe network water pressure prediction model; constructing a pipe explosion detection and water pressure control model; according to the physical data, constructing a rural water supply pipe network geometric model based on ArcGIS; and constructing a rural water supply pipe network system data model and realizing visualization of the operation state of the water supply pipe network. Rural water supply characteristics and water consumption characteristics are fully considered, intelligent management and control of the rural water supply system are achieved, and reliability and safety of rural water supply are effectively guaranteed.

The invention relates to the technical field of data processing, and discloses a real-time data stream processing method and device, electronic equipment and a storage medium. The method comprises the following steps: distributing a received to-be-processed data stream to a first distributed node through a preset load balancer; fragmenting the received distribution data stream at the first distributed node according to a preset fragmentation rule, then writing the fragmented data stream into a message queue, and performing parallel processing on fragmented data in the message queue through multiple threads; storing the intermediate state data in the fragmented data processing process, generating snapshot data according to the fragmented data, and synchronizing the snapshot data to the distributed storage system; acquiring load data of the first distributed node, and dynamically updating a data flow distribution strategy of the load balancer according to the load data through the resource use prediction model; and writing a processing result of the to-be-processed data stream into persistent storage through an asynchronous batch mechanism, and returning the processing result through a preset real-time communication protocol. The method improves the data stream processing efficiency and stability.

The invention discloses an intelligent data stream processing system based on Flink and an implementation method thereof, and belongs to the technical field of distributed streaming data processing, and the implementation method comprises the following steps: a data source access layer uniformly accesses multi-source data; the entity monitoring layer captures service entity change in a non-intrusive manner based on JPA, generates a standardized message body and asynchronously delivers the standardized message body to double channels; the unified event model layer maps the data into a standardized event and expands the standardized event; the Flink stream processing engine layer is used for complex event processing, state management and window calculation, and supports declarative pipeline definition and dynamic construction; the anomaly detection layer performs multi-dimensional anomaly detection; the distributed task scheduling layer schedules tasks according to a DAG model, and realizes load balancing through hotspot splitting and state transition; and the result storage and visualization layer adopts a cold and hot separation strategy to store data, and provides a dynamic visualization interface. The real-time performance, the consistency and the intelligent level of data processing are remarkably improved, and the development and maintenance cost is reduced.

The invention provides a financial data intelligent quality inspection method and system, and the method comprises the steps: S1, accessing a real-time transaction data flow through a dynamic rule engine, and enabling the dynamic rule engine to dynamically adjust the rule weight through a Bayesian network and reinforcement learning hybrid model; s2, calling a multi-modal LLM verification framework, performing joint semantic analysis on the text, the image and the time series data, and generating a risk early warning signal; s3, identifying a cross-entity risk path based on the financial knowledge graph, and converting the identified risk path into a structured risk report; and S4, a closed loop iteration system is formed according to the weight of the early warning feedback optimization rule. According to the method, full-life-cycle quality management and control of financial transactions can be realized through technical collaboration of real-time data stream processing, multi-dimensional semantic verification and cross-entity risk tracking.

The invention provides a risk early warning method and platform for power distribution network protection area monitoring, and relates to the technical field of power systems, and the method comprises the steps: traversing the safety demand information of a power distribution network protection area, and carrying out the operation risk recognition; monitoring the protection area of the power distribution network in real time; constructing a risk prediction model, and obtaining a risk prediction data set; introducing a risk rule to carry out risk assessment; and triggering an early warning mechanism according to a risk assessment result, generating early warning feedback information, and performing intelligent early warning on risk monitoring of the power distribution network protection area. The technical problem that potential risks cannot be found in time due to the fact that an existing power distribution network protection area monitoring method lacks real-time performance and comprehensiveness in the prior art can be solved, through real-time monitoring and data flow processing, risk rules are introduced, risk assessment is carried out, intelligent early warning is triggered, potential risk factors are comprehensively recognized, and the safety of the power distribution network protection area is improved. And the risk identification accuracy is improved.

The invention provides an urban space intelligent processing method based on multi-modal fusion, and the method comprises the steps: taking multi-modal data as input, and constructing a unified data stream processing and feature alignment mechanism; a physical space is used as a core framework, and the multi-modal data is converted into a space behavior graph with space-time position semantics; constructing an entity attribute-relation type-influence weight ternary interaction model on the basis of an interaction layer of the spatial behavior map, and analyzing a human, object and environment ternary interaction relation in the city and the park based on the ternary interaction model; designing a space intelligent engine with time sequence modeling and dynamic prediction capabilities; and constructing a task processing system. According to the invention, through deep combination of multi-modal fusion and a space intelligent technology, full-link upgrading of urban space from data perception to intelligent decision is realized, and powerful technical support is provided for fine management, efficient operation and safety guarantee of complex space scenes.

The invention discloses an integrated learning classification method and system with a concept drift detection function, and relates to the technical field of machine learning and data stream processing, and the method comprises the steps: obtaining high-dimensional data of an extensible sliding window, carrying out the dimension reduction of the high-dimensional data of the extensible sliding window through PCA, and carrying out the extraction to obtain key feature data; performing real-time monitoring on the key feature data based on an adaptive window early warning detection method, automatically expanding an adaptive window during drifting early warning, and obtaining original data of the adaptive window when concept drifting occurs; and inputting the original data of the adaptive window when the concept drift occurs into a pre-established learning model based on an integrated Hough tree to obtain a trained learning model based on the integrated Hough tree, and performing real-time classification of data streams based on the trained learning model based on the integrated Hough tree. The accuracy of stream data classification can be improved, and the adaptive ability of the model to concept drift is enhanced.

The invention relates to the technical field of enterprise project management, and discloses an enterprise project management method and system based on multi-source heterogeneous data integration, and the method comprises the steps: receiving heterogeneous data from a plurality of data sources; classifying the heterogeneous data into different-speed data streams of a high-speed stream, a medium-speed stream and a low-speed stream according to the updating frequency; allocating a logic timestamp for the different-speed data stream; the different-speed data flow is processed based on the logic timestamp, and the logic consistency of data processing is guaranteed; converting, aggregating and analyzing the processed data according to a preset data stream processing logic; the analysis result is applied to enterprise project management decision support; real-time performance, consistency and integrity of data in enterprise project management are achieved, efficiency and decision quality of enterprise project management are improved, and the problems of data islands, data inconsistency and data processing lag in traditional enterprise project management are solved.

A method for managing a data stream processing pipeline includes: monitoring, by an orchestrator, data stream ingestion of a streaming storage system (SSS) to obtain data stream metrics; analyzing, by the orchestrator, the data stream metrics based on a user-defined scaling policy; making, based on the analyzing and by the orchestrator, a first determination that task manager scaling is required; making, by the orchestrator, a second determination that the data stream ingestion is increased, in which the second determination indicates an increase in a number of parallel stream segments associated with a data stream, in which a segment store hosted by the SSS manages the parallel stream segments; and to increase a stream processing system's compute capability and by the orchestrator, initiating scaling of a number of the plurality of task managers to support the increase in the number of the parallel stream segments.

The invention relates to the technical field of building construction monitoring, and discloses a construction progress real-time monitoring system based on BIM and Internet of Things, and the system comprises an Internet of Things data collection module which obtains real-time construction data through a sensor network and divides the real-time construction data into a spatial position flow and an equipment state flow; the BIM data integration module processes the two types of data streams to generate a first progress index and a second progress index; the progress deviation analysis module is combined with a preset progress reference model, and the progress indexes are fused and then mapped into a progress deviation value; the response decision module calls a preset scheme based on the deviation value to generate a target response strategy; the central processing unit is responsible for data transmission and logic verification. In addition, the system is further provided with a construction linkage module, an instruction output module and a progress storage module which are respectively used for optimizing resource scheduling, sending control commands and archiving data. According to the system, real-time monitoring, deviation analysis and dynamic adjustment of the construction progress are realized, and the intelligence and accuracy of construction management are improved.

The invention discloses a fault analysis method and system applied to a 5G base station transmission link, and the method comprises the following steps: obtaining alarm information of the 5G base station transmission link through integrating a real-time data stream processing mechanism and an incremental learning model; the alarm information is preprocessed; based on a time dimension, a topology dimension and a text dimension, converting the pre-processed alarm information into multi-dimensional vector representation; grouping aggregation is carried out by combining a DBSCAN algorithm; through an Incident root cause analysis algorithm, performing deep suspected root cause analysis on the grouped and aggregated alarm information, and determining suspected root cause alarms; and performing fault diagnosis through the knowledge graph and the diagnosis rule, generating a repair process and recording the whole process. According to the method, the accuracy and speed of fault positioning can be remarkably improved, the intelligence and automation level of fault analysis is enhanced, and meanwhile, the real-time performance and flexibility of fault processing are ensured, so that the operation and maintenance management of a 5G network is optimized, and a solid guarantee is provided for the stability and reliability of communication service.

The invention relates to the technical field of storage systems, in particular to an electrolytic cell test data cloud storage method, which comprises the following steps: performing timestamp marking and partition size calculation on electrolytic cell test data, aggregating the data into partitions with fixed sizes according to timestamps, calculating a data density value and a write-in rate value of each partition, obtaining a partition data set, and storing the partition data set into a database; and comparing the write-in rate value in the partition data set with a preset partition threshold value. According to the method, through a dynamic partition management and multi-path data flow processing mechanism, in the storage process of the test data of the electrolytic cell, the data is subjected to partition aggregation through timestamp marking, a data density value and a write-in rate value reflect the dynamic change of the partition, and the size of the partition is adjusted in real time to cope with the fluctuation of a data write-in load. Multi-channel data stream distribution and merge sorting improve concurrent processing performance and eliminate single-channel bottleneck restriction. A double-buffer-area framework is matched with data access frequency analysis to support intelligent migration of cold and hot data, and IO resource configuration is optimized.

The invention provides a streaming data processing method, system and device based on Flink and a medium, and belongs to the technical field of data processing.The method comprises the following steps that data streams are received from different data sources, a connection pool is established according to the types of the data sources, and data in the data streams are processed; dividing the data stream into a plurality of windows for processing based on event time, storing the processing state of the data stream in an increment form, and performing partition management; task faults in the data processing process are detected, task restarting and load balancing are automatically carried out, meanwhile, data stream processing delay is monitored, and computing resources are dynamically adjusted when the delay exceeds the standard; and monitoring the state of the data stream in real time, carrying out visual display, and giving an alarm when the data stream is abnormal. According to the method, mass real-time data streams from different data sources are received and preprocessed, so that efficient and stable processing is realized; and the stability and the real-time performance of the system are guaranteed by means of mechanisms such as fault detection and load balancing.

The invention discloses a shared cache system applied to a real-time network terminal chip. The shared cache system comprises queue management, enqueue scheduling control, dequeue scheduling control, write bus control, read bus control, a data cache, a register module, an RXFIFO module and a TXFIFO module. According to the cache allocation strategy, the variable-length data is stored by using the fixed-length unit. The user can customize and configure the private space, the residual space is the shared space, and the storage utilization rate is high. The shared cache is mainly used for storing data packets sent by an external application layer through a host interface. And storing the received data packet, extracting frame information according to a network flow protocol, and directly discarding frames which do not meet the protocol. The shared cache is provided with a dynamic cache structure, the state of each queue can be updated in real time, frame loss is prevented, and the data stream processing capacity is improved. And meanwhile, complex hybrid scheduling of a real-time network is met. The whole shared cache is subjected to anti-radiation reinforcement, the scene requirement of space navigation application is met, and high reliability is achieved.

The invention provides a communication network optimization system and method based on spatio-temporal data fusion, and relates to the technical field of real-time data stream processing, and the method comprises the steps: carrying out the base station coverage analysis of a predicted driving track, and outputting K potential blind area sequences; performing driving characteristic analysis on the K potential blind area sequences, and outputting K channel switching request sequences to construct a communication network optimization library; and according to the real-time spatio-temporal data of the target vehicle, scheduling a switching instruction from the communication network optimization library to dynamically update the communication link of the target vehicle. The technical problems that in the prior art, due to the fact that data processing response of communication link switching analysis in a high-speed moving scene is delayed, the communication network coverage stability and reliability in the high-speed moving scene are poor, and the communication interruption risk is high are solved. The technical effects of improving the data analysis efficiency of communication link dynamic switching in a high-speed mobile communication scene, enhancing the reliability and coverage performance of a communication network and effectively reducing the risk of communication interruption are achieved.

The invention discloses a monitoring video information intelligent processing system based on cloud computing, which relates to the technical field of monitoring video information processing and comprises a monitoring information processing platform. The monitoring information processing platform is in communication connection with a video data acquisition module, a data transmission module, a cloud data storage module, a data stream processing module, a target analysis module, an anomaly detection module and an intelligent alarm management module. Through training the deep neural network model and learning the features of normal and abnormal behaviors, the method achieves the precise recognition and tracking of a target object in a video, detects a dynamic target in the video, extracts the motion and appearance features of the dynamic target, recognizes the abnormal behaviors through behavior analysis, calculates an abnormal score, and generates alarm information in time. The accuracy of the monitoring system is improved, the situations of false alarm and missing alarm are reduced, and safety monitoring is more efficient and reliable.

The invention discloses a big data integration system based on artificial intelligence, and particularly relates to the technical field of data integration, by deploying an edge data analysis model at a big data acquisition end and combining cloud computing resources for collaborative optimization, efficiency abnormity in the data integration process is timely warned, computing resource allocation is dynamically adjusted, and the efficiency of data integration is improved. A data processing strategy is optimized, so that delay and resource waste in an integration process are effectively reduced; a data stream processing path is optimized cooperatively through edge end preprocessing and cloud deep analysis; through combined application of key node monitoring, key node performance association rule mining, adaptive performance bottleneck diagnosis and dynamic scheduling, real-time optimization and closed-loop control of a big data integration process are realized, and abnormity of an actual value of integration efficiency is avoided. The problems of low integration efficiency, unbalanced resources, integration efficiency prediction lag and insufficient resource optimization are solved.

The invention provides a parallel computing method for supporting real-time data stream processing in parallel programming, which comprises the following steps of: dividing a real-time data stream into fragments through a parallel pipeline, and filtering specified fields by different threads to improve the efficiency; after filtering is completed, performing local aggregation on a result by using a local aggregation function of MapReduce to reduce the data volume; a distributed memory database based on Redis Cluster is established, and data are stored in a fragmented mode according to a time window; constructing a long-short-term memory recurrent neural network model, and performing parallel training through a parameter server architecture; if the long short-term memory recurrent neural network model needs incremental learning, a parameter interface is reserved for iterative updating, and repeated training is avoided; after training is completed, the model is deployed to a GPU server, and real-time data flow is predicted; through a task scheduler of a distributed memory computing framework, cooperation between threads and processes is automatically managed, and programming work is simplified.

The invention discloses a real-time data stream processing method for multi-device collaborative management of a dome cinema, which relates to the technical field of cinema device data management, and comprises the following steps of: establishing communication connection between a central management node and a plurality of playing devices, collecting device identifiers, projection area numbers and geometric position parameters of the playing devices, and establishing a central management node; generating an equipment mapping relation table; realizing local clock correction of the playing equipment based on a unified time synchronization protocol; the central management node generates a control instruction stream according to the equipment mapping relation and the playing content, the instruction stream comprises space frame distribution information and fusion control parameters, and the space frame distribution information and the fusion control parameters are distributed to the playing equipment through communication channel numbers; and the playing device completes image geometric transformation and edge fusion processing according to the received control instruction and then executes synchronous playing. According to the invention, multi-device automatic mapping is realized, the time synchronization precision is improved, the image playing is coordinated and consistent, and the collaborative display capability and the playing stability of the dome cinema are improved.

The invention provides a fusion type secure storage system, electronic equipment, a data management method and a computer readable storage medium. The system comprises a hardware security layer, an encryption processing layer, a verification error correction layer, a data stream processing layer and an intelligent management layer which are in communication connection, the hardware security layer provides a physical trusted root for the system and executes a security policy; the encryption processing layer is used for carrying out encryption processing on data based on the security service provided by the physical trusted root; the verification and error correction layer is used for carrying out integrity verification and self-adaptive error correction coding on the encrypted data; the data stream processing layer is used for executing a near data calculation task on the data subjected to verification and error correction; and the intelligent management layer collects real-time state data from the hardware security layer, the encryption processing layer, the check error correction layer and the data stream processing layer, and performs dynamic strategy configuration and global optimization decision, thereby improving the active calculation, dynamic adaptation and intelligent optimization capabilities of the system.

The invention belongs to the technical field of data mining and analysis processing, and particularly discloses a real-time data stream processing method and system based on distributed computing, and the method comprises the steps: obtaining real-time and historical data streams of a server program window, employing a time sequence synchronization mode to rapidly position and start a window matching path and historical abnormal performance, and obtaining a real-time data stream of the server program window; the high real-time requirement is met; historical abnormal event statistics and real-time data flow similarity analysis are combined to dynamically construct a distributed tracking link, a depth-first search algorithm is used to predict a risk event, and an abnormal probability and a deviation degree are accurately quantified; different optimization strategies are configured according to different performance parameters, a multi-level defense mechanism is formed, and the self-adaptive capacity of the system is improved. According to the method, the interaction track and the performance parameter set are comprehensively analyzed, the abnormal performance link relation is marked, abnormal event correlation analysis is refined, the maintainability and expansibility of the system are enhanced, and the data processing efficiency is remarkably improved.

The invention relates to the technical field of neural network multilayer perception, in particular to a sensitive data auditing method based on neural network multilayer perception, which comprises the steps of data collection, data preprocessing, feature selection, multilayer perception neural network integration, model training, real-time data stream processing, model evaluation, result analysis and privacy protection enhancement. According to the invention, through the deep learning ability of the multi-layer perceptron (MLP) neural network, the perception and extraction ability of the data features is significantly improved. Due to the introduction of a real-time data stream processing technology, the system can respond to sensitive data leakage in time, and the security risk is effectively reduced. The self-adaptive learning mechanism ensures that the model can continuously adapt to novel threats, and the efficient detection capability is kept. The differential privacy technology provides powerful privacy protection in the data processing and model training process, and the risk of user privacy disclosure is avoided.

A device to process data associated with a surgical event of a surgery may include a processor. The processor may be configured to receive multiple data streams during the surgical event. The processor may be configured to select a primary data stream based on a surgical data interface via which the primary data stream is received. The processor may be configured to select a secondary data stream based on a surgical data interface via which the second data stream is received. The processor may be configured to identify the surgical data interfaces. The processor may be configured to generate situational data associated with the primary data stream based on the secondary data stream. The situational data may indicate a medical decision-making factor of the surgical event. The primary data stream and the situational data may be sent during the surgical event.