368 results about "Data stream processing" patented technology

The invention discloses an experimental calibration method based on multi-sensor signal fusion processing, particularly relates to the field of data analysis, and comprises the steps of a stepped confrontation disturbance protocol, data stream processing and real-time analysis, and multi-stage logic judgment and calibration execution. A stepped confrontation disturbance protocol is created for the first time, potential errors of the sensor are excited in real time and corrected in a closed loop mode under the condition that system operation is not interrupted, and a traditional shutdown calibration mode is thoroughly replaced; a multi-stage disturbance defense system is constructed, and composite interferences such as spectrum aliasing, overload distortion and signal-to-noise ratio attenuation caused by electromechanical thermal multi-physical field strong coupling are effectively suppressed; laser space calibration and dynamic rigidity modulation technologies are fused, and space drift errors caused by deformation of the mounting base and mechanical loosening are accurately diagnosed and compensated. Finally, long-term precision stability and data fusion reliability of the multi-source sensor under complex working conditions are achieved.

The invention discloses an engineering project supervision real-time visual decision-making platform based on multi-source heterogeneous data fusion, and relates to the technical field of engineering supervision. The visual decision platform is in communication connection with a data acquisition module, a project risk feature module, a risk identification model construction module, a real-time data stream processing module, a risk early warning module and a decision support module. According to the invention, the risk in the engineering project is accurately identified in combination with historical multi-source heterogeneous data and project risk characteristics, and meanwhile, the real-time data stream processing module can analyze data in real time, identify potential risks and problems and generate early warning information, so that the accuracy and timeliness of risk identification are greatly improved, and the risk identification efficiency is improved. According to the invention, powerful decision support is provided for supervisors, and through the preset early warning level and the early warning signal, managers can rapidly understand the risk condition and take corresponding measures, thereby effectively reducing the project risk.

The invention provides a thermal power plant environment monitoring and early warning method and system based on the Internet of Things, and relates to the technical field of the Internet of Things, and the method comprises the steps: forming a preprocessing data stream for collected environment monitoring data through an edge calculation node; and carrying out real-time data stream processing by utilizing a distributed message queue, constructing a hierarchical distributed data storage structure, and carrying out homomorphic encryption and desensitization processing on data to form an encrypted data set. Based on an encrypted data set, a long-short term memory network, a random forest algorithm and a limit gradient boosting tree are adopted to construct a layered anomaly detection engine, environmental parameter spatial-temporal characteristics are extracted through a multi-head attention mechanism and residual connection, an early warning threshold value is iteratively calculated based on a Bayesian optimization algorithm, and an early warning level judgment standard is dynamically adjusted. And generating early warning information. And finally, through a three-dimensional visualization module and a knowledge inference engine based on a graph neural network and a deep reinforcement learning algorithm, visualization of early warning information and generation of a self-adaptive emergency processing strategy are realized respectively.

The invention relates to the technical field of data processing, and discloses a real-time data stream processing method and device, electronic equipment and a storage medium. The method comprises the following steps: distributing a received to-be-processed data stream to a first distributed node through a preset load balancer; fragmenting the received distribution data stream at the first distributed node according to a preset fragmentation rule, then writing the fragmented data stream into a message queue, and performing parallel processing on fragmented data in the message queue through multiple threads; storing the intermediate state data in the fragmented data processing process, generating snapshot data according to the fragmented data, and synchronizing the snapshot data to the distributed storage system; acquiring load data of the first distributed node, and dynamically updating a data flow distribution strategy of the load balancer according to the load data through the resource use prediction model; and writing a processing result of the to-be-processed data stream into persistent storage through an asynchronous batch mechanism, and returning the processing result through a preset real-time communication protocol. The method improves the data stream processing efficiency and stability.

The invention discloses an intelligent data stream processing system based on Flink and an implementation method thereof, and belongs to the technical field of distributed streaming data processing, and the implementation method comprises the following steps: a data source access layer uniformly accesses multi-source data; the entity monitoring layer captures service entity change in a non-intrusive manner based on JPA, generates a standardized message body and asynchronously delivers the standardized message body to double channels; the unified event model layer maps the data into a standardized event and expands the standardized event; the Flink stream processing engine layer is used for complex event processing, state management and window calculation, and supports declarative pipeline definition and dynamic construction; the anomaly detection layer performs multi-dimensional anomaly detection; the distributed task scheduling layer schedules tasks according to a DAG model, and realizes load balancing through hotspot splitting and state transition; and the result storage and visualization layer adopts a cold and hot separation strategy to store data, and provides a dynamic visualization interface. The real-time performance, the consistency and the intelligent level of data processing are remarkably improved, and the development and maintenance cost is reduced.

The invention provides a financial data intelligent quality inspection method and system, and the method comprises the steps: S1, accessing a real-time transaction data flow through a dynamic rule engine, and enabling the dynamic rule engine to dynamically adjust the rule weight through a Bayesian network and reinforcement learning hybrid model; s2, calling a multi-modal LLM verification framework, performing joint semantic analysis on the text, the image and the time series data, and generating a risk early warning signal; s3, identifying a cross-entity risk path based on the financial knowledge graph, and converting the identified risk path into a structured risk report; and S4, a closed loop iteration system is formed according to the weight of the early warning feedback optimization rule. According to the method, full-life-cycle quality management and control of financial transactions can be realized through technical collaboration of real-time data stream processing, multi-dimensional semantic verification and cross-entity risk tracking.

The invention provides an urban space intelligent processing method based on multi-modal fusion, and the method comprises the steps: taking multi-modal data as input, and constructing a unified data stream processing and feature alignment mechanism; a physical space is used as a core framework, and the multi-modal data is converted into a space behavior graph with space-time position semantics; constructing an entity attribute-relation type-influence weight ternary interaction model on the basis of an interaction layer of the spatial behavior map, and analyzing a human, object and environment ternary interaction relation in the city and the park based on the ternary interaction model; designing a space intelligent engine with time sequence modeling and dynamic prediction capabilities; and constructing a task processing system. According to the invention, through deep combination of multi-modal fusion and a space intelligent technology, full-link upgrading of urban space from data perception to intelligent decision is realized, and powerful technical support is provided for fine management, efficient operation and safety guarantee of complex space scenes.

The invention relates to the technical field of enterprise project management, and discloses an enterprise project management method and system based on multi-source heterogeneous data integration, and the method comprises the steps: receiving heterogeneous data from a plurality of data sources; classifying the heterogeneous data into different-speed data streams of a high-speed stream, a medium-speed stream and a low-speed stream according to the updating frequency; allocating a logic timestamp for the different-speed data stream; the different-speed data flow is processed based on the logic timestamp, and the logic consistency of data processing is guaranteed; converting, aggregating and analyzing the processed data according to a preset data stream processing logic; the analysis result is applied to enterprise project management decision support; real-time performance, consistency and integrity of data in enterprise project management are achieved, efficiency and decision quality of enterprise project management are improved, and the problems of data islands, data inconsistency and data processing lag in traditional enterprise project management are solved.

A method for managing a data stream processing pipeline includes: monitoring, by an orchestrator, data stream ingestion of a streaming storage system (SSS) to obtain data stream metrics; analyzing, by the orchestrator, the data stream metrics based on a user-defined scaling policy; making, based on the analyzing and by the orchestrator, a first determination that task manager scaling is required; making, by the orchestrator, a second determination that the data stream ingestion is increased, in which the second determination indicates an increase in a number of parallel stream segments associated with a data stream, in which a segment store hosted by the SSS manages the parallel stream segments; and to increase a stream processing system's compute capability and by the orchestrator, initiating scaling of a number of the plurality of task managers to support the increase in the number of the parallel stream segments.

The invention relates to the technical field of building construction monitoring, and discloses a construction progress real-time monitoring system based on BIM and Internet of Things, and the system comprises an Internet of Things data collection module which obtains real-time construction data through a sensor network and divides the real-time construction data into a spatial position flow and an equipment state flow; the BIM data integration module processes the two types of data streams to generate a first progress index and a second progress index; the progress deviation analysis module is combined with a preset progress reference model, and the progress indexes are fused and then mapped into a progress deviation value; the response decision module calls a preset scheme based on the deviation value to generate a target response strategy; the central processing unit is responsible for data transmission and logic verification. In addition, the system is further provided with a construction linkage module, an instruction output module and a progress storage module which are respectively used for optimizing resource scheduling, sending control commands and archiving data. According to the system, real-time monitoring, deviation analysis and dynamic adjustment of the construction progress are realized, and the intelligence and accuracy of construction management are improved.

The invention relates to the technical field of storage systems, in particular to an electrolytic cell test data cloud storage method, which comprises the following steps: performing timestamp marking and partition size calculation on electrolytic cell test data, aggregating the data into partitions with fixed sizes according to timestamps, calculating a data density value and a write-in rate value of each partition, obtaining a partition data set, and storing the partition data set into a database; and comparing the write-in rate value in the partition data set with a preset partition threshold value. According to the method, through a dynamic partition management and multi-path data flow processing mechanism, in the storage process of the test data of the electrolytic cell, the data is subjected to partition aggregation through timestamp marking, a data density value and a write-in rate value reflect the dynamic change of the partition, and the size of the partition is adjusted in real time to cope with the fluctuation of a data write-in load. Multi-channel data stream distribution and merge sorting improve concurrent processing performance and eliminate single-channel bottleneck restriction. A double-buffer-area framework is matched with data access frequency analysis to support intelligent migration of cold and hot data, and IO resource configuration is optimized.

The invention discloses a shared cache system applied to a real-time network terminal chip. The shared cache system comprises queue management, enqueue scheduling control, dequeue scheduling control, write bus control, read bus control, a data cache, a register module, an RXFIFO module and a TXFIFO module. According to the cache allocation strategy, the variable-length data is stored by using the fixed-length unit. The user can customize and configure the private space, the residual space is the shared space, and the storage utilization rate is high. The shared cache is mainly used for storing data packets sent by an external application layer through a host interface. And storing the received data packet, extracting frame information according to a network flow protocol, and directly discarding frames which do not meet the protocol. The shared cache is provided with a dynamic cache structure, the state of each queue can be updated in real time, frame loss is prevented, and the data stream processing capacity is improved. And meanwhile, complex hybrid scheduling of a real-time network is met. The whole shared cache is subjected to anti-radiation reinforcement, the scene requirement of space navigation application is met, and high reliability is achieved.

The invention discloses a big data integration system based on artificial intelligence, and particularly relates to the technical field of data integration, by deploying an edge data analysis model at a big data acquisition end and combining cloud computing resources for collaborative optimization, efficiency abnormity in the data integration process is timely warned, computing resource allocation is dynamically adjusted, and the efficiency of data integration is improved. A data processing strategy is optimized, so that delay and resource waste in an integration process are effectively reduced; a data stream processing path is optimized cooperatively through edge end preprocessing and cloud deep analysis; through combined application of key node monitoring, key node performance association rule mining, adaptive performance bottleneck diagnosis and dynamic scheduling, real-time optimization and closed-loop control of a big data integration process are realized, and abnormity of an actual value of integration efficiency is avoided. The problems of low integration efficiency, unbalanced resources, integration efficiency prediction lag and insufficient resource optimization are solved.

The invention discloses a real-time data stream processing method for multi-device collaborative management of a dome cinema, which relates to the technical field of cinema device data management, and comprises the following steps of: establishing communication connection between a central management node and a plurality of playing devices, collecting device identifiers, projection area numbers and geometric position parameters of the playing devices, and establishing a central management node; generating an equipment mapping relation table; realizing local clock correction of the playing equipment based on a unified time synchronization protocol; the central management node generates a control instruction stream according to the equipment mapping relation and the playing content, the instruction stream comprises space frame distribution information and fusion control parameters, and the space frame distribution information and the fusion control parameters are distributed to the playing equipment through communication channel numbers; and the playing device completes image geometric transformation and edge fusion processing according to the received control instruction and then executes synchronous playing. According to the invention, multi-device automatic mapping is realized, the time synchronization precision is improved, the image playing is coordinated and consistent, and the collaborative display capability and the playing stability of the dome cinema are improved.

The invention provides a fusion type secure storage system, electronic equipment, a data management method and a computer readable storage medium. The system comprises a hardware security layer, an encryption processing layer, a verification error correction layer, a data stream processing layer and an intelligent management layer which are in communication connection, the hardware security layer provides a physical trusted root for the system and executes a security policy; the encryption processing layer is used for carrying out encryption processing on data based on the security service provided by the physical trusted root; the verification and error correction layer is used for carrying out integrity verification and self-adaptive error correction coding on the encrypted data; the data stream processing layer is used for executing a near data calculation task on the data subjected to verification and error correction; and the intelligent management layer collects real-time state data from the hardware security layer, the encryption processing layer, the check error correction layer and the data stream processing layer, and performs dynamic strategy configuration and global optimization decision, thereby improving the active calculation, dynamic adaptation and intelligent optimization capabilities of the system.

The invention belongs to the technical field of data mining and analysis processing, and particularly discloses a real-time data stream processing method and system based on distributed computing, and the method comprises the steps: obtaining real-time and historical data streams of a server program window, employing a time sequence synchronization mode to rapidly position and start a window matching path and historical abnormal performance, and obtaining a real-time data stream of the server program window; the high real-time requirement is met; historical abnormal event statistics and real-time data flow similarity analysis are combined to dynamically construct a distributed tracking link, a depth-first search algorithm is used to predict a risk event, and an abnormal probability and a deviation degree are accurately quantified; different optimization strategies are configured according to different performance parameters, a multi-level defense mechanism is formed, and the self-adaptive capacity of the system is improved. According to the method, the interaction track and the performance parameter set are comprehensively analyzed, the abnormal performance link relation is marked, abnormal event correlation analysis is refined, the maintainability and expansibility of the system are enhanced, and the data processing efficiency is remarkably improved.

A device to process data associated with a surgical event of a surgery may include a processor. The processor may be configured to receive multiple data streams during the surgical event. The processor may be configured to select a primary data stream based on a surgical data interface via which the primary data stream is received. The processor may be configured to select a secondary data stream based on a surgical data interface via which the second data stream is received. The processor may be configured to identify the surgical data interfaces. The processor may be configured to generate situational data associated with the primary data stream based on the secondary data stream. The situational data may indicate a medical decision-making factor of the surgical event. The primary data stream and the situational data may be sent during the surgical event.

The invention provides an atomic index generation method based on real-time data streaming processing, and relates to the technical field of data processing, and the method comprises the steps: generating a cache key based on an attribute unique code and an atomic index code, and querying whether a corresponding index calculation object exists in a cache; if the cache object does not exist, initializing an index calculation object comprising an index value, a start timestamp, a calculation type, an atomic index code and a calculation rule, and storing the index calculation object into a cache; if the cache object exists, corresponding index calculation operation is executed according to the calculation type, the index calculation operation comprises duration calculation, accumulated value calculation, aggregation calculation or user-defined rule calculation, and an index value in the cache is updated; writing the cached index values into a target database in batches according to a preset time period, and avoiding repeated writing by taking timestamps and the atomic index codes as unique identifiers; and resetting the index value in the cache to prepare for the next calculation period.

The invention discloses a multi-task parallel data stream processing method and device based on an FPGA (Field Programmable Gate Array), which are applied to the technical field of data processing. Carrying out normalization processing on the decoded data to generate a data stream in a unified format; performing coding and ID identification code embedding processing on the normalized data to generate to-be-stored data with an identifier; carrying out FIFO (First In First Out) buffering and write-in priority judgment on the coded data to generate priority queue data; dDR writing and BANK scheduling processing are carried out on the priority queue data, and stably stored parallel data are generated; carrying out reading and output coordination processing on DDR storage data to generate a continuous output data stream, and triggering a reading application by decoding ID identification code positioning data, reading FIFO according to depth scheduling and monitoring data volume by channel temporary storage FIFO; and performing dynamic bandwidth allocation and energy efficiency optimization on the whole data stream processing process to generate an efficient parallel processing result.

The invention provides a hemodialysis hypotension prediction system based on machine learning, which relates to the technical field of medical health, and comprises a real-time data stream processing module, a dynamic modeling and prediction module, a real-time data stream processing module, a real-time data stream processing module, a real-time data stream processing module, a real-time data stream processing module and a real-time data stream processing module, the risk prediction module is used for dynamically modeling and predicting the hypotension risk of a patient according to physiological parameter data input in real time, and the personalized adaptation module is used for automatically adjusting a risk prediction model. According to the hemodialysis hypotension prediction system based on machine learning, by introducing the real-time data flow processing module, physiological parameter data in the hemodialysis process can be collected and processed in real time. By adopting the dynamic modeling and predicting module, the hypotension risk can be dynamically modeled and predicted based on the physiological parameter data input in real time, and the prediction model is automatically adjusted through the personalized adaptation module, so that the risk prediction result of each patient is more accurate.

A method for reducing latency and increasing throughput in a reconfigurable computing system includes receiving a compute graph for execution on a reconfigurable dataflow processor comprising a grid of compute units and grid of memory units interconnected with a switching array. The compute graph includes a node specifying an operation on a tensor. The node may be split into multiple nodes that each specify the operation on a distinctive portion of the tensor to produce a first modified compute graph. The first modified compute graph may be executed. In addition, the multiple nodes may be within a single meta-pipeline stage and may be processed in parallel. Furthermore, the compute graph may further comprise a separate node for gathering the distinctive portions of the tensor into a complete tensor, to produce a second modified compute graph.

The invention discloses a cluster collection communication system applied to AI data stream processing, which relates to the technical field of distributed systems and network engineering and comprises a topology network deployment module, a data stream spatio-temporal data acquisition module, an optimization verification module, a state monitoring and early warning module and a visual traceability module. The topology network deployment module synchronously processes AI data streams, the data stream spatio-temporal data acquisition module extracts an entropy value and a covariance matrix, the optimization verification module outputs optimization parameters and a credible verification result, the state monitoring early warning module sets an adaptive threshold value to realize graded early warning, and the visual traceability module generates a three-dimensional thermodynamic diagram and a traceability graph. According to the method, the data stream processing efficiency is improved through the star-ring hybrid topology and the low-delay protocol, the decision credibility is enhanced in combination with quantum optimization and block chain verification, accurate early warning and three-dimensional visual traceability acceleration fault positioning are realized by using dynamic weighted monitoring and an adaptive threshold, and an efficient and intelligent full-stack solution is provided for AI trunking communication.

The invention provides a data stream parallel acceleration processing method and system in chip design, and relates to the technical field of chip design, and the method comprises the steps: partitioning a data stream through a preset scale, building a computing resource pool with graded computing power and a storage resource pool with graded access speed, and achieving the optimal distribution of storage resources and calculation tasks; dividing parallel processing units, and determining a data processing sequence; setting a shared cache area to establish a distributed cache synchronization network, and realizing synchronization control based on a cache identifier and a Lamport logic clock mechanism; and generating a resource competition degree matrix, and rearranging the task according to the access popularity map. According to the method, the data stream processing efficiency and the resource utilization rate of the chip are improved, and resource competition conflicts are reduced.

The invention provides a flexible glove based on sparse tensile sensing and a three-dimensional hand fine-grained reconstruction method, and relates to the technical field of man-machine interaction. The method comprises the following steps: constructing a biological constraint sensing sensor array layout; according to the sensor array layout, a wearable glove integrated with six-channel sparse stretching sensing is designed; constructing a three-dimensional multi-modal dynamic hand data set according to the wearable glove; designing a data flow processing pipeline; inputting the data set into a data stream processing pipeline to obtain a preprocessed data set; constructing a multi-stage graph attention network; inputting the preprocessed data set into a multi-stage graph attention network to obtain space mapping data from a 6-channel signal to a 15-joint signal; the mapping data are input into a multi-scale Transform model, and motion feature data of the 15 joints on the time sequence are obtained; according to the motion feature data, a forward kinematics model is adopted for decoding, and the rotation quaternion of the 20 skeletons is output. According to the invention, the stability of the wearable interactive glove can be improved.

The invention relates to the technical field of servers, and particularly provides a PCIe BAR space mapping method, system and device for a data stream processor and a medium, and the method comprises the steps: reading the configuration space information of a PCIe device, and determining the total size of a memory space needing to be used by the data stream processor based on the configuration space information; obtaining the size of a special memory of each computing core; determining the size of a DDR (Double Data Rate) memory area for task calculation according to the total size of the memory space and the size of the special memory; instantiating a static container for a special memory equipped for each computing core, and instantiating a dynamic container for the DDR memory area; and distributing different BAR spaces for the static container and the dynamic container. According to the method, through a dual-mode address mapping mechanism of the static container and the dynamic container, the key problem that a 32-bit PCIe BAR space cannot support super 4GB memory access of a data stream processor is solved.

This invention discloses a method and apparatus for processing multi-task parallel data streams based on FPGA, applied in the field of data processing technology. The invention constructs a multi-format raw data receiving and decoding system; normalizes the decoded data to generate a unified format data stream; encodes and embeds ID identification codes into the normalized data to generate tagged data to be stored; performs FIFO buffering and write priority determination on the encoded data to generate a priority queue data; performs DDR write and BANK scheduling on the priority queue data to generate stable parallel data; and coordinates the reading and output of the DDR stored data to generate a continuous output data stream. Specifically, data is located by decoding the ID identification code, reads from the FIFO by depth scheduling, and the channel temporary storage FIFO monitors the data volume to trigger read requests. Dynamic bandwidth allocation and energy efficiency optimization are performed throughout the data stream processing process to generate efficient parallel processing results.

The invention discloses a low-delay data compression method and system for a 5G communication network, and the method comprises the steps: obtaining the characteristics of an input data flow, and determining an initial resource distribution proportion and an output load distribution parameter which are needed by dynamic load balancing through analyzing the size and arrival frequency of a data packet; according to the load distribution parameters, an edge node scheduling algorithm is adopted to adjust and calculate the resource distribution proportion, and if the data flow burst amount exceeds a preset flow threshold value, the edge node processing capacity is increased, and an optimized resource scheduling scheme is obtained; obtaining a current data stream processing request from an edge node through an optimized resource scheduling scheme, generating a compression task queue in combination with an efficient coding technology, and outputting a to-be-processed data block sequence; and for the to-be-processed data block sequence, executing high-compression-rate coding by adopting a rapid compression algorithm, and if the compression time exceeds 5G short-time slot limitation, switching to a low-calculation-complexity mode. According to the invention, the system resource utilization rate and the data transmission efficiency are improved.

The invention discloses a data stream processing method and device, electronic equipment and a storage medium, and relates to the technical field of distributed storage. According to the data stream processing method and device, the data stream transmission mode is pre-judged to be intra-node transmission or cross-node transmission based on a hardware topology mapping table, and read-write caches are dynamically allocated according to the pre-judged data stream transmission mode; data received by the front-end network card is directly written into a controller memory buffer area of the target equipment through point-to-point direct memory access for data streams transmitted in the nodes, and is read and forwarded from the target equipment by the rear-end network card; the sending cache and the receiving cache of the data stream transmitted across the nodes are locked to the non-uniform memory access node to which the target network card belongs, and the data are sent to the target node through the remote direct memory access register memory, so that the problem that the data stream processing is not optimized in combination with hardware topology characteristics can be solved; the technical effects of reducing data replication overhead, avoiding redundant access across hardware nodes, improving data stream transmission efficiency and realizing end-to-end zero replication transmission are achieved.

The invention discloses a data verification method and system based on a Drools rule engine, and belongs to the technical field of data analys.The method comprises the steps that a data stream is accessed in real time, and the data stream is preprocessed to form a structured data object; receiving a verification rule configured by a user, and performing dynamic compiling and hot loading on the verification rule to update a rule base in the Drools rule engine; and inserting the structured data object as a fact object into a Drools rule engine, and executing the verification rule configured in the rule base by the Drools rule engine based on a ReteOO algorithm to generate a verification result. According to the method, real-time data stream processing and large-scale field association verification are supported, so that the data quality, the service efficiency and the overall reliability of the system in the aviation field are improved.

Provided in the present disclosure is a data stream processing method. The method comprises: in response to the failure of a first load sharing link in a load sharing link group, determining an actual load sharing ratio of each second load sharing link among at least one second load sharing link, wherein the second load sharing link is a link in the load sharing link group which is in a normal state, and the actual load sharing ratio is used for representing an actual ratio of the second load sharing link in a forwarding table of the load sharing link group; on the basis of the actual load sharing ratio of each second load sharing link, determining at least one target load sharing link from among the at least one second load sharing link; and processing a data stream on the first load sharing link by means of the at least one target load sharing link. Further provided in the present disclosure are an electronic device, a computer-readable storage medium and a computer program product.