256 results about "Data stream processing" patented technology

The invention discloses an intelligent data stream processing system based on Flink and an implementation method thereof, and belongs to the technical field of distributed streaming data processing, and the implementation method comprises the following steps: a data source access layer uniformly accesses multi-source data; the entity monitoring layer captures service entity change in a non-intrusive manner based on JPA, generates a standardized message body and asynchronously delivers the standardized message body to double channels; the unified event model layer maps the data into a standardized event and expands the standardized event; the Flink stream processing engine layer is used for complex event processing, state management and window calculation, and supports declarative pipeline definition and dynamic construction; the anomaly detection layer performs multi-dimensional anomaly detection; the distributed task scheduling layer schedules tasks according to a DAG model, and realizes load balancing through hotspot splitting and state transition; and the result storage and visualization layer adopts a cold and hot separation strategy to store data, and provides a dynamic visualization interface. The real-time performance, the consistency and the intelligent level of data processing are remarkably improved, and the development and maintenance cost is reduced.

The invention provides an urban space intelligent processing method based on multi-modal fusion, and the method comprises the steps: taking multi-modal data as input, and constructing a unified data stream processing and feature alignment mechanism; a physical space is used as a core framework, and the multi-modal data is converted into a space behavior graph with space-time position semantics; constructing an entity attribute-relation type-influence weight ternary interaction model on the basis of an interaction layer of the spatial behavior map, and analyzing a human, object and environment ternary interaction relation in the city and the park based on the ternary interaction model; designing a space intelligent engine with time sequence modeling and dynamic prediction capabilities; and constructing a task processing system. According to the invention, through deep combination of multi-modal fusion and a space intelligent technology, full-link upgrading of urban space from data perception to intelligent decision is realized, and powerful technical support is provided for fine management, efficient operation and safety guarantee of complex space scenes.

The invention provides a fusion type secure storage system, electronic equipment, a data management method and a computer readable storage medium. The system comprises a hardware security layer, an encryption processing layer, a verification error correction layer, a data stream processing layer and an intelligent management layer which are in communication connection, the hardware security layer provides a physical trusted root for the system and executes a security policy; the encryption processing layer is used for carrying out encryption processing on data based on the security service provided by the physical trusted root; the verification and error correction layer is used for carrying out integrity verification and self-adaptive error correction coding on the encrypted data; the data stream processing layer is used for executing a near data calculation task on the data subjected to verification and error correction; and the intelligent management layer collects real-time state data from the hardware security layer, the encryption processing layer, the check error correction layer and the data stream processing layer, and performs dynamic strategy configuration and global optimization decision, thereby improving the active calculation, dynamic adaptation and intelligent optimization capabilities of the system.

A device to process data associated with a surgical event of a surgery may include a processor. The processor may be configured to receive multiple data streams during the surgical event. The processor may be configured to select a primary data stream based on a surgical data interface via which the primary data stream is received. The processor may be configured to select a secondary data stream based on a surgical data interface via which the second data stream is received. The processor may be configured to identify the surgical data interfaces. The processor may be configured to generate situational data associated with the primary data stream based on the secondary data stream. The situational data may indicate a medical decision-making factor of the surgical event. The primary data stream and the situational data may be sent during the surgical event.

The invention discloses a multi-task parallel data stream processing method and device based on an FPGA (Field Programmable Gate Array), which are applied to the technical field of data processing. Carrying out normalization processing on the decoded data to generate a data stream in a unified format; performing coding and ID identification code embedding processing on the normalized data to generate to-be-stored data with an identifier; carrying out FIFO (First In First Out) buffering and write-in priority judgment on the coded data to generate priority queue data; dDR writing and BANK scheduling processing are carried out on the priority queue data, and stably stored parallel data are generated; carrying out reading and output coordination processing on DDR storage data to generate a continuous output data stream, and triggering a reading application by decoding ID identification code positioning data, reading FIFO according to depth scheduling and monitoring data volume by channel temporary storage FIFO; and performing dynamic bandwidth allocation and energy efficiency optimization on the whole data stream processing process to generate an efficient parallel processing result.

The invention provides a hemodialysis hypotension prediction system based on machine learning, which relates to the technical field of medical health, and comprises a real-time data stream processing module, a dynamic modeling and prediction module, a real-time data stream processing module, a real-time data stream processing module, a real-time data stream processing module, a real-time data stream processing module and a real-time data stream processing module, the risk prediction module is used for dynamically modeling and predicting the hypotension risk of a patient according to physiological parameter data input in real time, and the personalized adaptation module is used for automatically adjusting a risk prediction model. According to the hemodialysis hypotension prediction system based on machine learning, by introducing the real-time data flow processing module, physiological parameter data in the hemodialysis process can be collected and processed in real time. By adopting the dynamic modeling and predicting module, the hypotension risk can be dynamically modeled and predicted based on the physiological parameter data input in real time, and the prediction model is automatically adjusted through the personalized adaptation module, so that the risk prediction result of each patient is more accurate.

The invention provides a data stream parallel acceleration processing method and system in chip design, and relates to the technical field of chip design, and the method comprises the steps: partitioning a data stream through a preset scale, building a computing resource pool with graded computing power and a storage resource pool with graded access speed, and achieving the optimal distribution of storage resources and calculation tasks; dividing parallel processing units, and determining a data processing sequence; setting a shared cache area to establish a distributed cache synchronization network, and realizing synchronization control based on a cache identifier and a Lamport logic clock mechanism; and generating a resource competition degree matrix, and rearranging the task according to the access popularity map. According to the method, the data stream processing efficiency and the resource utilization rate of the chip are improved, and resource competition conflicts are reduced.

The invention relates to the technical field of servers, and particularly provides a PCIe BAR space mapping method, system and device for a data stream processor and a medium, and the method comprises the steps: reading the configuration space information of a PCIe device, and determining the total size of a memory space needing to be used by the data stream processor based on the configuration space information; obtaining the size of a special memory of each computing core; determining the size of a DDR (Double Data Rate) memory area for task calculation according to the total size of the memory space and the size of the special memory; instantiating a static container for a special memory equipped for each computing core, and instantiating a dynamic container for the DDR memory area; and distributing different BAR spaces for the static container and the dynamic container. According to the method, through a dual-mode address mapping mechanism of the static container and the dynamic container, the key problem that a 32-bit PCIe BAR space cannot support super 4GB memory access of a data stream processor is solved.

This invention discloses a method and apparatus for processing multi-task parallel data streams based on FPGA, applied in the field of data processing technology. The invention constructs a multi-format raw data receiving and decoding system; normalizes the decoded data to generate a unified format data stream; encodes and embeds ID identification codes into the normalized data to generate tagged data to be stored; performs FIFO buffering and write priority determination on the encoded data to generate a priority queue data; performs DDR write and BANK scheduling on the priority queue data to generate stable parallel data; and coordinates the reading and output of the DDR stored data to generate a continuous output data stream. Specifically, data is located by decoding the ID identification code, reads from the FIFO by depth scheduling, and the channel temporary storage FIFO monitors the data volume to trigger read requests. Dynamic bandwidth allocation and energy efficiency optimization are performed throughout the data stream processing process to generate efficient parallel processing results.

The invention discloses a data stream processing method and device, electronic equipment and a storage medium, and relates to the technical field of distributed storage. According to the data stream processing method and device, the data stream transmission mode is pre-judged to be intra-node transmission or cross-node transmission based on a hardware topology mapping table, and read-write caches are dynamically allocated according to the pre-judged data stream transmission mode; data received by the front-end network card is directly written into a controller memory buffer area of the target equipment through point-to-point direct memory access for data streams transmitted in the nodes, and is read and forwarded from the target equipment by the rear-end network card; the sending cache and the receiving cache of the data stream transmitted across the nodes are locked to the non-uniform memory access node to which the target network card belongs, and the data are sent to the target node through the remote direct memory access register memory, so that the problem that the data stream processing is not optimized in combination with hardware topology characteristics can be solved; the technical effects of reducing data replication overhead, avoiding redundant access across hardware nodes, improving data stream transmission efficiency and realizing end-to-end zero replication transmission are achieved.

The invention discloses a data verification method and system based on a Drools rule engine, and belongs to the technical field of data analys.The method comprises the steps that a data stream is accessed in real time, and the data stream is preprocessed to form a structured data object; receiving a verification rule configured by a user, and performing dynamic compiling and hot loading on the verification rule to update a rule base in the Drools rule engine; and inserting the structured data object as a fact object into a Drools rule engine, and executing the verification rule configured in the rule base by the Drools rule engine based on a ReteOO algorithm to generate a verification result. According to the method, real-time data stream processing and large-scale field association verification are supported, so that the data quality, the service efficiency and the overall reliability of the system in the aviation field are improved.

Provided in the present disclosure is a data stream processing method. The method comprises: in response to the failure of a first load sharing link in a load sharing link group, determining an actual load sharing ratio of each second load sharing link among at least one second load sharing link, wherein the second load sharing link is a link in the load sharing link group which is in a normal state, and the actual load sharing ratio is used for representing an actual ratio of the second load sharing link in a forwarding table of the load sharing link group; on the basis of the actual load sharing ratio of each second load sharing link, determining at least one target load sharing link from among the at least one second load sharing link; and processing a data stream on the first load sharing link by means of the at least one target load sharing link. Further provided in the present disclosure are an electronic device, a computer-readable storage medium and a computer program product.

The invention relates to an optical storage and charging scene data stream processing and collaborative decision-making method, which relates to the technical field of energy management and comprises the following steps of: acquiring and preprocessing optical storage and charging multi-source data; establishing a cross-hierarchy data interaction mechanism, adopting a Modbus unified protocol to realize linkage instruction transmission between devices, connecting an EMS and a power grid dispatching platform upwards, connecting a user side system downwards, and formulating an integrated IEC 104 standard data interaction specification; construction of a knowledge graph: defining core entities such as equipment, indexes, scenes and the like; according to'perception-interaction-decision 'closed-loop cooperation, system data is collected to provide a basis for the knowledge graph, data interaction guarantees scene information integrity, and rules generated by the knowledge graph reversely guide operation of control instructions. The operation efficiency of the optical storage and charging system can be effectively improved, and the cost per kilowatt-hour is reduced.

The invention discloses a data stream processing method and system based on deep reinforcement learning, and relates to the technical field of industrial Internet of Things. Comprising the following steps: S1, constructing a topological graph neural network, determining a feature similarity weight of equipment topological features through a multi-head graph attention mechanism, and dynamically aligning the equipment topological features of a source domain and a target domain; s2, setting a main channel through a deep reinforcement learning model, setting an auxiliary channel through a time sequence contrast element learning model, and obtaining health degree evaluation and fault mode characteristics of the equipment; and S3, monitoring a data flow according to the health degree evaluation and the fault mode characteristics, and recombining a network structure through a neural architecture search model. According to the method, the transfer learning problem caused by equipment isomerism in the industrial Internet of Things is solved, the physical distance, the process coupling and the feature similarity can be dynamically balanced, the initial accuracy of the source domain model in the target domain is improved, and the annotation data volume required by the target domain is reduced.

The invention relates to the technical field of information, in particular to an information optimization method for cooperative supervision of functional departments, which comprises the following steps: for a structured supervision data set, classifying permission allocation modes by adopting an analysis algorithm, and determining permission feature distribution; according to the permission feature distribution, constructing a correlation model of permission configuration and supervision efficiency, and obtaining a contribution index of the permission configuration to the efficiency through a quantitative analysis technology; adjusting a permission allocation scheme according to the contribution degree index, and generating a dynamic permission configuration scheme; aiming at a collaborative path optimization result, acquiring information sharing data by adopting a data stream processing technology, and generating an information sharing efficiency index; according to the information sharing efficiency index and the contribution degree index, performing grading evaluation on the supervision efficiency through a classification algorithm to obtain an efficiency evaluation result; adjusting a cooperation mechanism model according to an efficiency evaluation result, and generating an optimized cooperation mechanism scheme; and the supervision efficiency and the information sharing level are improved.

The invention relates to a communication engineering full-process safety production risk dynamic assessment method, in particular to the technical field of communication engineering, and effectively overcomes the hysteresis quality and one-sidedness of a traditional static assessment method. According to the method, real-time data stream processing and an online learning mechanism are fused, continuous perception and quantitative evaluation of safety production risks are realized, short-term sudden risk response and long-term law research and judgment can be intelligently balanced, the timeliness of risk early warning and the accuracy of decision support are remarkably improved, the engineering field safety management and control capability is enhanced, and the engineering field safety management and control efficiency is improved. The occurrence probability of safety accidents is reduced.

The invention relates to the technical field of task scheduling, and discloses a task scheduling method, system and device for a data stream processor and a storage medium, a host end uses a heterogeneous computing framework, the total computing resource amount of a device end does not need to be considered during compiling, resource allocation is automatically allocated by a stream control core of the data stream processor, and the task scheduling efficiency is improved. And the resource configuration of hardware does not restrict the scale of the calculation task. And the C flow control core can analyze the DAG task graph, and issues the calculation task to the calculation node in a time-sharing manner. The DAG task graphs of multiple calculation tasks can be processed at the same time, the parallel execution capacity of the data flow processor is improved, then the processing efficiency of the data flow processor is improved, and the problem that in an existing task scheduling scheme of the data flow processor, the overall calculation resource utilization rate of the data flow processor is low is solved.

The invention provides a data stream processing method for binding a kernel and a network card queue, and relates to the technical field of data stream processing, and the method comprises the following steps: setting the number of queues of each gigabit network card and each 10-gigabit network card as the maximum number of queues supported by each network card and the minimum number of idle CPU (Central Processing Unit) cores; binding each queue of each gigabit network card and each 10-gigabit network card to an idle CPU core; the number of queues of each gigabit network card is set to be the minimum value of the maximum number of queues supported by each gigabit network card and the number of idle CPU cores, and the number of queues of each 10-gigabit network card is set to be the minimum value of the maximum number of queues supported by each 10-gigabit network card and the number of idle CPU cores; enabling each queue to process a corresponding data stream through an idle CPU; according to the invention, the problems of non-intelligent and unbalanced resource allocation and possible configuration failure in a mixed rate network card environment are effectively solved.

The invention relates to the technical field of data processing, and discloses a visual large-screen dynamic display method and system based on data driving, and the method comprises the steps: collecting business scene demand data, generating a display logic configuration instruction, and setting a display content type, a spatial layout and an interaction mode; the calling data interface is connected with a background storage system and obtains service data in real time through the data stream processing module; performing time sequence analysis, sliding window detection and association rule calculation on the data to obtain a freshness index, an abnormal fluctuation parameter and a coupling degree, and judging whether abnormal fusing is triggered or not; performing exception filtering and standardization on the processed data, mapping the processed data into a graph data structure, optimizing layout through the Fresnel's law, adjusting color matching through a color contrast algorithm, and generating a chart graph; and dynamically refreshing the interface by the rendering engine. According to the method, on-demand customization and dynamic adaptation are realized, and the influence of wrong data on the accuracy of a large screen is avoided.

The invention discloses an e-commerce data intelligent processing method and system based on dynamic flow optimization. The method comprises the following steps: acquiring and analyzing multi-platform e-commerce data, and converting the multi-platform e-commerce data into a standardized format; constructing a matching rule by adopting a decision tree algorithm, and performing classification and priority marking on the data; the data flow is dynamically scheduled based on the priority, so that the high-priority data is executed preferentially; data are processed through parallel nodes, performance indexes such as network delay and throughput are monitored in real time, and data stream processing efficiency is optimized in combination with a dynamic scheduling algorithm. The system correspondingly comprises a data standardization module, a rule matching module, a dynamic marking module, a parallel processing module and a monitoring scheduling module, efficient and self-adaptive processing of the multi-source e-commerce data is achieved, and the data processing real-time performance and the system throughput are remarkably improved.

A method for a reconfigurable computing system includes receiving a compute graph for execution on multiple RDPs interconnected with a ring network having R interconnected RDPs. A compute graph with a node specifying a reduction operation for a first and second tensor is detected. Executing the compute graph on the multiple RDPs.

The invention provides a data stream processing method and system based on cloud edge collaboration. The method comprises the following steps: acquiring an original data stream, and performing data cleaning on the original data stream to obtain a current data stream; carrying out object identification on the current data stream, and determining the state attribute of each object element; determining each portrait attitude based on the current data stream, and judging whether a control interaction relationship exists between each portrait attitude and each object element of the dynamic attribute; determining each object element having a control interaction relationship with the same portrait posture as a control interaction set, and sending the control interaction set to a user side; and determining at least part of the object elements as an interactive customized combination based on the selection of the user side on at least part of the object elements in the control interaction set corresponding to any portrait posture, and performing entry labeling on the interactive customized combination by using the obtained customized entry created by the user side. According to the method, the corresponding computing power network is combined, so that the customized demand experience is at least improved.

The invention discloses a distributed acoustic sensing earthquake monitoring data stream processing method and system, and belongs to the technical field of earthquake monitoring and data processing. The method comprises the following steps: after preprocessing original seismic monitoring data, slicing the original seismic monitoring data by adopting a sliding processing time window to obtain data fragments suitable for real-time detection; designing a self-adaptive short-time average / long-time average detection algorithm, detecting data segments, and dynamically adjusting self-adaptive parameters in the algorithm according to data characteristics; and finally, multi-channel fusion detection is performed on the initial candidate list of the seismic events, so that a final detection list of the seismic events is generated. The method is based on a distributed streaming data processing architecture, combines the innovation advantages of a self-adaptive earthquake detection algorithm, can improve the earthquake detection precision, guarantees the real-time performance of a detection result, and can provide efficient and reliable technical support for a large-scale earthquake monitoring task.

The purpose of this invention is to address the issue of accuracy in processing spinning time-series data streams, which are characterized by high noise levels and distinct time-domain features generated during the operation of spinning workshops, by employing a fuzzy hierarchical clustering method that integrates time-domain characteristics. This method reduces the impact of noise during the classification process and considers both time-domain features and noise effects. The technical solution of this invention is to provide a fuzzy hierarchical clustering analysis method for spinning time-series data that integrates time-domain characteristics. This invention proposes a fuzzy hierarchical clustering analysis method for spinning time-series data that integrates time-domain features. It iterates between the DTS feature matrix and the MTS feature matrix, considering the time-domain characteristics and noise effects in the spinning time-series data. Without increasing time complexity, it incorporates the time-frequency characteristics and noise effects in the spinning time-series data during the iteration process. Compared with the latest methods, this invention can more accurately process newly generated time-series data in spinning manufacturing.

An engine identifying segments or portions of one source material or source file common to or found in another source material or file. The engine may receive a first data stream in binary form as well as a second stream in binary form. The engine may include a data stream processor or pre-processor programed to translate the first and second data streams to generate respective first and second processed data streams. The commonality between the first and second processed data streams may be greater than the commonality between the first and second data streams themselves. Also, a comparator may be programmed to compare the first and second process data streams and identify binary segments found in both the first and second processed data streams.

The invention relates to the technical field of data processing, and discloses a data stream processing method and device based on dynamic bandwidth allocation, equipment and a medium, when a data request of a client is received, a new redirection request is generated by judging a data stream type and applying redirection logic when the data stream type is confirmed to be a target data type, and the data stream is redirected. Therefore, the data streams can be quickly and accurately classified and guided, congestion of the data streams on unnecessary paths is avoided, and the condition of resource competition is relieved. According to the method, the channel specially used for transmitting the target data stream is determined, and independent transmission paths are provided for different types of data streams, so that mutual interference among data is reduced, and the occurrence probability of system jamming is remarkably reduced. According to the method, the traffic resource of each channel is calculated and allocated by applying a traffic dynamic allocation algorithm, so that the processing capability of high-concurrency scenes of financial and medical data processing is greatly improved, and the processing efficiency and the response speed of data streams are remarkably improved.

The application discloses a data anomaly detection method and device, a processor and electronic equipment. The method comprises the following steps: obtaining the identification information of at least one preset calculation module and at least one preset storage module; filling the identification information into a preset adaptation table of a preset data warehouse; predicting the adaptation probability of the preset calculation module and the preset storage module according to a preset adaptation model, and filling the adaptation probability into the preset adaptation table, wherein the preset adaptation model is obtained by training a deep learning algorithm based on a self-attention mechanism; performing data caching of the preset adaptation table by calling stream data and a remote dictionary service through the preset calculation module, to obtain a data stream processing table, wherein the remote dictionary service and the preset calculation module establish a mapping relationship in the form of a key-value pair; and analyzing the data stream processing table to realize data anomaly detection. The application solves the technical problem of low data anomaly detection efficiency.

Various techniques are provided for providing neural networks with increased efficiency. In one example, a system includes a first artificial neural network (ANN), a second ANN, and a logic device. The first ANN is configured to receive a first plurality of data inputs associated with a data stream and process the first data inputs to generate a first inference output after a first latency. The second ANN is configured to receive a second plurality of data inputs associated with the data stream and process the second data inputs to generate a second inference output after a second latency less than the first latency. The logic device is configured to receive the second inference output before the first inference output is generated. Additional systems and methods are also provided.

The invention discloses a numerical weather forecast data correction method based on collaborative optimization, and the method comprises the steps: carrying out the grading processing of ANO, KF and RF according to error characteristics, forming a progressive correction path of system error-time sequence error-nonlinear residual error, combining the dynamic weight distribution of three correction methods, and carrying out the correction of the numerical weather forecast data. The method can effectively improve the accuracy of each parameter in numerical weather forecast data, supports real-time data stream processing, can be embedded into an existing forecast service system, improves the refinement level of weather forecast, provides weather information service with higher reference value for each industry, and enhances the ability to deal with meteorological disasters.

The invention relates to a compiler design and code optimization technology, in particular to a compiler infrastructure based on multi-order intermediate representation and an implementation method of the compiler infrastructure. A traditional compiler adopts single intermediate representation, so that the coupling degree of the optimization process is high; the method is closely related to high-level / low-level languages, so that the compiler is poor in universality. In order to solve the problems, the invention provides the compiler infrastructure based on the multi-level intermediate representation and the implementation method of the compiler infrastructure, through modular design and hierarchical processing, the relevance of high-level / low-level languages is stripped, and the multi-level intermediate representation is designed, so that the problem that the compiler infrastructure cannot be used in the process of converting the high-level languages into the low-level languages is solved; the processing complexity of the control flow and the data flow is high, the universality is poor, and the subsequent maintenance difficulty is high. The method is suitable for real-time compiling, heterogeneous hardware adaptation and multi-language mixed programming and static optimization scenes, and the development and maintenance cost of the compiler can be greatly reduced.