22 results about "Distributed data flow" patented technology

The invention relates to the technical field of network communication, in particular to a multipath transmission load balancing method and system based on network state analysis. The method comprises the steps of collecting network state information of each transmission link from network equipment, calculating and obtaining the weight of each transmission link according to the collected network state information, setting a flow distribution strategy according to the weight obtained by calculation of each transmission link, and distributing data flow to each transmission link in proportion, and collecting the network state information of each transmission link in real time, updating the weight of each transmission link according to the change of each parameter in the network state information, performing threshold judgment on the updated weight, and dynamically adjusting the flow distribution strategy according to the weight meeting the threshold requirement. According to the method, the defects of a traditional static strategy can be effectively overcome, link congestion or idle is avoided, high efficiency and stability of data transmission are guaranteed, and finally the requirement of a user for high-quality network service is met.

The embodiment of the invention discloses a product pushing method, electronic equipment, a storage medium and a program product, and relates to the technical field of financial science and technology, and the method comprises the steps: obtaining behavior data of a user in a target application before the current moment from a distributed publishing and subscribing system through a distributed data flow engine, obtaining real-time behavior data, and transmitting the real-time behavior data to a server; obtaining offline behavior data of the user from the data warehouse to obtain historical behavior data; fusing the real-time behavior data and the historical behavior data to obtain user behavior data; determining a pushing rule satisfied by the user behavior data from a preset pushing rule base to obtain a plurality of candidate rules; obtaining a static feature and a dynamic feature of each candidate rule, and determining a priority score of the corresponding candidate rule based on the static feature and the dynamic feature; and the optimal rule is determined based on the priority score of each candidate rule, and the target product corresponding to the optimal rule is pushed to the user terminal of the user, so that the timeliness and accuracy of product pushing are improved.

The invention discloses a real-time processing method and system for distributed data streams, and particularly relates to the technical field of distributed data processing, and the method comprises the following steps: collecting multi-source data streams and marking priorities, combining data types and priorities to carry out adaptive segmentation, carrying out load balancing distribution on processing units, and resolving data conflicts; and incremental integration and caching of hotspot data are carried out, and full-link tracing and dynamic tuning are carried out. The system comprises five modules, namely an acquisition labeling module, a segmentation module, a parallel processing module, an integration optimization module and a tracing tuning module. According to the real-time processing method and system for the distributed data flow, priority processing of core data is achieved, data consistency is guaranteed, the resource utilization rate and the system maintainability are improved, and the processing requirements for high real-time performance and high reliability are met.

The invention discloses an identity authentication method and device in a distributed data circulation environment, equipment and a medium, and relates to the technical field of computers. Comprising the following steps: acquiring an identity registration request of an access subject through a regional function node, verifying real-name identity information of the access subject, and returning a corresponding identity registration result to the access subject based on a verification result of the real-name identity information; obtaining an access connector registration request of an access main body through a regional function node, and issuing a trusted identity certificate to the access main body and a target access connector according to an information auditing result of the target access connector; and if a login request of the access main body is received through the regional function node, the access main body is accessed to a preset service processing platform through the target access connector based on the trusted identity certificate to complete an identity authentication process. Therefore, cross-domain identity mutual recognition and hierarchical authorization can be realized, and basic identity guarantee is provided for safe, efficient and compliant circulation of data elements.

A data processing method, a data processing apparatus, a distributed data flow programming framework, an electronic device, and a storage medium. The data processing method includes: dividing a data processing task into a plurality of data processing subtasks (S101); determining, in a Field Programmable Gate Array (FPGA) accelerator side, a target FPGA acceleration board corresponding to each of the data processing subtasks (S102); and sending data to be computed to the target FPGA acceleration board, and executing the corresponding data processing subtask by use of each of the target FPGA acceleration boards to obtain a data processing result (S103). According to the method, a physical limitation of host interfaces on the number of FPGA acceleration boards in an FPGA accelerator side may be avoided, thereby improving the data processing efficiency.

The invention relates to the technical field of network congestion control, and provides an RDMA network congestion control method based on the law of conservation of data packets. The method comprises the following steps: a sender sends a data packet to a corresponding receiver at the maximum rate of receiving the data packet, and waits for the ACK of the receiver; when the data flow passes through the port of the switch, the switch judges the current port state and packages the port state mark into a data packet passing through the port at present; after receiving the marked data packet, the receiver extracts the port state, marks the port state, packages the port state in ACK and returns the port state to the sender; after receiving the ACK with the port state mark, the sender calculates the ideal sending rate of the current data packet based on the law of conservation of data packets and transmits the ideal sending rate to a game driving module, and the sender calculates the Nash equilibrium rate among distributed data streams through the game driving module based on the ideal sending rate and transmits the Nash equilibrium rate to the game driving module; and speed regulation is realized by adopting a self-adaptive mechanism.

The invention provides a distributed data flow processing method and device, electronic equipment and a storage medium. The method comprises the following steps: defining a data structure, a data processing requirement and a data flow process for each data flow, and defining a distributed data processing interface in the data flow process; establishing an event response mechanism based on the data flow, the event response mechanism being used for responding to the system boundary event, generating a corresponding processing instruction, and adding the processing instruction into an instruction queue; extracting the processing instruction from the instruction queue according to a preset strategy, and executing the processing instruction according to the data flow; and establishing a process state management mechanism according to a predefined process cooperation relationship, the process state management mechanism being used for monitoring a data process state, and performing collaborative management of the data process according to the data process state. According to the method, data processing and process management can be integrated, cross-system boundary and heterogeneous system collaborative data process processing is supported, and the data circulation efficiency and timeliness are improved.

The present invention provides an intelligent railway inspection method and system WALLET (rall W A Yinte L Ligent insp Ection method and sysTem) based on multi-mode data fusion, comprising the following steps: step one: fusion expression of multi-modal data of inspection scenes; step two: accurately characterizing the characteristics of railway equipment hidden in multi-modal data through multi-modal object feature extraction and description, and realizing multi-modal data analysis; step three: constructing an end-edge-cloud fusion architecture consisting of an intelligent inspection instrument, an inspection edge server and an inspection analysis platform to realize dynamic transmission and calculation; step four: oriented to intelligent railway inspection applications, dynamically constructing a distributed data flow graph to realize parallel acceleration based on data flow.

Methods, systems, and apparatus, including computer programs encoded on computer storage media, for distributing machine learning workloads, e.g., computations for training a neural network or computing an inference using a neural network, across multiple hardware accelerators. One of the systems comprises a plurality of accelerator islands, each hardware accelerator island comprising a respective plurality of hardware devices that include a plurality of hardware accelerators and a corresponding host for each of the plurality of hardware accelerators; and a respective scheduler for each of the accelerator islands that is configured to schedule workloads across the plurality of accelerators and corresponding hosts in the accelerator island, wherein the system is configured to: receive data representing a machine learning workload; and assign a respective portion of the machine learning workload to each of the plurality of accelerator islands for scheduling by the respective scheduler for the accelerator island.

The application discloses a kind of distributed database synchronization method and system and device and medium, it is related to data synchronization technical field, the method includes the following steps: the first parameter information of configuration distributed database;Configuration database management system second parameter information;Configuration satisfies the third parameter information of distributed data stream processing unit operation;The first parameter information, the second parameter information and the third parameter information are parsed and verified, and parsing verification result is obtained;Based on the parsing verification result, create structured query language and generate real-time warehousing code script;The structured query language and the real-time warehousing code script are uploaded to the distributed data stream processing unit release and execute synchronization task and obtain data synchronization result.The application adopts distributed data stream processing unit to execute data synchronization task, improves synchronization efficiency, significantly reduces distributed database full data synchronization time.

Techniques for log storage in distributed data streaming systems are described. A cluster of brokers receive log records from publishers and send log records to subscribers. The log is represented as a group of segments, each segment subdivided into chunks. Metadata describes the log structure. Log records are stored in chunks at least in a remote storage location shared amongst the brokers in the cluster.

The invention provides a data stream adaptive scheduling method, system and device and a medium, and belongs to the technical field of distributed data stream processing. The method comprises the following steps: acquiring NiFi system operation index data, and extracting standardized features by using feature engineering; a machine learning model combination comprising a neural network, a gradient elevator and a reinforcement learning model is constructed for prediction analysis, and an execution time prediction value, a failure probability prediction value and scheduling strategy parameters are generated; dynamically adjusting system configuration parameters based on the prediction result; risk assessment and graded alarm are realized through a multi-stage rule engine; performing real-time state display and configuration management by adopting a visual engine; and finally, an optimization suggestion report is generated through the data analysis platform. According to the invention, adaptive scheduling and intelligent operation and maintenance of the data stream processing system are realized, and the resource utilization efficiency and the operation stability of the system are effectively improved.

The invention relates to the technical field of computer architecture, and provides a computer system and a data processing method, and the system comprises the steps that an on-chip network distributes data traffic according to an address interleaving strategy; the data link module comprises a plurality of data link units, and each data link unit comprises a plurality of data links; the network transmission module comprises a plurality of network transmission layers, the plurality of network transmission layers are in communication connection with the network-on-chip, each network transmission layer is in communication connection with the plurality of data links in one data link unit, and the network transmission layers receive first data sent by the network-on-chip and send the first data to the network-on-chip; distributing the first data to at least one data link in the data link unit according to a preset scheduling strategy; and transmitting second data received from at least one data link in the data link unit to the network-on-chip. According to the invention, through a one-to-many hierarchical link architecture, when a certain link fails, the link can be switched to other links, so that the stability and reliability of the system are enhanced.

The invention provides a distributed vehicle sensor system anomaly detection method based on streaming data. The system is composed of a coordinator Cord and sensor systems of m vehicles. Comprising the following five steps: step 1, a cold start stage; step 2, maintaining a local accumulative Frobenius norm Fi and a global accumulative Frobenius norm; step 3, updating a local matrix sketch D through a matrix sketch algorithm on a distributed sliding window, and returning two matrixes; 4, sending a matrix to the coordinator Cord through a channel, and notifying the coordinator Cord to update a global test window matrix sketch K2; 5, the vector ait is used for updating an index histogram matrix sketch mEH, if the timestamp et of an element e with the minimum timestamp in the updated mEH is smaller than the size W of a time subtraction window at the current time, the site Si sends the element e to the coordinator Cord through a channel, and the coordinator is notified to execute expiration discarding operation on the element. Therefore, the effect of efficiently identifying the abnormal behavior in the distributed data stream of the vehicle sensor system is achieved.

The invention provides a data traffic intelligent scheduling method and device, electronic equipment and a storage medium. The method comprises the following steps: sending a management message to a network interconnection device through a plurality of links, wherein the management message is a message used for detecting interconnection and interworking states of the plurality of links corresponding to the network interconnection device; determining link quality parameters corresponding to the plurality of links according to a received response message fed back by the network interconnection equipment; and scheduling and distributing data traffic corresponding to the plurality of links based on the link quality parameters. According to the method and the device, the link quality of the plurality of links can be detected on the premise of not increasing the network load, so that the possible upper limit of data transmission of each link can be predicted in advance, the data traffic distributed to each link is adjusted, and the data transmission quality is ensured.

A modular and distributed architecture for data stream processing and analysis is described to incorporate data stream analytics capabilities, called Data Stream Analytics Service (DSAS) in the IoT / M2M service layer. Each service layer node hosting DSAS can be split into two independent modules, Stream Forwarder and Stream Analytics Engine. Stream Forwarder is a light weight processing modules that can be responsible for data preprocessing and routing. Stream Analytics Engine is responsible for performing actual analytics on the data stream. Separating the two functionalities enables the service layer nodes to efficiently distribute stream analytics tasks across multiple nodes.

The invention provides a product information pushing method, device, medium and system for a mobile banking, and the method comprises the steps: collecting user transaction behavior data of the mobile banking through employing a data processing pipeline tool under the condition of obtaining user authorization, and caching the user transaction behavior data through employing a distributed message caching queue; summarizing the cached user transaction behavior data by adopting a distributed data flow calculation engine according to a preset dimension rule to obtain a summarizing result, and storing the summarizing result into a column database; and querying the data in the column database by adopting the query statement to obtain user behavior data and product transaction information, processing the user behavior data and the product transaction information to obtain potential product information, and pushing the potential product information. Therefore, the problem that in the prior art, a platform can only query the user funnel diagram to achieve a display function, but cannot push the product information for the customer loss rate in the user funnel is solved.

The application provides an interface parallel distribution method and system for comprehensive monitoring system reconstruction, and the system comprises an interface parallel distribution system for connecting a new platform, an old platform and each specialty; the interface parallel distribution system comprises a data receiving and distribution device, a first address processing device and a second address processing device connected with the data receiving and distribution device through internal interfaces; the first address processing device is connected with the old platform FEP through a first external interface; the second address processing device is connected with the new platform FEP through a second external interface; and the data receiving and distribution device is connected with each specialty through a third external interface. When the system receives and distributes data flow, IP address conversion is performed through the first address processing device and the second address processing device, so that each specialty can be connected with the new system and the old system in parallel, and if a fault is found during operation, the FEP can be quickly and directly connected with the specialty for fast recovery.

A modular and distributed architecture for data stream processing and analysis is described to incorporate data stream analytics capabilities, called Data Stream Analytics Service (DSAS) in the IoT / M2M service layer. Each service layer node hosting DSAS can be split into two independent modules, Stream Forwarder and Stream Analytics Engine. Stream Forwarder is a light weight processing modules that can be responsible for data preprocessing and routing. Stream Analytics Engine is responsible for performing actual analytics on the data stream. Separating the two functionalities enables the service layer nodes to efficiently distribute stream analytics tasks across multiple nodes.

Embodiments of the present application disclose a data processing method and related devices for distributing data flow execution rules in a packet forwarding path. In the present application, a network controller can determine the packet forwarding path of a data flow and determine the segmentation policies of each of multiple network devices to obtain a data flow execution rule. Then, the network controller can publish the data flow execution rule to the multiple network devices so that each of the multiple network devices can obtain the corresponding segmentation policy in the data flow execution rule and perform corresponding service functions on the data flow, achieving unified orchestration of the service functions of each network device in the packet forwarding path according to user requirements, without the need for the user to orchestrate each network device one by one, reducing the complexity of orchestrating service functions and improving the efficiency of orchestrating service functions at the same time.