60 results about "Distributive data processing" patented technology

The invention relates to the technical field of distributed data processing systems, and discloses a data processing system supporting full localization deployment and a compliance guarantee method thereof.The data processing system supporting full localization deployment comprises an audit event collection module used for collecting and storing processing event hash; the event organization module is used for realizing logic time sequence management of events through topological sorting and causal relationship marking; the hash chain construction module is used for constructing a local-to-global double-layer hash chain by utilizing the constructed directed acyclic graph and event hash; the increment synchronization module is used for executing increment synchronization based on Hash tree difference identification; the conflict solving module is used for detecting and solving event conflicts in the distributed environment by applying a version vector technology aiming at data differences found in the incremental synchronization process; according to the method, the lightweight linear hash chain is used for replacing a traditional binary tree structure, high-efficiency and low-cost audit chain maintenance is realized, and the data integrity is guaranteed in an unstable network environment.

A system and method are disclosed for generating a privacy-preserving data corpus for Artificial Intelligence (AI) training. The system comprises a relying partner (RP) computing environment and a trusted, independent identity provider (IdP) computing system. Upon a user authentication request, the IdP provides the RP with only a PII-free, persistent pseudonymous identifier (gUserID) for the user. Any authentication artifacts containing Personally Identifiable Information (PII), such as an OAuth token, are programmatically neutralized by the IdP. This is achieved by generating a transient public-private encryption key pair, immediately destroying the private key, and encrypting the PII-laden artifact with the remaining public key, rendering the PII therein permanently irrecoverable. This enforcement of “PII unknowability” at the RP enables the aggregation of pseudonymous user data, linked by the persistent gUserID, from multiple independent RPs into a rich, cross-organizational corpus for AI training, without ever exposing user PII to the RP.

The invention relates to the technical field of industrial equipment monitoring, in particular to an industrial equipment remote monitoring and early warning method based on data fusion, which comprises a multi-dimensional parameter dynamic correlation analysis module, an edge side distributed data processing unit, a self-adaptive threshold generator and an abnormal state early warning mechanism. Through capturing a multi-parameter nonlinear relationship, distributed data processing, dynamic threshold adjustment and a hierarchical early warning strategy, the monitoring real-time performance and accuracy are significantly improved, and the false alarm rate and the missing report rate are reduced. The method can effectively solve the problems of long monitoring period, high abnormity identification delay and the like, is suitable for large-scale industrial scenes, and has a wide application prospect.

The invention discloses an edge intelligent computing platform device and a data processing method thereof, and belongs to the field of computers, and the device comprises a customized hardware layer and a system software layer. The hardware layer is integrated with a plurality of high-performance GPUs supporting full-precision calculation, a large-capacity memory and an NVMe flash disk to form a strong computing power and storage basis. A system software layer is integrated, such as a Ray distributed framework, and the key improvement of the system is that an NVMe flash disk is configured as overflow storage of a memory, when the memory is insufficient due to processing of mass data or large model reasoning tasks, intermediate data can be automatically overflowed to a high-speed flash memory, and system crash is avoided. Through deep collaborative optimization of hardware and software, the technical problem that a large model cannot be efficiently operated and big data cannot be processed due to resource limitation on an edge side is effectively solved, and full-stack AI reasoning and high-performance distributed data processing at the edge side are realized.

An electronic device, according to one embodiment, may: acquire idle computing power information for an electronic device group; determine a first external electronic device from among the electronic device group on the basis of the idle computing power information when an execution command for a function requiring distributed processing of data is received; transmit a request for first data processing for performing the function to the first external electronic device; and perform the function on the basis of a first result for the first data processing received from the first external electronic device and a second result for second data processing performed by the electronic device for the function.

The technical scheme of the invention discloses a multi-source data fusion and intelligent processing method for vehicle network interaction. According to the invention, a multi-source data fusion and intelligent processing architecture oriented to vehicle network interaction is constructed, and technologies of multi-source heterogeneous data access, standardized modeling, distributed data processing, machine learning prediction, optimization scheduling and the like are comprehensively applied; real-time information interaction and collaborative optimization control among multiple subjects such as an electric vehicle, a charging facility and a power grid dispatching system are realized. Compared with the prior art, the scheme disclosed by the invention has the advantages that the data processing capability, the prediction precision, the response speed and the system adaptability are remarkably improved, and the requirements of vehicle network interaction large-scale application can be better met.

The invention discloses a distributed data processing method and system for an intelligent factory, and relates to the technical field of communication data processing. Determining target equipment according to the original data; based on the centrality index of the target equipment in the factory equipment topological graph, calculating the centrality index to obtain a topological criticality score; analyzing the original data based on a preset rule base to obtain an event severity score; multiplying the topology criticality score by the event severity score to obtain a final priority; inserting the original data into a priority scheduling queue according to the final priority; the storage nodes with the resource loads lower than a preset health threshold value are summarized into a current available node pool; and taking out the target data with the highest final priority from the head of the priority scheduling queue, selecting a storage node with the lowest resource load from the current available node pool as a target storage node, and preferentially distributing the target data to the target storage node for processing. By implementing the technical scheme, priority processing of high-value data is ensured.

The invention discloses a distributed data processing method in a local area network based on a UDP (User Datagram Protocol), which comprises the following steps: a distributed monitoring node sends a broadcast to the local area network, and a computing node in the local area network reports own identity information to the distributed monitoring node after receiving the broadcast; after a user initiates a data processing task, the task is sent to a target computing node in a structured instruction form through a UDP, the structured instruction comprises a task ID input data path algorithm type parameter set and priority, and a distributed monitoring node dynamically allocates the task according to the comprehensive load score of each node; the computing node sends a heartbeat message to the distributed monitoring node according to a heartbeat cycle, wherein the heartbeat message comprises a node identifier, a current running state, a current task execution identifier and node resource occupation summary information; according to the method, lightweight and low-delay distributed task scheduling can be realized. And meanwhile, node fault detection and task automatic recovery are realized, manual intervention is avoided, and safe and reliable execution of the task is guaranteed.

The invention discloses an edge intelligent distributed data processing method, and relates to the field of data processing. The method comprises the following steps: acquiring state data of at least one first edge device; determining at least one data processing task according to the target demand, wherein the data processing task comprises a task type, a processing deadline and a resource demand of the data processing task; based on the task type, the processing deadline and the resource demand of each data processing task, determining the task importance of the corresponding data processing task; according to the importance degree of each task, the computing resource occupancy rate of each first edge device and the network bandwidth occupancy rate, respectively calculating the processing adaptation degree of each first edge device and each data processing task; determining the stability of each first edge device based on the historical failure rate and the operation duration of the first edge device; and determining a second edge device based on the stability and the processing adaptation degree of each first edge device, and processing the data processing task through the second edge device. The data processing efficiency is improved.

The invention discloses an intelligent garden data management method and system based on cloud computing, and relates to the crossing field of cloud computing technologies, and the method comprises the steps: collecting and preprocessing multi-dimensional garden data through a distributed terminal, and adding classification and space-time labels; using a hybrid cloud architecture to realize data security storage and collaborative management, and establishing a classification index; a deep learning model is trained based on historical data, an analysis result is output, a personalized maintenance scheme is generated, and dynamic optimization is performed through real-time feedback; secure data intercommunication between the platform and various terminals is realized by means of a multi-terminal collaborative interaction module, and data security is guaranteed by adopting encrypted transmission and authority control; monitoring and encryption protection are implemented in the whole process, a traceable log is generated and filed regularly, and continuous iterative upgrading is performed in combination with a model effect. The method has the advantages that based on the hybrid cloud architecture and distributed data processing, garden management is promoted to be transformed to data intelligent driving through full-process closed-loop optimization, and the management efficiency and the refinement level are greatly improved.

A system and method generate a dynamic and query adaptive Bloom filter for optimal joining of tables such as large tables in distributed data processing systems. A Bloom filter is dynamically constructed and tuned based on historical query patterns including frequency and selectivity. Metrics are collected from execution of an Adaptive Query Execution (AQE) based execution plan and execution of a Bloom filter-based execution plan generated from the constructed and tuned Bloom filter. A score measuring performance of the execution plans is generated from the metrics of each of the execution plans. The execution plan with a greater score is selected to optimally join the tables. The method implements the system.

This invention discloses an intelligent sales system for electronic components based on demand forecasting and dynamic pricing, belonging to the field of distributed data processing technology. It includes a data acquisition module that collects microservice call chain data through a distributed link tracing system; a semantic compression module that receives high-dimensional tensors output by an external prediction model and compresses them using an autoencoder neural network to generate low-dimensional feature vectors that the pricing engine can directly consume; an early warning module that predicts the probability of future resource conflicts based on historical resource competition index sequences using a time-series prediction model; a scheduling module that constructs a coupling degree matrix of system variables and calculates coupling entropy, and executes resource scheduling strategies according to nonlinear scheduling decision factors; and a feedback optimization module that dynamically updates the early warning threshold by collecting performance data after scheduling. This invention significantly reduces system response latency and resource consumption.

The invention relates to the technical field of edge computing and distributed data processing, and discloses a multi-source data analysis and acquisition system and method for an automobile radar, and the system comprises the steps: dividing a safe reserved water level and a transaction quota water level in a shared memory pool of a radar on-chip system; the evaluation module calculates a perception difference degree between the radar and the visual frame, and triggers capture when the perception difference degree exceeds a threshold value; the coding module generates a standardized inventory index; the arbitration module queries a local bloom filter, executes speculative locking when the local bloom filter is not hit and the number of the to-be-determined memory blocks is lower than the quota water level, or executes fusing; and the settlement module sends the index to the cloud and dispatches the controller to transmit data according to a supply and demand saturation instruction fed back by the cloud. Through end-cloud cooperative verification and memory resource level-to-level management, the problem that vehicle-mounted high-frequency data acquisition and a driving safety function compete for resources is solved, and the effectiveness and safety of data acquisition are improved.

The invention discloses a distributed power operation state monitoring method. The method comprises the following steps that an edge server carries out distributed data processing on secondary equipment data information; the edge server carries out encryption processing on the distributed data processing result and transmits the result to the processing center; the processing center receives the distributed data processing result transmitted by the edge server, and generates a monitoring report based on the distributed data processing result; the distributed processing of the secondary equipment data information is realized through the edge server, the encrypted data information is transmitted to the processing center, the processing center generates the monitoring report based on the distributed data processing result, and the operation state of the power equipment can be monitored in real time.

The application discloses a parallel decision system and method for distributed data processing. The system comprises: an initial logical node generation component, which generates an initial logical node topology; a direct neighborhood marking component, which traverses the initial logical node topology to obtain a direct neighborhood topology of a selected initial logical node and marks the obtained direct neighborhood topology one by one; a sub-neighborhood cost calculation component, which traverses each sub-neighborhood topology in the selected marked direct neighborhood topology, and calculates the cost sum of the sub-neighborhood topology under each random combination of the sub-neighborhood topology and the node cost sum of each initial logical node; a sub-neighborhood parallel decision component, which calculates the cost sum difference between the node cost sum of each initial logical node under all random combinations, and determines the sub-neighborhood parallel scheme of the sub-neighborhood topology; and a parallel decision termination component, which terminates the repeated traversal process under the condition that the parallel scheme of each direct neighborhood obtained by two consecutive traversals and the node cost sum assigned to each initial logical node remain unchanged.

This disclosure relates to quantum computing systems including a quantum processor that implements one or more operations on multiple qubits and a distributed data processing system programmed to perform calculations to determine control sequences that, when applied to the quantum processor, reduces decoherence, decoherence-induced errors, and control-imperfection-induced errors on the one or more operations on the multiple qubits. A user interface device remote from the distributed data processing system receives from a user of the quantum processor characteristics of the quantum processor including operational constraints and / or desired performance, and sends the characteristics to the data processing system to cause the data processing system to perform the calculations to determine the control sequence based on the characteristics.

This application discloses a method and related products for asynchronous order data processing, applicable to the fields of e-commerce and distributed data processing technology. The method includes: retrieving order data and sending the corresponding order messages to a customer queue; processing the order messages in the customer queue based on a first distributed lock, saving the customer information corresponding to the order messages, and sending the order messages to an order queue; processing the order messages in the order queue based on a second distributed lock, saving the order information corresponding to the order messages, and sending the order messages to a notification queue; and processing the order messages in the notification queue and saving the notification information corresponding to the order messages. Thus, by asynchronously processing order data based on three independent queues, business logic decoupling and resource isolation are achieved, mutual blocking is avoided, and the stability of data processing is improved.

A data processing system for determining dependencies and controlling resources in distributed data processing systems, comprising a plurality of mutually distributed processing units, a data acquisition unit for recording state and processing data of the processing units, a control unit for allocating computing resources, and a storage unit for storing state information, characterized in that a hardware-implemented dependency analysis unit is provided, which is configured to capture, model, and represent dependencies between processing operations and data states in real time in a structured dependency representation, and that a resource control unit is provided, which deterministically controls the allocation of computing resources depending on the determined dependencies.

The invention relates to a data processing method and device based on distributed computing, equipment and a medium. The method comprises the following steps: acquiring running state data of each distributed computing node, and generating a node state information set; grouping the computing nodes based on the node state information set, determining representative nodes, and generating a representative node list; constructing a data convergence topological structure based on the representative node list, and generating an initial convergence topology; and monitoring the network performance in the data aggregation process, and dynamically adjusting the initial aggregation topology according to the performance change to generate an optimized aggregation topology. By adopting the method, the convergence topology can be dynamically optimized according to the real-time state of the node, and the efficiency and the stability of distributed data processing are improved.

Embodiments of the present application relate to the technical field of computer, and particularly relate to a distributed data processing method and system, which are used to reduce the amount of data transmitted to the total node and reduce the processing pressure of the total node. Embodiments of the present application include: a sub-node generates record data according to a processing flow of the sub-node on request data; the sub-node is any sub-node of N sub-nodes of a distributed system; the sub-node performs preliminary pairing verification on the record data to obtain preliminary verification success data and preliminary verification unsuccessful data; the sub-node preliminarily aggregates the preliminary verification success data to obtain preliminary aggregation data; and the sub-node sends the preliminary aggregation data and the preliminary verification unsuccessful data to a total node of the distributed system, so that the total node performs re-pairing verification on the preliminary verification unsuccessful data of the N sub-nodes to obtain re-verification success data, and re-aggregates the re-verification success data and the preliminary aggregation data.

The invention relates to the technical field of radio detection, in particular to a radio detection device-oriented distributed data processing system, which comprises a network communication unit, a process management unit, a data processing unit, a terminal management unit and a target display control unit, the network communication unit is also connected with terminal equipment; the data processing unit, the terminal management unit, the target display control unit and the network communication unit are functional units running in independent processes, and the process management unit performs scheduling and state monitoring on the processes of the functional units; data transmission among the functional units is realized through the Redis component; the data processing unit executes flight path fusion processing and simulation training tasks; and a protocol converter module is embedded in the network communication unit, so that bidirectional conversion between a standard protocol in the system and a private communication protocol of terminal equipment is realized, and an equipment control function and a data transmission function are decoupled. Through the system, efficient cooperation of multiple terminals can be realized, and the system has high-precision track fusion capability.

A system and method are disclosed for generating a privacy-preserving data corpus for Artificial Intelligence (AI) training. The system comprises a relying partner (RP) computing environment and a trusted, independent identity provider (IdP) computing system. Upon a user authentication request, the IdP provides the RP with only a PII-free, persistent pseudonymous identifier (gUserID) for the user. Any authentication artifacts containing Personally Identifiable Information (PII), such as an OAuth token, are programmatically neutralized by the IdP. This is achieved by generating a transient public-private encryption key pair, immediately destroying the private key, and encrypting the PII-laden artifact with the remaining public key, rendering the PII therein permanently irrecoverable. This enforcement of “PII unknowability” at the RP enables the aggregation of pseudonymous user data, linked by the persistent gUserID, from multiple independent RPs into a rich, cross-organizational corpus for AI training, without ever exposing user PII to the RP.

The invention relates to the technical field of power monitoring and data acquisition, and particularly discloses an SCADA (Supervisory Control and Data Acquisition) distributed data processing method and system for new energy power generation monitoring. According to the method, an application-cluster-fragmentation three-level distributed system is constructed, and multi-source heterogeneous data are uniformly accessed through a pluggable protocol adaptation layer. And performing application division based on the power generation group and service characteristics to realize service isolation and load balancing. In each application, a fragment cluster is constructed by taking a station as granularity and adopting a consistent Hash algorithm, and transverse expansion and parallel processing are realized. High availability is ensured by establishing a main and standby auxiliary group and log synchronization mechanism, and data consistency during fault switching is ensured by introducing uniqueness verification based on service keys. Finally, the processing results are classified and aggregated, and efficient and safe data access is provided for the outside through the unified data service layer. According to the scheme, the problems of expansibility, reliability and processing efficiency under mass and high-concurrency measuring point data are effectively solved.

This invention provides a distributed group selection SQL automatic generation and calculation method based on multi-level nested JSON parsing, belonging to the field of distributed data processing technology. The method includes: acquiring rules organized in a multi-level nested JSON format and performing parsing processing; converting the processed rules into structured query language text and sending it for execution to output results. The parsing processing includes: identifying cross-entity domain nodes and decomposing them into reduced-dimensional subtrees; calculating and sorting the branch nesting depth, extracting shallow-depth branches as preceding independent fragments, and retaining deep branches as subsequent dependent fragments. The text conversion includes: controlling the asynchronous execution of preceding independent fragments, constructing a feature existence mapping table in memory using the returned data; and applying probe constraints to subsequent fragments using the mapping table to obtain the final optimized text. This invention can discard invalid data rows in advance, reducing the computational resource overhead when multiple tables are joined at the underlying nodes.

The application discloses a kind of distributed data processing method and device.Therein, the distributed data processing method includes: obtaining the information to be identified in the target area, wherein the information to be identified includes passenger trip information and / or driver driving information;In the case where the information to be identified includes the passenger trip information and the driver driving information, the input information to be identified is processed by a local decision model to obtain a target matching result;In the case where the information to be identified includes the driver driving information, the input information to be identified is processed by the local decision model to obtain cruise guidance information.Based on the technical scheme of the embodiment of the application, the accuracy of the determined target matching result and / or cruise guidance information can be improved.

This invention provides a secure distributed data processing system and method for continuous-variable quantum key distribution. The system includes a main data processing computing unit and at least one secondary data processing computing unit. The main data processing computing unit periodically monitors the quantum link state and the state information of the secondary data processing computing units, and dynamically allocates data processing tasks to each secondary data processing computing unit. The secondary data processing computing units execute data processing and send the processing results back to the main data processing computing unit. This invention differentiates processing based on the apparent signal-to-noise ratio and normalized total noise characteristics of the data processing tasks, assigning high-computing-power secondary data processing computing units to high-computing-power secondary data processing computing units and low-computing-power secondary data processing computing units to low-computing-power secondary data processing computing units, thus fully utilizing the computing resources of multiple computing units and improving data processing efficiency.

This invention discloses an environmentally adaptable and secure waterproof telephone system, apparatus, and method, relating to the field of communication equipment technology. It includes a multimodal sensor assembly comprising a humidity sensor, a temperature sensor, and a gas sensor, arranged in an array on the waterproof telephone's outer casing and key internal locations. The humidity sensor collects real-time environmental humidity data, the temperature sensor collects real-time environmental temperature data, and the gas sensor collects real-time environmental gas composition and concentration data. A data processing component, connected to the multimodal sensor assembly, receives the data collected by the sensors. By employing a multimodal sensor fusion system with a distributed data processing architecture, real-time monitoring of environmental changes and dynamic adjustment of protection strategies are achieved, significantly improving the waterproof telephone's safety performance and lifespan in harsh environments.

Systems and techniques may generally be used for streamlining a deployment and scaling of data processing and machine learning workloads on a distributed system. An example method may include receiving, from a user at a user interface, a plugin command, an input including a name key and a query, and a driver code including a code package. The method may include filtering the query based on a scope of the plugin command, obtaining input data by querying an input measure group, and slicing the obtained input data into one or more slices based on the slicing key. The method may include determining a number of containers to be used, and assigning at least one slice for each container. The method may include executing at each container the respective assigned at least one slice, generating an output for each input, and storing each output at a respective output measure group.