203 results about "Distributive data processing" patented technology

The invention relates to the technical field of data transfer and verification, in particular to a business data testing method and system and a storage medium. The method comprises the following steps: acquiring an original service data stream; deploying a supervision submission shadow processing sandbox based on the original business data flow to obtain a data isolation verification environment; configuring an alliance block chain server based on the original service data flow, and constructing a distributed data processing traceability platform; based on the distributed data processing traceability platform, performing data processing proof generation by using the original service data stream to obtain a data processing hash chain; configuring differential privacy protection processor parameters based on the data isolation verification environment to obtain a privacy protection test data generator; a privacy protection test data generator is trained to generate a synthetic test data set. By combining the alliance block chain and the differential privacy technology, a high-credibility, automatic and traceable data testing system is constructed, and the data security, the verification efficiency and the system consistency are remarkably improved.

The invention relates to the technical field of data management and the field of financial science and technology, and discloses a data management method based on cold data migration, which comprises the following steps: extracting metadata in a distributed file system, and importing the metadata into a metadata storage library of a distributed data processing platform. And analyzing file information in the metadata storage library through an offline analysis task, identifying cold data, and storing file path information of the cold data in a database. And according to the identified cold data, generating a data migration plan comprising a migration target position, migration priority ranking and migration time arrangement. And migrating the cold data from the original storage position to the target far-end storage, and updating a table partition path in the metadata storage library to point to a new storage position. And after migration is completed, deleting the cold data in the original storage position. According to the method, the cold data and the computing resources are effectively separated, the utilization efficiency of the storage resources is optimized, the overall resource consumption of the system is reduced, and meanwhile the expansibility and the performance of the system are improved.

The invention discloses a data analysis method, system and device based on distributed communication and a medium, and relates to the technical field of data processing, and the method comprises the steps: monitoring a node state in real time based on a distributed communication network, obtaining a node resource state data set, building a node performance evaluation model through a machine learning algorithm, and obtaining a node performance evaluation result; computing resources of the data analysis tasks are dynamically allocated to optimize resource allocation, a task allocation strategy is designed according to requirements and node performance of the data analysis tasks, and a distributed data processing framework is established; the method comprises the following steps: performing distributed decomposition and executing a calculation data analysis task, aggregating and collecting distributed calculation results, and dynamically adjusting and optimizing resource task allocation to obtain a final data analysis result, so that a node state can be monitored in real time, and a resource allocation scheme can be intelligently adjusted; according to the advantages of dynamically optimizing task allocation according to the communication condition and the computing power of the node, the problems of insufficient dynamic resource management and poor task scheduling flexibility in the prior art are solved.

The disclosed system and method automate the transpilation of query source code to a distributed data processing system. The process involves instantiating transpiler libraries and source code, executing the source code, and transpiling the source code into a transpiled code using the transpiler libraries. A machine learning algorithm is utilized to control the operation of the transpilation process wherever different transpilation options are available as an automated mechanism for adapting transpilation based on tracked downstream operational characteristics. The system is configured for automatic source code generation in the target language as an output, which can then be deployed to a distributed data processing or query system. Variants are described using a combination of test and production environments for initial testing and tuning of the machine learning model architecture until a performance threshold is met. A same code segment may be transpiled periodically for continual automatic optimization.

The invention discloses a remote centralized control distributed data processing and analysis method based on a JMS, and the method comprises the steps: achieving the message routing and resource scheduling through the real-time monitoring of the state of a system, reinforcement of learning decision, deep learning optimization and multi-dimensional message distribution; dynamically adjusting message flow rate and queue length based on real-time network conditions, node load and flow fluctuation; through edge node intelligent task scheduling, distributed collaboration of calculation and storage is realized; adaptive compression algorithm selection is carried out by using deep learning, and the consumption of message transmission bandwidth is reduced; non-tampering storage of messages is realized through a block chain technology; based on a container arrangement technology, dynamic expansion and contraction of computing resources are realized; and dynamic resource scheduling and optimization decision are realized by utilizing reinforcement learning and multi-dimensional monitoring data. According to the invention, the bottleneck problem of the traditional message transmission system is overcome, and the system can efficiently and reliably process mass data in a high-concurrency and high-throughput scene.

The invention discloses an ODPS-based customer service data aggregation method and server, and relates to the field of electric digital data processing, and the method comprises the steps: carrying out the field classification of collected source data, and obtaining a plurality of source data sets; loading the source data sets to an ODPS platform, and performing first aggregation processing on the source data in the plurality of source data sets by using the ODPS platform to obtain a plurality of field wide tables; pulling the field wide table to a DRDS platform, and performing distributed data processing on the field wide table by using the DRDS platform to obtain a plurality of pieces of distributed data; determining a target data feature corresponding to a target report in a report request instruction according to the report request instruction sent by the user terminal; and extracting multiple pieces of target data corresponding to the target data features in the multiple pieces of distributed data, and performing second aggregation processing on the target data to generate a target report. By implementing the method, the data processing efficiency can be improved when the multi-azimuth data display of the customer service staff is carried out.

The invention relates to the technical field of electric meter online monitoring, in particular to an electric energy data intelligent acquisition terminal based on federal learning. The data acquisition and preprocessing module is used for acquiring electric energy data in a power grid in real time and carrying out de-noising and normalization preprocessing on the electric energy data; the model training module based on federated learning is used for training a neural network model for the preprocessed electric energy data by using local data so as to obtain a local model; the homomorphic encryption algorithm application module is used for encrypting and updating data and model parameters in a local data training process by adopting a homomorphic encryption algorithm; and the global model updating module is used for sending the neural network model subjected to multiple times of local data training and encryption updating to a central server, and optimizing and updating the neural network model through a weighted federated average aggregation algorithm. According to the invention, distributed data processing and machine learning technologies are combined to realize intelligent data analysis of electric power data, so that the intelligent level of electric power monitoring and fault early warning is improved.

The invention relates to the field of database reading, and particularly discloses an efficient database reading method based on multi-level cache optimization and dynamic index fragment.The efficient database reading method comprises the steps that high-frequency query data is accurately recognized through a sliding window algorithm, a dynamic cache loading and preheating strategy is implemented, query delay is effectively shortened, and the database burden is relieved; intelligent distribution of index fragments is realized by applying a consistent Hash algorithm, and the query efficiency is greatly improved in cooperation with distributed query routing and transaction processing optimization; the query load balancing ensures stable and efficient operation of the system in a high-concurrency scene by means of weighted polling and a minimum connection strategy in combination with real-time monitoring and an elastic capacity expansion and contraction mechanism. According to the method, the database access efficiency and the dynamic adaptive capacity are remarkably improved, and the method is particularly suitable for a large-scale distributed data processing environment.

A distributed data processing system and a distributed data processing method are provided. The distributed data processing system includes a computing device and at least one additional computing device. In the method, the computing device receives a dataset to be processed, selects at least one computing device capable of executing a first machine learning model on the dataset from the computing device and the additional computing device to process the dataset and generate a first prediction result, selects at least one computing device capable of executing each of multiple machine learning models on the dataset from the computing device and the additional computing device to process the dataset and generate at least a second prediction result and a third prediction result, selects a second machine learning model from the multiple machine learning models based on a comparison between the first prediction result and at least the second prediction result and the third prediction result, and instructs the at least one computing device for executing the second machine learning model to process the dataset.

The invention is applicable to the field of equipment data collection and processing, and provides a field video equipment control method which comprises the following steps: acquiring a network identifier of video equipment; according to the network identifier, mutually connecting the video devices with the same network identifier, establishing a data transmission network, and obtaining a theoretical monitoring range and an actual observable area of the video devices with the same network identifier to obtain a monitorable area; generating a video equipment control instruction according to the monitorable area; according to the monitorable area, splicing images in the monitorable area to obtain a spliced image; according to a video equipment control instruction and a spliced image, video equipment is deployed in a unified manner, through unified task deployment and distributed data processing, resources of a plurality of cameras are fully utilized, species information at different angles and distances is captured, more comprehensive observation data is provided, the monitoring efficiency is improved, and the coverage range is expanded. The system is suitable for large-range species monitoring requirements, and can timely deploy tasks according to feedback data, change a shooting angle or increase monitoring frequency and the like.

The invention relates to the technical field of distributed data processing systems, and discloses a data processing system supporting full localization deployment and a compliance guarantee method thereof.The data processing system supporting full localization deployment comprises an audit event collection module used for collecting and storing processing event hash; the event organization module is used for realizing logic time sequence management of events through topological sorting and causal relationship marking; the hash chain construction module is used for constructing a local-to-global double-layer hash chain by utilizing the constructed directed acyclic graph and event hash; the increment synchronization module is used for executing increment synchronization based on Hash tree difference identification; the conflict solving module is used for detecting and solving event conflicts in the distributed environment by applying a version vector technology aiming at data differences found in the incremental synchronization process; according to the method, the lightweight linear hash chain is used for replacing a traditional binary tree structure, high-efficiency and low-cost audit chain maintenance is realized, and the data integrity is guaranteed in an unstable network environment.

The invention relates to the technical field of artificial intelligence, in particular to a data processing method based on artificial intelligence. Comprising the following steps: a) a data preprocessing module: performing primary processing on input data, including but not limited to filling vacant values, identifying and processing abnormal values, removing repeated data and unifying data formats, so as to ensure the quality and consistency of the data; b) a feature selection and dimension reduction module: screening out key features from original data and reducing data dimensions by using an automatic feature selection method; c) a data security and privacy protection module; d) a distributed data processing and storage module; e) an intelligent model training and evaluation module; and f) a feedback and optimization module. The invention provides the artificial intelligence-based data processing method which can improve the data preprocessing capability, optimize the feature selection and dimension reduction effect, enhance the data security and privacy protection capability, improve the distributed data processing and storage efficiency, improve the intelligent model training and evaluation performance and perfect the feedback and optimization mechanism.

A distributed data processing system includes a first data processing system and a second data processing system. The first data processing system includes a first housing, a first data processor, and a first optical module that is configured to convert output electrical signals from the first data processor to output optical signals that are provided to a first optical fiber cable. The second data processing system includes a second housing, a second data processor, and a second optical module that is configured to convert output electrical signals from the second data processor to output optical signals that are provided to a second optical fiber cable. An optical power supply includes at least one laser that is configured to provide a first light source to the first optical module through a first optical link and to provide a second light source to the second optical module through a second optical link.

The invention discloses a GNSS network RTK base station distributed data processing method and device, belongs to the technical field of satellite navigation and positioning, and is used for solving key problems of base station networking, baseline solution, error correction calculation, model establishment and the like in a large GNSS network RTK system. The method is especially suitable for efficient calculation, task allocation and dynamic load balancing processing of a large-scale base station network. The method comprises the following steps: 1) dynamically networking a core server, determining a baseline to be solved, generating a baseline solving task and establishing an error model; (2) a plurality of data processing servers solve all baselines in parallel, and baseline error correction numbers are extracted; (3) each base station calculates relative corrections of other stations relative to the base station according to the related baseline corrections, and establishes an error model, and (4) dynamically allocates a baseline resolving task and an error modeling task. Through multi-thread and distributed parallel computing, the task processing efficiency and the resource utilization rate of a large-scale base station network are remarkably improved.

The invention discloses a distributed aggregation method based on a one-way isolation network, and relates to the technical field of distributed data processing. Comprising the following steps: step 1, constructing a one-way isolation network: establishing a one-way transmission channel, and allowing an aggregation request to be transmitted to a data storage node in a one-way manner, step 2, sending an encryption aggregation request: before sending the aggregation request, encrypting the aggregation request by using an encryption algorithm, 3, local aggregation of the data storage node: receiving the encrypted aggregation request through the data storage node, decrypting the aggregation request and executing the aggregation operation of local data, and 4, obtaining an encrypted transmission aggregation result: after the data storage node completes the local aggregation, encrypting the aggregation result again, and sending the encrypted transmission aggregation result to the data storage node; and 5, carrying out global aggregation: collecting encrypted aggregation results of all the data storage nodes, carrying out final aggregation operation based on the aggregation results to obtain a global aggregation result, and decrypting and outputting the global aggregation result as required.

The invention relates to the technical field of data processing, and particularly discloses a distributed data processing method which comprises the steps of establishing a distributed node network, monitoring node states, analyzing node operation loads, judging node load states, judging data processing priorities, distributing data processing tasks and optimizing distributed processing nodes. A distributed node network is established, an area covered by a data processing system is divided into a plurality of processing areas, and running state parameters of nodes on servers in the processing areas are monitored in real time and comprise the CPU utilization rate, the memory utilization rate, the network bandwidth occupancy rate and the disk space utilization rate. According to the method, the fault or performance problem of the node can be found and processed in time, and meanwhile, through a refined task allocation and node optimization strategy, each server resource in the distributed processing area can be efficiently utilized to a great extent, so that the priority processing of a core task and the stable operation of the overall system are ensured.

This disclosure relates to quantum computing systems including a quantum processor that implements one or more operations on multiple qubits and a distributed data processing system programmed to perform calculations to determine control sequences that, when applied to the quantum processor, reduces decoherence, decoherence-induced errors, and control-imperfection-induced errors on the one or more operations on the multiple qubits. A user interface device remote from the distributed data processing system receives from a user of the quantum processor characteristics of the quantum processor including operational constraints and / or desired performance, and sends the characteristics to the data processing system to cause the data processing system to perform the calculations to determine the control sequence based on the characteristics.

The invention provides a local knowledge base automatic construction system based on multi-source acquisition and distributed computing, and relates to the technical field of computer systems. The system comprises a multi-source data collector and a distributed data processing and analyzing module, wherein the multi-source data collector realizes access, data collection, data transmission and incremental updating of multi-source data, and generates an incremental change record; and the distributed data processing and analyzing module comprises a plurality of distributed nodes and is used for carrying out distributed processing and analysis on data files according to incremental change records on the data collected by the multi-source data collector and establishing a local knowledge base. According to the system, automatic construction of the local knowledge base can be realized, the knowledge base construction efficiency is improved, the knowledge base construction period is shortened, the knowledge data freshness is ensured, and the data quality is improved.

A system and method are disclosed for generating a privacy-preserving data corpus for Artificial Intelligence (AI) training. The system comprises a relying partner (RP) computing environment and a trusted, independent identity provider (IdP) computing system. Upon a user authentication request, the IdP provides the RP with only a PII-free, persistent pseudonymous identifier (gUserID) for the user. Any authentication artifacts containing Personally Identifiable Information (PII), such as an OAuth token, are programmatically neutralized by the IdP. This is achieved by generating a transient public-private encryption key pair, immediately destroying the private key, and encrypting the PII-laden artifact with the remaining public key, rendering the PII therein permanently irrecoverable. This enforcement of “PII unknowability” at the RP enables the aggregation of pseudonymous user data, linked by the persistent gUserID, from multiple independent RPs into a rich, cross-organizational corpus for AI training, without ever exposing user PII to the RP.

The invention provides a predictive maintenance model management method for a communication information system, belongs to the field of artificial intelligence, and realizes predictive maintenance of equipment and network nodes by utilizing real-time data in a communication network and combining a PPO algorithm. The method is suitable for a dynamic environment of a large-scale communication network, integrates multi-source data through a data sharing platform, and optimizes the learning and prediction capabilities of the model. Comprising distributed data acquisition, dynamic model updating, maintenance strategy generation, model optimization and real-time data synchronization. The problem that traditional centralized model management cannot adapt to communication network complexity and dynamic change is solved. Through distributed data processing and adaptive optimization of the PPO algorithm, the prediction precision and response speed of the system are improved, and the operation efficiency and maintenance capability of the communication information system are significantly improved.

The invention provides a data row and column permission control and desensitization method based on a Spark engine, and relates to the technical field of data processing, and the method comprises the steps: receiving a user data processing request, obtaining a data access permission rule of a user from a permission management server, analyzing the permission rule into a Spark SQL syntax tree, and injecting a row-level filtering condition and a column-level field for filtering; and constructing a Spark distributed processing task including data scanning, filtering, projection and desensitization nodes, and submitting and executing the task. According to the method, refined data access control can be realized in the distributed data processing process, the data security is ensured, the data processing efficiency is improved, and the method is suitable for a large-scale data processing scene.

The invention provides a cement industry internet-of-things intelligent logistics system, which comprises the following steps of: according to standardized sales order data and delivery order data, judging the relevance between the sales order data and the delivery order data by adopting a decision tree algorithm, establishing a mapping relationship, and realizing information connection between sales and logistics links; vehicle positioning data and loading and unloading confirmation information are obtained, a real-time tracking record of the whole logistics process is generated through data fusion processing, and a visual track is generated in combination with map information; for mass logistics data, a distributed data processing platform is established, the execution efficiency of logistics tasks is analyzed by adopting a clustering algorithm, abnormal tasks are identified, and intelligent early warning signals are generated; and according to the intelligent early warning signal, the priority of the logistics task and the vehicle scheduling strategy are adjusted, the distribution of logistics resources is optimized, and the overall logistics efficiency is improved.

The invention relates to the technical field of information technology, in particular to a decentration-based peer-to-peer group social network system and a building method thereof, and the method comprises the following steps: setting network nodes, each network node having local data collection, primary processing and decision-making capabilities, the network nodes have self-organization and dynamic participation capabilities, the system adopts social communication protocol design, realizes data transmission among the network nodes through a wireless communication technology, supports flexible network node arrangement and communication coverage, adopts distributed data processing and storage, and is high in reliability and high in reliability. And the network nodes share data and carry out cooperative processing. Through autonomous interaction and data sharing among network nodes, the system can realize efficient resource utilization, real-time environment perception and dynamic decision adjustment, improve the resource allocation efficiency, realize real-time data acquisition and optimize dynamic environment response, and significantly improve the management efficiency and the user experience.

The invention relates to the technical field of e-commerce, in particular to a distributed data processing system based on big data, and the system comprises an internal data collection end which is used for collecting user transaction data, commodity data and behavior logs of an e-commerce platform in real time; the external data acquisition end is used for acquiring consumer credit data and social network behavior data of the third-party platform through an API (Application Program Interface); the data classification module is used for carrying out initial classification on the collected data by adopting a multi-level label system and generating classification labels with weight coefficients; the data screening module is used for dynamically filtering the classified data according to a preset threshold condition, and the threshold condition comprises data timeliness, integrity and confidence; the data error correction module is used for correcting the labels of the misclassification data set and feeding back the corrected data to the input end of the classification module; and the plurality of distributed data analysis centers generate 16 user portrait dimensions from A1 to D4, and the analysis centers execute data modeling of the corresponding dimensions in parallel.

The invention relates to the technical field of industrial equipment monitoring, in particular to an industrial equipment remote monitoring and early warning method based on data fusion, which comprises a multi-dimensional parameter dynamic correlation analysis module, an edge side distributed data processing unit, a self-adaptive threshold generator and an abnormal state early warning mechanism. Through capturing a multi-parameter nonlinear relationship, distributed data processing, dynamic threshold adjustment and a hierarchical early warning strategy, the monitoring real-time performance and accuracy are significantly improved, and the false alarm rate and the missing report rate are reduced. The method can effectively solve the problems of long monitoring period, high abnormity identification delay and the like, is suitable for large-scale industrial scenes, and has a wide application prospect.

The invention discloses an edge intelligent computing platform device and a data processing method thereof, and belongs to the field of computers, and the device comprises a customized hardware layer and a system software layer. The hardware layer is integrated with a plurality of high-performance GPUs supporting full-precision calculation, a large-capacity memory and an NVMe flash disk to form a strong computing power and storage basis. A system software layer is integrated, such as a Ray distributed framework, and the key improvement of the system is that an NVMe flash disk is configured as overflow storage of a memory, when the memory is insufficient due to processing of mass data or large model reasoning tasks, intermediate data can be automatically overflowed to a high-speed flash memory, and system crash is avoided. Through deep collaborative optimization of hardware and software, the technical problem that a large model cannot be efficiently operated and big data cannot be processed due to resource limitation on an edge side is effectively solved, and full-stack AI reasoning and high-performance distributed data processing at the edge side are realized.

Systems and methods are provide for automatically constructing data lineage representations for distributed data processing pipelines. These data lineage representations (which are constructed and stored in a central repository shared by the multiple data processing sites) can be used to among other things, clone the distributed data processing pipeline for quality assurance or debugging purposes. Examples of the presently disclosed technology are able to construct data lineage representations for distributed data processing pipelines by (1) generating a hash content value for universally identifying each data artifact of the distributed data processing pipeline across the multiple processing stages / processing sites of the distributed data processing pipeline; and (2) creating an data processing pipeline abstraction hierarchy for associating each data artifact to input and output events for given executions of given data processing stages (performed by the multiple data processing sites).

The present invention provides a data row and column permission control and desensitization method based on the Spark engine, and relates to the field of data processing technology, including receiving a user data processing request, obtaining a user's data access permission rule from a permission management server, parsing the permission rule into a Spark SQL syntax tree and injecting row-level filtering conditions and column-level field filtering, and constructing a Spark distributed processing task including data scanning, filtering, projection and desensitization nodes and submitting it for execution. The present invention can realize refined data access control in the distributed data processing process, ensure data security, improve data processing efficiency, and is suitable for large-scale data processing scenarios.

The embodiments of the present application provide a distributed data processing method and apparatus, and an electronic device. The distributed data processing method comprises: acquiring a data processing model of an application program; according to a preset subgraph partitioning rule, partitioning the data processing model to obtain a plurality of subgraphs; and on the basis of a preset scheduling sequence configuration file, respectively sending a corresponding task to each server among a plurality of servers, wherein the task comprises some subgraphs among the plurality of subgraphs, so that each server performs data processing on the basis of the received subgraph to obtain a processing result; for each server among the plurality of servers, receiving a processing result sent by the server; and merging the plurality of processing results to obtain a target result of the data processing model.

The invention relates to the technical field of distributed data processing, and discloses a distributed financial data processing system. The system comprises a multi-source data acquisition module, a heterogeneous data integration module, a distributed calculation verification module, a service flow modeling module, a resource scheduling module and the like. The multi-source data acquisition module captures multi-dimensional financial transaction data in real time by using a distributed edge computing node, and distributes an acquisition path through a dynamic load balancing algorithm; the heterogeneous data integration module analyzes the cross-platform data format and generates a standardized data stream; the distributed calculation verification module constructs a credible verification model based on a hybrid consensus mechanism; the business flow modeling module predicts a risk conduction path; and the resource scheduling module optimizes resource allocation. The system effectively solves the problems of a traditional financial data processing system in the aspects of data acquisition, integration, calculation verification and the like, improves the data processing efficiency, the risk prediction capability and the resource utilization efficiency, and guarantees the safety and compliance of financial transactions.