Patents
Literature
Patsnap Eureka AI that helps you search prior art, draft patents, and assess FTO risks, powered by patent and scientific literature data.

1312 results about "Stream processing" patented technology

Stream processing is a computer programming paradigm, equivalent to dataflow programming, event stream processing, and reactive programming, that allows some applications to more easily exploit a limited form of parallel processing. Such applications can use multiple computational units, such as the floating point unit on a graphics processing unit or field-programmable gate arrays (FPGAs), without explicitly managing allocation, synchronization, or communication among those units.

Multi-source heterogeneous data fusion and processing method based on big data

The present invention relates to the technical field of big data processing, and discloses a multi-source heterogeneous data fusion and processing method based on big data, including the following steps: data preprocessing, cleaning, complementing and feature extraction of multi-source heterogeneous data, providing standardized and high-quality input for subsequent processing; dynamic knowledge graph modeling, constructing a dynamic knowledge graph based on the semantic relationship of multi-source data, and generating a unified semantic representation; cross-modal alignment and deep fusion, semantically aligning multi-modal data, and generating a unified comprehensive representation through deep fusion of features; real-time stream processing, dynamically updating and processing real-time updated stream data to achieve real-time fusion of data; adaptive optimization, dynamically adjusting fusion strategy parameters, and improving the adaptability and robustness of the fusion strategy. The present invention realizes efficient fusion, real-time processing and semantic consistency expression of multi-source heterogeneous data through dynamic knowledge graph modeling, cross-modal alignment, real-time stream processing and adaptive optimization.
Owner:CHENGDU BIG DATA GRP CO LTD

Experimental calibration method based on multi-sensor signal fusion processing

The invention discloses an experimental calibration method based on multi-sensor signal fusion processing, particularly relates to the field of data analysis, and comprises the steps of a stepped confrontation disturbance protocol, data stream processing and real-time analysis, and multi-stage logic judgment and calibration execution. A stepped confrontation disturbance protocol is created for the first time, potential errors of the sensor are excited in real time and corrected in a closed loop mode under the condition that system operation is not interrupted, and a traditional shutdown calibration mode is thoroughly replaced; a multi-stage disturbance defense system is constructed, and composite interferences such as spectrum aliasing, overload distortion and signal-to-noise ratio attenuation caused by electromechanical thermal multi-physical field strong coupling are effectively suppressed; laser space calibration and dynamic rigidity modulation technologies are fused, and space drift errors caused by deformation of the mounting base and mechanical loosening are accurately diagnosed and compensated. Finally, long-term precision stability and data fusion reliability of the multi-source sensor under complex working conditions are achieved.
Owner:LONGYAN UNIV

Network line intelligent operation and maintenance monitoring management system and method

The invention discloses a network line intelligent operation and maintenance monitoring management system and method. The method comprises the steps that an intelligent network monitoring and optimizing system regularly collects network equipment information through a protocol, and a monitoring agent collects server and machine room environment data; processing data in real time by adopting distributed storage and a stream processing framework; predicting a network fault by adopting threshold monitoring and a machine learning model, and identifying a bottleneck and scheduling resources based on flow analysis; in combination with deep packet inspection and an AI algorithm, network traffic is monitored in real time, security threats are identified, and IPS system protection is deployed; a repair operation is automatically triggered through a self-healing strategy, and backup resources are started; the AI engine analyzes historical and real-time data, generates network topology and link state visual content, and automatically generates reports, optimization and decision support; according to the intelligent operation and maintenance monitoring management system and method for the network line, a fault tree analysis and decision support system is introduced, and a plurality of fault reasons are intelligently analyzed and repair measures are automatically executed.
Owner:GUANGZHOU YUNJILIDUN INFORMATION TECHNOLOGY CO LTD

Anti-interference control method applied to radar

The invention relates to the technical field of radar control, in particular to an anti-interference control method applied to a radar, and aims to solve the problem that an existing radar anti-interference technology is insufficient in anti-interference capability in a complex electromagnetic environment. According to the method, multi-dimensional input data of a radar system are synchronously collected, interference feature extraction is performed on multi-dimensional data streams by using a dual-channel deep network, and interference fingerprint vectors are output to a dynamic feature library in real time. And based on the interference feature priority list, performing cooperative processing, including spatial filtering, frequency domain notch, polarization projection and residual extraction and sparse reconstruction of a target signal. And dynamically adjusting the processing parameters according to the real-time evaluation result. And data distribution processing is carried out on the suppressing interference and the deception interference, and final target trajectory information is output through result fusion. Parallel processing is also realized on the FPGA chip, and the processing speed is improved. On the whole, the anti-interference capability, the stability and the real-time performance of the radar system in a complex electromagnetic environment are improved.
Owner:ANHUI YAOFENG RADAR TECH CO LTD

Distributed real-time data stream processing system and method

The invention relates to the technical field of real-time data processing, in particular to a distributed real-time data stream processing system and method, and the system comprises a state mapping modeling module, a load dynamic calculation module, a node task adjustment module and a resource scheduling execution module. According to the method, the dynamic identification and classification of the flow state are realized by analyzing the data flow change sequence and identifying the frequency and trend of state switching, the monitoring and understanding capability of real-time data is improved, the node resource pressure is accurately quantified, the predictability of load balancing is enhanced in the resource occupation and load range analysis, and the load balancing efficiency is improved. A scientific basis is provided for resource scheduling, task priority ranking and dynamic task adjustment are combined, the node computing resource utilization rate and the task processing response speed are optimized, flexible configuration of internal storage and computing resources is achieved, peak load pressure is relieved, and data processing efficiency and system stability are improved. And the adaptability and the reliability of processing large-scale data streams by the system are improved.
Owner:廊坊市讯云数据科技有限公司

Wide-area landslide rapid identification method based on interpretable intelligent algorithm

The invention discloses a wide-area landslide rapid identification method based on an interpretable intelligent algorithm, and relates to the field of remote sensing science and technology, and the method comprises the steps: building a dual-channel feature extraction architecture through multi-source spatio-temporal data fusion and knowledge graph dynamic weighting: capturing image local textures through a lightweight CNN, modeling geological spatial correlation through a graph convolutional network, and carrying out the recognition of the landslide. Combining the SHAP value and causal reasoning to generate an interpretable contribution degree thermodynamic diagram and a rule chain; a knowledge graph bidirectional verification system is introduced, spatial logic contradictions are verified by using prior rules, and co-evolution of a model and a rule base is triggered based on misjudgment samples; outputting a multi-dimensional credibility report, quantifying uncertainty by Monte Carlo Dropout, and customizing interpretation granularity according to roles; a terrain-adaptive block-stream processing architecture is adopted, and edge lightweight deployment and federated learning are combined, so that wide-area real-time early warning and model dynamic updating are realized. According to the scheme, the limitation of a traditional black box model is broken through, and a disaster prevention closed loop with physical driving, transparent decision and second-level response is formed.
Owner:CHENGDU UNIV

Operation business risk analysis method and device, equipment and storage medium

The invention provides an operation business risk analysis method, device and equipment and a storage medium, and the method comprises the steps: carrying out the multi-modal preprocessing of multi-source heterogeneous data, generating a structured risk feature data set, carrying out the staged training and dynamic optimization of a preset multi-branch deep neural network model, and obtaining a structured risk feature data set; and obtaining a trained risk analysis model. And carrying out distributed deployment and streaming processing configuration on the model, and constructing a real-time risk inference system. And inputting real-time operation business data into the system, and generating a risk analysis result and corresponding feature importance distribution. And through a multi-model integration framework, carrying out comprehensive assessment and interpretability analysis on a risk analysis result and feature importance distribution to form a comprehensive risk assessment report. According to the method, through multi-modal preprocessing of multi-source heterogeneous data, risk features of different data sources are fully mined and fused, and the problem of inaccurate analysis caused by single data and unprocessed heterogeneity in a traditional method is solved.
Owner:SHENZHEN DOLPHIN INTERNET CO LTD

Urban greening system based on big data and optimization method thereof

The invention discloses an urban greening system based on big data and an optimization method thereof, relates to the technical field of artificial intelligence, and solves the problems of difficulty in heterogeneous data integration, insufficient real-time analysis capability of dynamic data and low management efficiency in an existing urban greening system. Multi-source heterogeneous data are collected through an API interface and Internet of Things communication, and are cleaned, aligned, converted and fused through a distributed real-time data stream processing platform; image recognition and a machine learning algorithm are used for monitoring diseases and insect pests in real time, and a growth prediction model is trained in combination with an LSTM network, so that the quick response capability of the system to environmental changes is enhanced; according to the method, target weights are dynamically adjusted and optimized through the deep Q network, a multi-target optimization model is constructed in combination with mixed integer linear programming, context awareness and a genetic algorithm, intelligent optimization of irrigation, fertilization, pest control and greening layout is achieved, and the management efficiency and the resource utilization rate of an urban greening system are remarkably improved.
Owner:KAIFENG IND -CITY INTEGRATION CONSTRUCTION ENGINEERING CO LTD

Geological disaster early warning method and accurate early warning system based on multi-source data fusion

The invention discloses a geological disaster early warning method and a precise early warning system based on multi-source data fusion, and relates to the technical field of geological disaster early warning. According to the method, multi-source heterogeneous data such as remote sensing, meteorological and geological monitoring are fused, a standardized protocol is utilized to unify a data format and temporal-spatial resolution, a standardized data set is formed, key features are extracted by adopting principal component analysis and a recursive feature elimination algorithm, and a long-short-term memory network and a convolutional neural network model are combined, so that the real-time performance of the system is improved. According to the method, the disaster risk is accurately predicted, the space risk distribution diagram is generated, in addition, through application of the real-time stream processing framework and the self-adaptive learning algorithm, rapid distribution of early warning signals and dynamic optimization of model parameters are achieved, the accuracy and timeliness of an early warning system are remarkably improved, and the geological disaster risk is effectively reduced.
Owner:SICHUAN ZHIXIN RENYI TECHNOLOGY SERVICE CO LTD

Multi-source heterogeneous data fusion engineering project supervision real-time visual decision-making platform

The invention discloses an engineering project supervision real-time visual decision-making platform based on multi-source heterogeneous data fusion, and relates to the technical field of engineering supervision. The visual decision platform is in communication connection with a data acquisition module, a project risk feature module, a risk identification model construction module, a real-time data stream processing module, a risk early warning module and a decision support module. According to the invention, the risk in the engineering project is accurately identified in combination with historical multi-source heterogeneous data and project risk characteristics, and meanwhile, the real-time data stream processing module can analyze data in real time, identify potential risks and problems and generate early warning information, so that the accuracy and timeliness of risk identification are greatly improved, and the risk identification efficiency is improved. According to the invention, powerful decision support is provided for supervisors, and through the preset early warning level and the early warning signal, managers can rapidly understand the risk condition and take corresponding measures, thereby effectively reducing the project risk.
Owner:NINGXIA HUIYUAN PROJECT MANAGEMENT CO LTD

Intelligent data production method and system based on graph neural network and adaptive learning

The invention provides an intelligent data production method and system based on a graph neural network and adaptive learning, and the method comprises the steps: collecting a text, a time sequence, an image and sensor data in real time, and converting unstructured data into a structured JSON format through a format analysis tool; extracting text entity features based on a large language model, modeling a cross-modal entity association relationship in combination with a graph neural network, and fusing multi-source heterogeneous data through a dynamic knowledge graph; the detection threshold is dynamically optimized by adopting a reinforcement learning algorithm, and the false alarm rate is reduced in combination with a double-track verification mechanism; and predicting a data trend by using the space-time diagram convolutional network, and outputting an interpretability analysis report through the generative large model. According to the invention, a multi-protocol adaptation engine and a distributed stream processing architecture are adopted, and a protocol analysis layer automatically identifies a plurality of industrial protocols, so that the manual adaptation time is reduced; and real-time synchronization of production line-level data is realized through edge node parallel acquisition and compression transmission.
Owner:AACAT TECHNOLOGY LTD

Group behavior analysis method for multi-modal information fusion and dynamic updating

The invention discloses a multi-modal information fusion and dynamic updating group behavior analysis method. The method comprises the steps of information integration and dynamic updating, multi-modal information obtaining and information preprocessing, feature extraction, feature representation conversion, alignment and fusion, and multi-modal information real-time processing and dynamic updating through a stream processing framework. A self-learning mechanism and a self-optimization mechanism are introduced; group portrait generation: extracting population statistical characteristics, geographic position distribution, behavior and emotional psychological characteristics, social relationships, social media and economic characteristics from the multi-modal data to generate a group portrait; analyzing group behaviors; and outputting visualization, and outputting group behavior prediction results in a visualized manner, including a behavior trend graph, a group clustering graph, an association rule graph and an abnormal behavior detection graph. According to the method, multi-modal data from various sources and forms are effectively integrated and analyzed, self-learning and self-optimization mechanisms are introduced, and finally group behavior analysis is performed.
Owner:GUANGZHOU UNIVERSITY

Cache information processing method and equipment for port platform, and medium

The invention provides a cache information processing method and device of a port platform and a medium, and belongs to the technical field of port distributed information processing. An update request is received, and authentication and flow limiting processing are performed on the request; obtaining a cache key prefix and a data structure type, routing a cache key to a fragment node based on a consumption write-back event, executing a write-in operation in a distributed cache service, and setting survival time and multi-copy synchronization; delivering the message body to the message middleware and obtaining delivery confirmation; the messages are pulled from the queue according to the configured concurrency, deduplication verification is carried out on idempotent identifiers of the messages, and data persistence operation is executed after the verification message version is higher than the current version of the database; index collection points are implanted into all modules of the whole link, and the request delay, the cache hit rate and the message accumulation depth index are visually displayed. The reasonable distribution and optimization of system resources are realized; the consistency and integrity of the data are guaranteed, and the reliability of message transmission is improved at the same time.
Owner:QINGDAO PORT INT CO LTD +1

Enterprise scientific and technological achievement adaptation method based on big data accurate retrieval and query

The invention discloses an enterprise scientific and technological achievement adaptation method based on big data accurate retrieval and query, and relates to the technical field of big data processing and knowledge maps, and the method comprises the steps: constructing an enterprise technical field knowledge map based on a standardized multi-modal feature matrix, receiving incremental technical data in real time by adopting an Apache Flink streaming processing framework, and carrying out the real-time retrieval and query of the enterprise technical field knowledge map. Dynamically updating a graph node relationship, removing expired nodes through a pruning algorithm, and outputting a dynamic knowledge graph with a timestamp; and receiving a query request of a user, extracting a query vector by utilizing the cross-modal embedding model, retrieving a Top-N candidate node list through a dynamic knowledge graph, calculating a comprehensive similarity score in combination with a graph relation weight, and outputting a sorted technical achievement list. Multi-source heterogeneous data is mapped to a unified semantic space through a cross-modal embedding model, and the intelligent level and the actual application effect of scientific and technological achievement adaptation are comprehensively improved in combination with construction and updating of a dynamic knowledge graph.
Owner:KUNMING SCI & TECH SMALL & MEDIUM ENTERPRISES TECH INNOVATION FUND MANAGEMENT CENT (KUNMING PRODUCTIVITY PROMOTION CENT)

Thermal power plant environment monitoring and early warning method and system based on Internet of Things

The invention provides a thermal power plant environment monitoring and early warning method and system based on the Internet of Things, and relates to the technical field of the Internet of Things, and the method comprises the steps: forming a preprocessing data stream for collected environment monitoring data through an edge calculation node; and carrying out real-time data stream processing by utilizing a distributed message queue, constructing a hierarchical distributed data storage structure, and carrying out homomorphic encryption and desensitization processing on data to form an encrypted data set. Based on an encrypted data set, a long-short term memory network, a random forest algorithm and a limit gradient boosting tree are adopted to construct a layered anomaly detection engine, environmental parameter spatial-temporal characteristics are extracted through a multi-head attention mechanism and residual connection, an early warning threshold value is iteratively calculated based on a Bayesian optimization algorithm, and an early warning level judgment standard is dynamically adjusted. And generating early warning information. And finally, through a three-dimensional visualization module and a knowledge inference engine based on a graph neural network and a deep reinforcement learning algorithm, visualization of early warning information and generation of a self-adaptive emergency processing strategy are realized respectively.
Owner:GUODIAN KARAMAY POWER GENERATION CO LTD

Power management in deterministic tensor streaming processors

Embodiments pertain to reducing power consumption in a computing system comprising one or more deterministic processors. A controller generates a plurality of control signals for a voltage regulator to regulate a supply voltage of a respective one of the one or more deterministic processors. A power management module determines an initial profile for power consumption and performance of an algorithm executed on the respective deterministic processor having an initial value for the supply voltage and an initial value for a clock frequency. The power management module further determines a target profile for power consumption and performance of the algorithm executed on the respective deterministic processor. The controller modifies the plurality of control signals based on the initial profile and the target profile. The respective deterministic processor executes the algorithm while the supply voltage is dynamically modified by the voltage regulator based on the modified plurality of control signals.
Owner:GROQ INC

Video stream processing method for dynamic Gaussian compression and adaptive code rate regulation

The invention discloses a video stream processing method for dynamic Gaussian compression and adaptive code rate regulation, which is suitable for scenes such as virtual reality, augmented reality and three-dimensional video, and comprises the following steps: S1, Gaussian attribute modeling and initialization; s2, constructing a binary hash grid; s3, constructing a deformation prediction network; s4, designing a mask pruning mechanism; s5, entropy modeling and arithmetic coding and decoding module design; s6, model training; and S7, video stream transmission under multiple code rates. According to the method, a unified scheme combining Gaussian volume cloud coding and adaptive video transmission is proposed for the first time, the video data storage and transmission cost is remarkably reduced, and the comprehensive performance superior to that of an existing method is obtained on multiple real and synthetic data sets.
Owner:THE CHINESE UNIV OF HONG KONG (SHENZHEN) FUTURE NETWORK OF INTELLIGENCE INST +1

Internet of Things equipment intelligent management platform based on edge computing

The invention relates to an Internet of Things equipment intelligent management platform based on edge computing, and aims to realize efficient and intelligent management of Internet of Things equipment through an edge computing technology. According to the platform, the edge nodes are deployed near the data source, so that local processing and storage of the data are realized, the data transmission delay is remarkably reduced, and the response speed of the system is improved. And the platform is provided with an intelligent management module, so that the equipment state can be monitored in real time, faults can be predicted, repair operation can be automatically executed, and the stability and reliability of the system are enhanced. And a self-adaptive data stream processing algorithm is adopted, and the data stream is dynamically adjusted according to a real-time network state and equipment requirements, so that the high efficiency of data processing is ensured. The secure transmission module protects the data transmission process through an end-to-end encryption technology, and prevents data leakage and tampering. The cloud-side collaborative resource scheduling system realizes optimal allocation of resources, and a user-defined rule engine provides high flexibility and personalized management strategies to meet different user requirements.
Owner:SHENZHEN WANGUANGDA ELECTRONIC TECHNOLOGY CO LTD

Label-based user portrait adaptive updating method, medium and equipment

The invention discloses a label-based user portrait self-adaptive updating method, a medium and equipment, and solves the problems of poor data timeliness and lagging rule updating in a traditional offline batch processing mode through a collaborative mechanism of real-time stream processing and historical batch processing. A dynamic weight adjustment strategy is adopted, real-time user tags generated by real-time behavior flow data and batch user tags generated by historical behavior batch data analysis are subjected to aging weighted fusion, and dynamic changes of user behavior modes are effectively captured; through feedback behavior data of the user and a closed-loop iteration mechanism of the rule base, self-adaptive evolution of the label rule is realized, and the accuracy and the response speed of a recommendation system are remarkably improved.
Owner:ZHONG FU TONG CO LTD

System and method for hybrid processing of mass offline data and mass real-time data

The invention relates to the technical field of electric data processing, and discloses a system and a method for hybrid processing of mass offline data and mass real-time data, and the system comprises a queue management module which receives task requests, identifies and marks task types, distributes tasks to a stream processing queue or a batch processing queue, and sorts the tasks according to priorities; the task types comprise flow tasks and batch tasks; the thread pool management module establishes a thread pool, sequentially processes tasks according to priorities and collects real-time parameters; the calculation module is responsible for calculating task priorities and calculating task jitter factors based on real-time parameters; the scheduling decision module compares the task jitter factor with a reference value, determines a weight adjustment coefficient, corrects the priority and formulates a scheduling strategy; the alarm module judges whether to trigger an alarm. According to the invention, by introducing the dynamic priority scheduling and preemptive processing mechanism driven by the jitter factor, the real-time response capability and the resource utilization efficiency of the system are improved.
Owner:BEIJING LIUJINSUIYUE TECH CO LTD

Distributed outbound system optimization method based on cloud computing

The invention relates to a distributed outbound system optimization method based on cloud computing. The method comprises the following steps: disassembling an outbound task into fine-grained subtasks including dialing, recording, interaction and recording; generating a unique identifier for each subtask; based on a distributed task scheduler; according to node real-time performance and network delay, dynamically distributing tasks, and introducing a task priority rule; monitoring a task execution state in real time through a distributed tracking system; recovering and redistributing abnormal and failed tasks; fragmenting the call-out data according to regions or customer groups, and distributing the call-out data to different nodes; processing the call record and the client feedback by using a stream processing framework comprising Apache Flink; adding a call task including Kafka into a distributed task queue; setting a dynamic priority queue, and adjusting the priority according to the task timeliness; setting an overtime threshold value for the call task, and monitoring a task state in real time; and when the task is overtime or failed, triggering a retry strategy including adjusting the call period and switching the node.
Owner:SHIJIAZHUANG LINGYUE TECHNOLOGY CO LTD

Power grid fault on-line identification system and method based on big data

The invention discloses a power grid fault online identification system and method based on big data, and relates to the technical field of power systems, and the method comprises the steps: collecting multi-source data of a power grid system through a sensor; data filtering is performed through a hierarchical storage strategy, a data lake is constructed in combination with multi-dimensional data indexes, and multi-source data is stored in a classified manner; aggregating and converting data in a data lake by adopting a real-time stream processing framework, and extracting high-dimensional features related to a fault by combining wavelet transform with principal component analysis; based on the high-dimensional features, identifying the real-time state of the power grid by using a graph neural network in combination with a deep learning model; and according to the identification result of the real-time state, combining a power grid topological structure, and carrying out fault positioning through a time difference method and a graph theory algorithm. The power grid fault on-line identification system based on big data provided by the invention can efficiently manage and store a large amount of multi-source data in a power grid, and improves the data processing and access speed.
Owner:BEIJING SHENGXINNUO TECH CO LTD

Speech recognition feature extraction and reasoning method for artificial intelligence dialogue system

The invention discloses a speech recognition streaming feature extraction and batch reasoning method for a high-concurrency artificial intelligence real-time dialogue system. According to the method, a heterogeneous computing architecture in which a CPU (Central Processing Unit) and a GPU (Graphics Processing Unit) cooperatively work is adopted, operations such as preprocessing, feature extraction and normalization of voice data are executed through the CPU, and batch feature reasoning is executed by utilizing the GPU; the method specifically comprises the following steps: after segmenting and preprocessing voice data at a CPU (Central Processing Unit) end, executing Mel filter bank feature extraction and cepstrum mean variance normalization to obtain a standardized acoustic feature vector; extracting feature vectors from the feature buffer area in batches at the GPU end, filling and aligning the feature vectors, and sending the feature vectors to a neural network model for parallel reasoning calculation; according to the method, streaming processing and batch reasoning in the speech recognition process are realized, and the concurrent processing capability and the real-time performance of the system are effectively improved.
Owner:GUANGZHOU SENJI SOFTWARE TECH CO LTD

Financial statement generation method and system based on cloud computing

The invention provides a financial statement generation method and system based on cloud computing, and the method comprises the steps: collecting multi-source data of an internal business system of an enterprise through a centralized data integration platform based on a cloud computing environment, carrying out the classified storage, and constructing a data set; verifying the data set, and obtaining key index fields related to financial statement generation in the verified data set; performing grouping calculation on the key index fields by adopting a data aggregation algorithm to form a financial data summarization result; dynamically mapping the financial data summarization result by adopting a report generation engine, and matching with a preset financial report template to obtain a report data framework; and performing secondary verification and encryption on the report data framework to obtain a final financial report. According to the technical scheme, automation, real-time performance and unmanned performance of financial statement generation are realized through full-link innovation of edge intelligent acquisition, streaming processing, incremental aggregation and zero-trust security.
Owner:GUANGXI POLYTECHNIC OF IND & COMMERCE

Real-time data stream processing method and device, electronic equipment and storage medium

The invention relates to the technical field of data processing, and discloses a real-time data stream processing method and device, electronic equipment and a storage medium. The method comprises the following steps: distributing a received to-be-processed data stream to a first distributed node through a preset load balancer; fragmenting the received distribution data stream at the first distributed node according to a preset fragmentation rule, then writing the fragmented data stream into a message queue, and performing parallel processing on fragmented data in the message queue through multiple threads; storing the intermediate state data in the fragmented data processing process, generating snapshot data according to the fragmented data, and synchronizing the snapshot data to the distributed storage system; acquiring load data of the first distributed node, and dynamically updating a data flow distribution strategy of the load balancer according to the load data through the resource use prediction model; and writing a processing result of the to-be-processed data stream into persistent storage through an asynchronous batch mechanism, and returning the processing result through a preset real-time communication protocol. The method improves the data stream processing efficiency and stability.
Owner:PING AN HEALTH INSURANCE CO LTD

Cross-border e-commerce big data analysis system and method

The invention provides a cross-border e-commerce big data analysis system and method, and the method comprises the steps: obtaining multi-dimensional original business data of a cross-border e-commerce platform, carrying out the real-time collection through a distributed data collection system, and obtaining a structured business data set; data collected in real time is cleaned, converted and structured, a unified data model is constructed, and integration of heterogeneous data sources is achieved; storing the processed structured data in a time sequence database to provide data support for real-time analysis; performing real-time aggregation and statistical analysis on the newly added data to generate business index and key performance index data; a Lambda framework is adopted, batch processing and stream processing are combined, the historical data and the real-time data are subjected to full-amount analysis, and an analysis result is obtained; based on an analysis result, a data mining algorithm is adopted to identify a service mode and predict a future trend; a multi-dimensional data cube is constructed, multi-dimensional and multi-granularity data exploration and interactive analysis are supported, and management decision making is assisted.
Owner:CHENGDU TAILYOU TECHNOLOGY CO LTD

Intelligent data stream processing system based on Flink and implementation method thereof

The invention discloses an intelligent data stream processing system based on Flink and an implementation method thereof, and belongs to the technical field of distributed streaming data processing, and the implementation method comprises the following steps: a data source access layer uniformly accesses multi-source data; the entity monitoring layer captures service entity change in a non-intrusive manner based on JPA, generates a standardized message body and asynchronously delivers the standardized message body to double channels; the unified event model layer maps the data into a standardized event and expands the standardized event; the Flink stream processing engine layer is used for complex event processing, state management and window calculation, and supports declarative pipeline definition and dynamic construction; the anomaly detection layer performs multi-dimensional anomaly detection; the distributed task scheduling layer schedules tasks according to a DAG model, and realizes load balancing through hotspot splitting and state transition; and the result storage and visualization layer adopts a cold and hot separation strategy to store data, and provides a dynamic visualization interface. The real-time performance, the consistency and the intelligent level of data processing are remarkably improved, and the development and maintenance cost is reduced.
Owner:BEIJING NEUSOFT HUIJU INFORMATION TECH HLDG CO LTD

Financial data intelligent quality inspection method and system

The invention provides a financial data intelligent quality inspection method and system, and the method comprises the steps: S1, accessing a real-time transaction data flow through a dynamic rule engine, and enabling the dynamic rule engine to dynamically adjust the rule weight through a Bayesian network and reinforcement learning hybrid model; s2, calling a multi-modal LLM verification framework, performing joint semantic analysis on the text, the image and the time series data, and generating a risk early warning signal; s3, identifying a cross-entity risk path based on the financial knowledge graph, and converting the identified risk path into a structured risk report; and S4, a closed loop iteration system is formed according to the weight of the early warning feedback optimization rule. According to the method, full-life-cycle quality management and control of financial transactions can be realized through technical collaboration of real-time data stream processing, multi-dimensional semantic verification and cross-entity risk tracking.
Owner:AACAT TECHNOLOGY LTD

Data quality monitoring and repairing system

The invention provides a data quality monitoring and repairing system, and belongs to the technical field of data repairing, and the system comprises an identification module which obtains data of a plurality of data sources, and automatically identifies a plurality of data quality problems existing in the data; the first determination module is used for determining a root cause and an influence path of a data quality problem based on a causal relationship reasoning technology; the evaluation module is used for evaluating the severity and the influence range of each data quality problem; the selection module is used for distributing priority for each data quality problem on the basis of a predefined rule according to the severity and the influence range, determining a processing sequence and selecting an optimal repair strategy for automatic repair; and the automatic repairing module is used for continuously monitoring the repaired data in real time, detecting a new data quality problem by utilizing a stream processing technology and a real-time analysis algorithm, and automatically repairing the data. The problems that data is inaccurate due to low identification efficiency and automatic and real-time processing of the data quality problem cannot be guaranteed are solved.
Owner:TAIJI COMPUTER CORPORATION LIMITED

Low-delay video stream real-time processing method and device

The invention relates to the technical field of computer video processing, and discloses a low-delay video stream real-time processing method and device, and the method comprises the steps: obtaining original video stream data, and processing the original video stream data through employing a lightweight motion prediction method; processing the macro block data set and the predicted coding configuration parameter by adopting multi-thread assembly line coding to obtain a coded data block; establishing a data transmission mechanism to perform data flow control on the unified memory access interface; a heterogeneous task scheduling strategy is adopted to distribute task division results; a lightweight neural network is adopted to carry out parameter adaptive adjustment, and an optimized video stream processing result is obtained; according to the method, a zero-copy data transmission technology is adopted, and optimal configuration and efficient utilization of computing resources are achieved.
Owner:HUNAN BEICHUANG INTELLIGENT TECHNOLOGY CO LTD