1056 results about "Stream processing" patented technology

The invention discloses an experimental calibration method based on multi-sensor signal fusion processing, particularly relates to the field of data analysis, and comprises the steps of a stepped confrontation disturbance protocol, data stream processing and real-time analysis, and multi-stage logic judgment and calibration execution. A stepped confrontation disturbance protocol is created for the first time, potential errors of the sensor are excited in real time and corrected in a closed loop mode under the condition that system operation is not interrupted, and a traditional shutdown calibration mode is thoroughly replaced; a multi-stage disturbance defense system is constructed, and composite interferences such as spectrum aliasing, overload distortion and signal-to-noise ratio attenuation caused by electromechanical thermal multi-physical field strong coupling are effectively suppressed; laser space calibration and dynamic rigidity modulation technologies are fused, and space drift errors caused by deformation of the mounting base and mechanical loosening are accurately diagnosed and compensated. Finally, long-term precision stability and data fusion reliability of the multi-source sensor under complex working conditions are achieved.

The invention discloses a wide-area landslide rapid identification method based on an interpretable intelligent algorithm, and relates to the field of remote sensing science and technology, and the method comprises the steps: building a dual-channel feature extraction architecture through multi-source spatio-temporal data fusion and knowledge graph dynamic weighting: capturing image local textures through a lightweight CNN, modeling geological spatial correlation through a graph convolutional network, and carrying out the recognition of the landslide. Combining the SHAP value and causal reasoning to generate an interpretable contribution degree thermodynamic diagram and a rule chain; a knowledge graph bidirectional verification system is introduced, spatial logic contradictions are verified by using prior rules, and co-evolution of a model and a rule base is triggered based on misjudgment samples; outputting a multi-dimensional credibility report, quantifying uncertainty by Monte Carlo Dropout, and customizing interpretation granularity according to roles; a terrain-adaptive block-stream processing architecture is adopted, and edge lightweight deployment and federated learning are combined, so that wide-area real-time early warning and model dynamic updating are realized. According to the scheme, the limitation of a traditional black box model is broken through, and a disaster prevention closed loop with physical driving, transparent decision and second-level response is formed.

The invention discloses a geological disaster early warning method and a precise early warning system based on multi-source data fusion, and relates to the technical field of geological disaster early warning. According to the method, multi-source heterogeneous data such as remote sensing, meteorological and geological monitoring are fused, a standardized protocol is utilized to unify a data format and temporal-spatial resolution, a standardized data set is formed, key features are extracted by adopting principal component analysis and a recursive feature elimination algorithm, and a long-short-term memory network and a convolutional neural network model are combined, so that the real-time performance of the system is improved. According to the method, the disaster risk is accurately predicted, the space risk distribution diagram is generated, in addition, through application of the real-time stream processing framework and the self-adaptive learning algorithm, rapid distribution of early warning signals and dynamic optimization of model parameters are achieved, the accuracy and timeliness of an early warning system are remarkably improved, and the geological disaster risk is effectively reduced.

The invention discloses an engineering project supervision real-time visual decision-making platform based on multi-source heterogeneous data fusion, and relates to the technical field of engineering supervision. The visual decision platform is in communication connection with a data acquisition module, a project risk feature module, a risk identification model construction module, a real-time data stream processing module, a risk early warning module and a decision support module. According to the invention, the risk in the engineering project is accurately identified in combination with historical multi-source heterogeneous data and project risk characteristics, and meanwhile, the real-time data stream processing module can analyze data in real time, identify potential risks and problems and generate early warning information, so that the accuracy and timeliness of risk identification are greatly improved, and the risk identification efficiency is improved. According to the invention, powerful decision support is provided for supervisors, and through the preset early warning level and the early warning signal, managers can rapidly understand the risk condition and take corresponding measures, thereby effectively reducing the project risk.

The invention provides an intelligent data production method and system based on a graph neural network and adaptive learning, and the method comprises the steps: collecting a text, a time sequence, an image and sensor data in real time, and converting unstructured data into a structured JSON format through a format analysis tool; extracting text entity features based on a large language model, modeling a cross-modal entity association relationship in combination with a graph neural network, and fusing multi-source heterogeneous data through a dynamic knowledge graph; the detection threshold is dynamically optimized by adopting a reinforcement learning algorithm, and the false alarm rate is reduced in combination with a double-track verification mechanism; and predicting a data trend by using the space-time diagram convolutional network, and outputting an interpretability analysis report through the generative large model. According to the invention, a multi-protocol adaptation engine and a distributed stream processing architecture are adopted, and a protocol analysis layer automatically identifies a plurality of industrial protocols, so that the manual adaptation time is reduced; and real-time synchronization of production line-level data is realized through edge node parallel acquisition and compression transmission.

The invention provides a cache information processing method and device of a port platform and a medium, and belongs to the technical field of port distributed information processing. An update request is received, and authentication and flow limiting processing are performed on the request; obtaining a cache key prefix and a data structure type, routing a cache key to a fragment node based on a consumption write-back event, executing a write-in operation in a distributed cache service, and setting survival time and multi-copy synchronization; delivering the message body to the message middleware and obtaining delivery confirmation; the messages are pulled from the queue according to the configured concurrency, deduplication verification is carried out on idempotent identifiers of the messages, and data persistence operation is executed after the verification message version is higher than the current version of the database; index collection points are implanted into all modules of the whole link, and the request delay, the cache hit rate and the message accumulation depth index are visually displayed. The reasonable distribution and optimization of system resources are realized; the consistency and integrity of the data are guaranteed, and the reliability of message transmission is improved at the same time.

The invention discloses an enterprise scientific and technological achievement adaptation method based on big data accurate retrieval and query, and relates to the technical field of big data processing and knowledge maps, and the method comprises the steps: constructing an enterprise technical field knowledge map based on a standardized multi-modal feature matrix, receiving incremental technical data in real time by adopting an Apache Flink streaming processing framework, and carrying out the real-time retrieval and query of the enterprise technical field knowledge map. Dynamically updating a graph node relationship, removing expired nodes through a pruning algorithm, and outputting a dynamic knowledge graph with a timestamp; and receiving a query request of a user, extracting a query vector by utilizing the cross-modal embedding model, retrieving a Top-N candidate node list through a dynamic knowledge graph, calculating a comprehensive similarity score in combination with a graph relation weight, and outputting a sorted technical achievement list. Multi-source heterogeneous data is mapped to a unified semantic space through a cross-modal embedding model, and the intelligent level and the actual application effect of scientific and technological achievement adaptation are comprehensively improved in combination with construction and updating of a dynamic knowledge graph.

The invention provides a thermal power plant environment monitoring and early warning method and system based on the Internet of Things, and relates to the technical field of the Internet of Things, and the method comprises the steps: forming a preprocessing data stream for collected environment monitoring data through an edge calculation node; and carrying out real-time data stream processing by utilizing a distributed message queue, constructing a hierarchical distributed data storage structure, and carrying out homomorphic encryption and desensitization processing on data to form an encrypted data set. Based on an encrypted data set, a long-short term memory network, a random forest algorithm and a limit gradient boosting tree are adopted to construct a layered anomaly detection engine, environmental parameter spatial-temporal characteristics are extracted through a multi-head attention mechanism and residual connection, an early warning threshold value is iteratively calculated based on a Bayesian optimization algorithm, and an early warning level judgment standard is dynamically adjusted. And generating early warning information. And finally, through a three-dimensional visualization module and a knowledge inference engine based on a graph neural network and a deep reinforcement learning algorithm, visualization of early warning information and generation of a self-adaptive emergency processing strategy are realized respectively.

The invention discloses a video stream processing method for dynamic Gaussian compression and adaptive code rate regulation, which is suitable for scenes such as virtual reality, augmented reality and three-dimensional video, and comprises the following steps: S1, Gaussian attribute modeling and initialization; s2, constructing a binary hash grid; s3, constructing a deformation prediction network; s4, designing a mask pruning mechanism; s5, entropy modeling and arithmetic coding and decoding module design; s6, model training; and S7, video stream transmission under multiple code rates. According to the method, a unified scheme combining Gaussian volume cloud coding and adaptive video transmission is proposed for the first time, the video data storage and transmission cost is remarkably reduced, and the comprehensive performance superior to that of an existing method is obtained on multiple real and synthetic data sets.

The invention discloses a label-based user portrait self-adaptive updating method, a medium and equipment, and solves the problems of poor data timeliness and lagging rule updating in a traditional offline batch processing mode through a collaborative mechanism of real-time stream processing and historical batch processing. A dynamic weight adjustment strategy is adopted, real-time user tags generated by real-time behavior flow data and batch user tags generated by historical behavior batch data analysis are subjected to aging weighted fusion, and dynamic changes of user behavior modes are effectively captured; through feedback behavior data of the user and a closed-loop iteration mechanism of the rule base, self-adaptive evolution of the label rule is realized, and the accuracy and the response speed of a recommendation system are remarkably improved.

The invention relates to the technical field of electric data processing, and discloses a system and a method for hybrid processing of mass offline data and mass real-time data, and the system comprises a queue management module which receives task requests, identifies and marks task types, distributes tasks to a stream processing queue or a batch processing queue, and sorts the tasks according to priorities; the task types comprise flow tasks and batch tasks; the thread pool management module establishes a thread pool, sequentially processes tasks according to priorities and collects real-time parameters; the calculation module is responsible for calculating task priorities and calculating task jitter factors based on real-time parameters; the scheduling decision module compares the task jitter factor with a reference value, determines a weight adjustment coefficient, corrects the priority and formulates a scheduling strategy; the alarm module judges whether to trigger an alarm. According to the invention, by introducing the dynamic priority scheduling and preemptive processing mechanism driven by the jitter factor, the real-time response capability and the resource utilization efficiency of the system are improved.

The invention relates to a distributed outbound system optimization method based on cloud computing. The method comprises the following steps: disassembling an outbound task into fine-grained subtasks including dialing, recording, interaction and recording; generating a unique identifier for each subtask; based on a distributed task scheduler; according to node real-time performance and network delay, dynamically distributing tasks, and introducing a task priority rule; monitoring a task execution state in real time through a distributed tracking system; recovering and redistributing abnormal and failed tasks; fragmenting the call-out data according to regions or customer groups, and distributing the call-out data to different nodes; processing the call record and the client feedback by using a stream processing framework comprising Apache Flink; adding a call task including Kafka into a distributed task queue; setting a dynamic priority queue, and adjusting the priority according to the task timeliness; setting an overtime threshold value for the call task, and monitoring a task state in real time; and when the task is overtime or failed, triggering a retry strategy including adjusting the call period and switching the node.

The invention provides a financial statement generation method and system based on cloud computing, and the method comprises the steps: collecting multi-source data of an internal business system of an enterprise through a centralized data integration platform based on a cloud computing environment, carrying out the classified storage, and constructing a data set; verifying the data set, and obtaining key index fields related to financial statement generation in the verified data set; performing grouping calculation on the key index fields by adopting a data aggregation algorithm to form a financial data summarization result; dynamically mapping the financial data summarization result by adopting a report generation engine, and matching with a preset financial report template to obtain a report data framework; and performing secondary verification and encryption on the report data framework to obtain a final financial report. According to the technical scheme, automation, real-time performance and unmanned performance of financial statement generation are realized through full-link innovation of edge intelligent acquisition, streaming processing, incremental aggregation and zero-trust security.

The invention relates to the technical field of data processing, and discloses a real-time data stream processing method and device, electronic equipment and a storage medium. The method comprises the following steps: distributing a received to-be-processed data stream to a first distributed node through a preset load balancer; fragmenting the received distribution data stream at the first distributed node according to a preset fragmentation rule, then writing the fragmented data stream into a message queue, and performing parallel processing on fragmented data in the message queue through multiple threads; storing the intermediate state data in the fragmented data processing process, generating snapshot data according to the fragmented data, and synchronizing the snapshot data to the distributed storage system; acquiring load data of the first distributed node, and dynamically updating a data flow distribution strategy of the load balancer according to the load data through the resource use prediction model; and writing a processing result of the to-be-processed data stream into persistent storage through an asynchronous batch mechanism, and returning the processing result through a preset real-time communication protocol. The method improves the data stream processing efficiency and stability.

The invention provides a cross-border e-commerce big data analysis system and method, and the method comprises the steps: obtaining multi-dimensional original business data of a cross-border e-commerce platform, carrying out the real-time collection through a distributed data collection system, and obtaining a structured business data set; data collected in real time is cleaned, converted and structured, a unified data model is constructed, and integration of heterogeneous data sources is achieved; storing the processed structured data in a time sequence database to provide data support for real-time analysis; performing real-time aggregation and statistical analysis on the newly added data to generate business index and key performance index data; a Lambda framework is adopted, batch processing and stream processing are combined, the historical data and the real-time data are subjected to full-amount analysis, and an analysis result is obtained; based on an analysis result, a data mining algorithm is adopted to identify a service mode and predict a future trend; a multi-dimensional data cube is constructed, multi-dimensional and multi-granularity data exploration and interactive analysis are supported, and management decision making is assisted.

The invention discloses an intelligent data stream processing system based on Flink and an implementation method thereof, and belongs to the technical field of distributed streaming data processing, and the implementation method comprises the following steps: a data source access layer uniformly accesses multi-source data; the entity monitoring layer captures service entity change in a non-intrusive manner based on JPA, generates a standardized message body and asynchronously delivers the standardized message body to double channels; the unified event model layer maps the data into a standardized event and expands the standardized event; the Flink stream processing engine layer is used for complex event processing, state management and window calculation, and supports declarative pipeline definition and dynamic construction; the anomaly detection layer performs multi-dimensional anomaly detection; the distributed task scheduling layer schedules tasks according to a DAG model, and realizes load balancing through hotspot splitting and state transition; and the result storage and visualization layer adopts a cold and hot separation strategy to store data, and provides a dynamic visualization interface. The real-time performance, the consistency and the intelligent level of data processing are remarkably improved, and the development and maintenance cost is reduced.

The invention provides a financial data intelligent quality inspection method and system, and the method comprises the steps: S1, accessing a real-time transaction data flow through a dynamic rule engine, and enabling the dynamic rule engine to dynamically adjust the rule weight through a Bayesian network and reinforcement learning hybrid model; s2, calling a multi-modal LLM verification framework, performing joint semantic analysis on the text, the image and the time series data, and generating a risk early warning signal; s3, identifying a cross-entity risk path based on the financial knowledge graph, and converting the identified risk path into a structured risk report; and S4, a closed loop iteration system is formed according to the weight of the early warning feedback optimization rule. According to the method, full-life-cycle quality management and control of financial transactions can be realized through technical collaboration of real-time data stream processing, multi-dimensional semantic verification and cross-entity risk tracking.

The invention relates to the technical field of computer video processing, and discloses a low-delay video stream real-time processing method and device, and the method comprises the steps: obtaining original video stream data, and processing the original video stream data through employing a lightweight motion prediction method; processing the macro block data set and the predicted coding configuration parameter by adopting multi-thread assembly line coding to obtain a coded data block; establishing a data transmission mechanism to perform data flow control on the unified memory access interface; a heterogeneous task scheduling strategy is adopted to distribute task division results; a lightweight neural network is adopted to carry out parameter adaptive adjustment, and an optimized video stream processing result is obtained; according to the method, a zero-copy data transmission technology is adopted, and optimal configuration and efficient utilization of computing resources are achieved.

The invention discloses an electronic product personalized marketing method based on a big data accurate portrait, and relates to the technical field of big data processing, and the method comprises the following steps: multi-source data collection and cross-domain fusion processing: obtaining a multi-source data set related to an electronic product from each platform of the Internet by using a web crawler and a streaming processing engine, the multi-source data set is subjected to cleaning classification and alignment fusion processing to generate multi-modal data, and then the multi-modal data is updated in real time to dynamically reflect data changes, so that a foundation is laid for subsequent data analysis; according to the method, multi-source data is collected by using a web crawler and a streaming processing engine, quantitative analysis is performed on the data by using a Hofscode culture dimension model and a Maslow demand level theory, a user portrait is constructed by using a distributed data flow engine Apache Flink to dynamically update a label, and a GAN (GAN Adversarial Generative Network) anomaly detection and evaluation analysis method is applied, so that the user portrait is obtained. And comprehensive and accurate data acquisition and processing are realized.

The invention relates to the technical field of data management, and discloses an iterative updating method for a data consanguinity atlas, which comprises the following steps of: capturing data increment and metadata change; constructing a directed acyclic graph of the data blood relationship; performing layered modeling on the data blood relationship graph; performing iterative updating on the blood relationship map based on a layered modeling result; processing and storage of incremental data are optimized through distributed calculation; real-time blood relationship map updating is realized by using a stream processing technology; a machine learning model is used for predicting blood relationship path change, abnormal nodes are detected and repaired, and the step of capturing data increment and metadata change comprises the substeps that data snapshots are obtained, and the data snapshots comprise new snapshots and old snapshots of data. Through a stream processing technology, a machine learning model and an automatic repair mechanism, efficient real-time updating of the data blood relationship map, intelligent path prediction and automatic repair of abnormal nodes are realized, and the system stability and the updating efficiency are improved.

The invention discloses an AI workflow processing method and system for intelligent hardware, and the method comprises the steps: obtaining multi-person cooperation input information, including voice, text and image information, through an edge device; the multi-user cooperation input information is processed, workflow related data is generated, and the workflow related data comprises a multi-user cooperation input information integration result, a work priority processing result, workflow distribution processing information, flow and computing power distribution information, plug flow interruption emergency processing information, live broadcast effect data and live broadcast setting optimization information; sending the workflow related data to cloud service equipment; receiving target format instruction information sent by the cloud service equipment; and determining a target application program based on the target format instruction information, and sending the target format instruction information to the target application program for the target application program to process.

The invention provides an urban space intelligent processing method based on multi-modal fusion, and the method comprises the steps: taking multi-modal data as input, and constructing a unified data stream processing and feature alignment mechanism; a physical space is used as a core framework, and the multi-modal data is converted into a space behavior graph with space-time position semantics; constructing an entity attribute-relation type-influence weight ternary interaction model on the basis of an interaction layer of the spatial behavior map, and analyzing a human, object and environment ternary interaction relation in the city and the park based on the ternary interaction model; designing a space intelligent engine with time sequence modeling and dynamic prediction capabilities; and constructing a task processing system. According to the invention, through deep combination of multi-modal fusion and a space intelligent technology, full-link upgrading of urban space from data perception to intelligent decision is realized, and powerful technical support is provided for fine management, efficient operation and safety guarantee of complex space scenes.

The invention relates to the technical field of edge computing, and discloses an intelligent edge computing co-processing system based on an integrated circuit, which comprises a heterogeneous computing cluster module, a hardware computing unit set, a multi-core control processor based on RISC-V, a programmable pulse tensor computing array and a reconfigurable engine oriented to streaming processing, according to the intelligent edge computing cooperative processing system based on the integrated circuit, zero-delay data exchange is realized through silicon intermediate layer integration of the heterogeneous computing cluster module and a snakelike data channel, and traditional bus arbitration delay is eliminated through real-time operation code analysis and optimal optical communication path mapping of the hardware task routing matrix (TRF) module; the priority path distribution of the task scheduling subsystem is synchronously coordinated with the time-sensitive task, so that the collaborative efficiency of the computing unit is improved in multiple dimensions, the non-blocking transmission of the high-priority task is ensured, and the effect of enhancing the overall processing capability and response speed of the intelligent edge computing is achieved.

A method for managing a data stream processing pipeline includes: monitoring, by an orchestrator, data stream ingestion of a streaming storage system (SSS) to obtain data stream metrics; analyzing, by the orchestrator, the data stream metrics based on a user-defined scaling policy; making, based on the analyzing and by the orchestrator, a first determination that task manager scaling is required; making, by the orchestrator, a second determination that the data stream ingestion is increased, in which the second determination indicates an increase in a number of parallel stream segments associated with a data stream, in which a segment store hosted by the SSS manages the parallel stream segments; and to increase a stream processing system's compute capability and by the orchestrator, initiating scaling of a number of the plurality of task managers to support the increase in the number of the parallel stream segments.

The invention provides a parallel computing method and device, electronic equipment and a storage medium, and relates to the technical field of parallel computing, and the method comprises the steps: carrying out the first protocol operation of a target tensor based on each computing core in each stream processor cluster, and generating a data block containing the computing result of each computing core; writing a data block generated by each stream processor cluster into a shared cache; under the condition that each stream processor cluster completes the first protocol operation, reading a data block written by each stream processor cluster from the shared cache; and executing a second protocol operation on the data block read from the shared cache to generate a calculation result of the target tensor. According to the method and device provided by the invention, the parallel architecture and memory access characteristics of the artificial intelligence chip can be better matched, the unnecessary calculation delay and synchronization overhead of the cross-flow processor cluster in the parallel calculation process are reduced, the bandwidth utilization rate of the shared cache is improved, and the overall performance and calculation efficiency of parallel calculation are remarkably improved.

Improved placement of memory and functional modules, ‘tiles’, within a tiled processor architecture are disclosed for linear algebra calculations involving vectors and matrices comprising large amounts of data. The improved placement places the data in close proximity to the functional modules performing calculations using the data. These modules enable these calculations to be performed more quickly while using less energy. These modules, in particular, improve the efficiency of the training and application of deep learning and artificial neural network systems. This Abstract and the independent Claims are concise signifiers of embodiments of the claimed inventions. The Abstract does not limit the scope of the claimed inventions.

The invention discloses a digital human real-time interaction system and an AI voice dialogue chip circuit system thereof, and the system employs a modular design, and comprises the following modules: a BaseReal module which is responsible for audio frame management, TTS service initialization, video recording and customized audio and video cycle management; the media stream management module has an audio stream processing function and is responsible for real-time transmission of audio and video streams; the speech recognition module is responsible for speech feature extraction and text conversion; the text-to-voice module is used for converting the collected audio data into a text; the lip shape synchronization module is used for realizing real-time lip shape synchronization animation generation through lip shape synchronization algorithm optimization and inheritance of a basic class; and the large model module comprises an LLM large model based on mass data fine tuning, an api interface or a workflow agent. The method has the effects of voice recognition, text-to-voice conversion, lip shape synchronization, real-time audio and video interaction, real-time translation and the like.

The invention discloses an equipment digital inspection evaluation method and system based on big data, and belongs to the field of artificial intelligence, and the method comprises the steps: S1, collecting multi-source heterogeneous data; s2, unifying timestamps by adopting a time synchronization technology, and realizing accurate alignment of multi-source data; s3, cleaning and structuring data by using a distributed stream processing framework, eliminating invalid redundancy, and completing coding standardization; s4, constructing a graph neural network model based on equipment topology, and extracting a state feature vector; s5, combining with historical fault labels, applying a reinforcement learning training evaluation strategy, and improving prediction accuracy and inspection efficiency; s6, dynamically sorting the inspection tasks according to a strategy, and determining a priority and a path in combination with the risk and the dependency relationship; s7, generating a digital inspection evaluation report fusing real-time analysis and a model result; and S8, performing digital signature and timestamp registration through an intelligent contract uplink evidence storage, and ensuring that a result is verifiable, traceable and non-tampered. The beneficial effects are that inspection efficiency is improved and data safety and reliability are ensured.

The invention relates to the technical field of information management, in particular to a student enrollment information management system and method, and the system comprises a student enrollment data integration module, a student enrollment risk assessment module, a student enrollment resource optimization module, a data flow adjustment module and a student enrollment effect analysis module. And carrying out data matching on the basic information, the score and the professional intention. According to the method, when the data are screened and matched, the risk management capability is enhanced through identification and quantitative processing of the abnormal points, analysis of enrollment resources and demand differences and identification of performance bottlenecks caused by wrong decisions are avoided, the information processing flow is optimized, the system response time is shortened, the operation efficiency is improved, and the system reliability is improved. By monitoring the deviation between the data stream and the expected target in real time and adjusting the data stream processing process in time, the accuracy and real-time performance of the data are ensured, so that the transparency of the enrollment process and the evaluability of the execution effect are improved, and the management efficiency of the whole enrollment process is optimized.

The invention relates to the technical field of intelligent payment, and discloses a payment interface market dynamic matching and optimization method, which comprises the steps of obtaining multi-dimensional data, and constructing a user portrait based on the multi-dimensional data; based on a stream processing engine, aggregating the user behavior data, and updating a user portrait; identifying the user portrait based on the decision tree model, determining a transaction scene, constructing a differentiation strategy, and generating a payment mode recommendation list based on the differentiation strategy; inputting the user portrait into a pre-trained probability model for comparison, and updating a payment mode recommendation list based on a multi-target algorithm; determining a payment channel state based on the multi-dimensional data, obtaining a comprehensive score of a payment channel based on the payment channel state, and generating a candidate channel based on the comprehensive score; and obtaining an equipment health degree state, and determining an optimal payment channel based on the equipment health degree state. According to the invention, the operation fluency of a user in a complex use scene is improved.