Intelligent factory process optimization method and system based on multimodal data fusion

By combining multimodal data fusion and Bayesian causal graphs, the problem of low accuracy of multi-source heterogeneous data in process optimization is solved, and efficient process parameter adjustment and yield improvement are achieved.

WO2026232712A1PCT designated stage Publication Date: 2026-11-12SHENZHEN XUANYU TECHNOLOGY CO LTD
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
PCT/CN2025/114617
Authority / Receiving Office
WO · WO
Patent Type
Applications
Current Assignee / Owner
Priority Date
2025-05-07
Filing Date
2025-08-14
Publication Date
2026-11-12

AI Technical Summary

Technical Problem

In existing technologies, the sampling frequency, dimensionality, and semantic differences of multi-source heterogeneous data are large, resulting in low accuracy of process optimization in industrial production processes, and manual parameter tuning is difficult to adapt to dynamic operating conditions.

Method used

Factory production data is acquired through sensors, preprocessed to generate multimodal tensors, decoupled from image and acoustic submodal data, a defect identification network is constructed, and Bayesian causal graphs are used to output adjustment parameters and generate process adjustment schemes.

Benefits of technology

It achieves efficient fusion of heterogeneous data, reveals the causal relationship between process parameters and quality indicators, improves the accuracy and yield of process optimization, and has dynamic adaptive capabilities.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN2025114617_12112026_PF_FP_ABST
    Figure CN2025114617_12112026_PF_FP_ABST
Patent Text Reader

Abstract

An intelligent factory process optimization method and system based on multimodal data fusion. The method comprises: acquiring, by means of a sensor, production data generated during factory operation; preprocessing the production data to obtain first data, and generating a multimodal tensor on the basis of data sources and association relationships of the first data; decoupling the multimodal tensor, inputting obtained image sub-modal data and acoustic sub-modal data into a defect recognition network, and outputting defect features; constructing a Bayesian causal graph on the basis of the defect features, physical features representing a physical state of a process in the production data, and environmental features, and outputting adjustment parameters of a production process by means of the Bayesian causal graph; and on the basis of the adjustment parameters and the production data, generating a process adjustment scheme. Closed-loop optimization from data perception to decision execution is formed, such that process parameters are dynamically self-adaptive, thereby improving the accuracy of process optimization and the yield rate.
Need to check novelty before this filing date? Find Prior Art

Description

A Smart Factory Process Optimization Method and System Based on Multimodal Data Fusion Technical Field

[0001] This application relates to the field of industrial data processing technology, and more specifically, to a smart factory process optimization method and system based on multimodal data fusion. Background Technology

[0002] With the rapid development of Industry 4.0 and smart manufacturing, the production processes in modern factories are becoming increasingly complex, involving the real-time acquisition and analysis of multi-source heterogeneous data, such as sensor time-series data, visual images, acoustic signals, and process logs. Multimodal data fusion technology integrates diverse information such as equipment status, production processes, and environmental parameters to achieve real-time monitoring and intelligent control of the production process. For example, in the field of smart manufacturing, the fusion of sensor data, image data, and video data can improve production efficiency and product quality.

[0003] In practical applications, manual parameter tuning relies on expert experience and is difficult to adapt to dynamic operating conditions. Furthermore, the sampling frequency, dimensionality, and semantic differences between different modal data are significant, making direct correlation analysis difficult. This results in low accuracy of process optimization in industrial production. Summary of the Invention

[0004] This application provides a method and system for optimizing intelligent factory processes based on multimodal data fusion, which can at least partially solve the problem of low accuracy in process optimization during industrial production.

[0005] Other features and advantages of this application will become apparent from the following detailed description, or may be learned in part from practice of this application.

[0006] According to one aspect of this application, a smart factory process optimization method based on multimodal data fusion is provided, comprising: acquiring production data during factory operation through sensors; preprocessing the production data to obtain first data, generating a multimodal tensor based on the data source and correlation of the first data; decoupling the multimodal tensor, inputting the obtained image submodal data and acoustic submodal data into a defect recognition network, and outputting defect features; constructing a Bayesian causal graph based on the defect features, physical features representing the physical state of the process in the production data, and environmental features, and outputting adjustment parameters of the production process through the Bayesian causal graph; and generating a process adjustment scheme based on the adjustment parameters and the production data.

[0007] In this application, based on the aforementioned scheme, the step of acquiring production data during the factory operation process through sensors includes: acquiring temperature data, image streams, waveform data, and production logs during the factory operation process through sensors, as the production data.

[0008] In this application, based on the aforementioned scheme, the step of preprocessing the production data to obtain first data and generating a multimodal tensor according to the data source and correlation of the first data includes: cleaning the production data to obtain the first data, and determining the correlation between the first data and the corresponding time vector; generating tensor nodes based on the data source corresponding to the first data, and generating tensor hyperedges according to the correlation between the first data and the time vector; and generating a multimodal tensor according to the tensor nodes and the tensor hyperedges.

[0009] In this application, based on the aforementioned scheme, the step of decoupling the multimodal tensor and inputting the obtained image submodal data and acoustic submodal data into the defect recognition network to output defect features includes: decoupling the multimodal tensor to separate image submodal data and acoustic submodal data; constructing a defect recognition network based on an encoder, generator, and discriminator, and training the defect recognition network with mixed data consisting of normal samples and defect samples; inputting the image submodal data and acoustic submodal data into the defect recognition network respectively, separating abnormal features through reconstruction loss and distribution alignment loss, and outputting defect features.

[0010] In this application, based on the aforementioned scheme, the step of constructing a Bayesian causal graph based on defect features, physical features representing the physical state of the process in production data, and environmental features, and outputting adjustment parameters of the production process through the Bayesian causal graph, includes: constructing a Bayesian causal graph based on defect features, physical features representing the physical state of the process in production data, and environmental features; simulating and adjusting the parameters in the Bayesian causal graph to obtain the influence parameters of the simulated and adjusted parameters on the relationship chains in the Bayesian causal graph; and adjusting the parameters in the Bayesian causal graph based on the influence parameters to obtain the adjustment parameters of the production process.

[0011] In this application, based on the aforementioned scheme, the step of generating a process adjustment scheme according to the adjustment parameters and the production data includes: generating candidate parameter combinations based on the adjustment parameters and the production data, and determining the objective function and constraints according to actual production needs; deriving diverse initial parameter combinations based on the candidate parameter combinations, and searching for the optimal parameter combination from the initial parameter combinations according to the objective function and constraints; and generating a process adjustment scheme based on the optimal parameter combination.

[0012] In this application, based on the aforementioned scheme, after generating the process adjustment scheme according to the adjustment parameters and the production data, the method further includes: forwarding the process adjustment scheme to the control terminal and displaying it on the interface of the control terminal.

[0013] According to one aspect of this application, a smart factory process optimization system based on multimodal data fusion is provided, comprising:

[0014] The acquisition unit is used to acquire production data during the factory operation process through sensors;

[0015] Tensor units are used to preprocess the production data to obtain first data, and generate multimodal tensors based on the data source and correlation of the first data;

[0016] The detection unit is used to decouple the multimodal tensor, input the obtained image submodal data and acoustic submodal data into the defect recognition network, and output defect features;

[0017] The adjustment unit is used to construct a Bayesian causal graph based on defect features, physical features representing the physical state of the process in the production data, and environmental features, and output the adjustment parameters of the production process through the Bayesian causal graph.

[0018] The scheme unit is used to generate a process adjustment scheme based on the adjustment parameters and the production data.

[0019] According to one aspect of this application, a computer-readable medium is provided having a computer program stored thereon, which, when executed by a processor, implements the smart factory process optimization method based on multimodal data fusion as described in the above embodiments.

[0020] According to one aspect of this application, an electronic device is provided, comprising: one or more processors; and a storage device for storing one or more programs, which, when executed by the one or more processors, cause the one or more processors to implement the smart factory process optimization method based on multimodal data fusion as described in the above embodiments.

[0021] According to one aspect of this application, a computer program product or computer program is provided, comprising computer instructions stored in a computer-readable storage medium. A processor of a computer device reads the computer instructions from the computer-readable storage medium and executes the computer instructions, causing the computer device to perform the smart factory process optimization method based on multimodal data fusion provided in the various optional implementations described above.

[0022] This application's technical solution acquires production data during factory operations through sensors; preprocesses the production data to obtain first data; generates a multimodal tensor based on the data source and correlation of the first data; decouples the multimodal tensor, inputs the obtained image submodal data and acoustic submodal data into a defect recognition network, and outputs defect features; constructs a Bayesian causal graph based on the defect features, physical features representing the physical state of the process in the production data, and environmental features; outputs adjustment parameters for the production process through the Bayesian causal graph; and generates a process adjustment scheme based on the adjustment parameters and the production data. By constructing a spatiotemporally aligned multimodal tensor through multimodal data fusion, efficient fusion of heterogeneous data is achieved. The Bayesian causal graph reveals the causal relationship between process parameters and quality indicators, avoiding the spurious correlation defects of correlation analysis. Finally, a process adjustment scheme is generated through multi-objective optimization, forming a closed-loop optimization from data perception to decision execution, enabling process parameters to have dynamic adaptive capabilities and improving the accuracy and yield of process optimization.

[0023] It should be understood that the above general description and the following detailed description are exemplary and explanatory only, and do not limit this application. Attached Figure Description

[0024] The accompanying drawings, which are incorporated in and form part of this specification, illustrate embodiments consistent with this application and, together with the description, serve to explain the principles of this application. It is obvious that the drawings described below are merely some embodiments of this application, and those skilled in the art can obtain other drawings based on these drawings without any inventive effort.

[0025] Figure 1 schematically illustrates a flowchart of a smart factory process optimization method based on multimodal data fusion in one embodiment of this application.

[0026] Figure 2 schematically illustrates a flowchart of generating multimodal tensors in one embodiment of this application.

[0027] Figure 3 schematically illustrates a smart factory process optimization system based on multimodal data fusion in one embodiment of this application.

[0028] Figure 4 shows a schematic diagram of the structure of a computer system suitable for implementing the electronic device of the present application. Detailed Implementation

[0029] Exemplary embodiments will now be described more fully with reference to the accompanying drawings. However, these exemplary embodiments can be implemented in many forms and should not be construed as limited to the examples set forth herein; rather, these embodiments are provided to make this application more comprehensive and complete, and to fully convey the concept of the exemplary embodiments to those skilled in the art.

[0030] Furthermore, the described features, structures, or characteristics can be combined in any suitable manner in one or more embodiments. Numerous specific details are provided in the following description to give a thorough understanding of embodiments of this application. However, those skilled in the art will recognize that the technical solutions of this application can be practiced without one or more of the specific details, or other methods, components, apparatuses, steps, etc., can be employed. In other instances, well-known methods, apparatuses, implementations, or operations are not shown or described in detail to avoid obscuring various aspects of this application.

[0031] The block diagrams shown in the accompanying drawings are merely functional entities and do not necessarily correspond to physically independent entities. That is, these functional entities can be implemented in software, in one or more hardware modules or integrated circuits, or in different network and / or processor devices and / or microcontroller devices.

[0032] The flowcharts shown in the accompanying drawings are merely illustrative and do not necessarily include all content and operations / steps, nor do they necessarily have to be performed in the described order. For example, some operations / steps can be broken down, while others can be combined or partially combined; therefore, the actual execution order may change depending on the specific circumstances.

[0033] The implementation details of the technical solution of this application are described below:

[0034] Figure 1 shows a flowchart of a smart factory process optimization method based on multimodal data fusion according to an embodiment of this application. Referring to Figure 1, the smart factory process optimization method based on multimodal data fusion includes at least steps S110 to S150, which are described in detail below:

[0035] In step S110, production data during the factory operation process is acquired through sensors.

[0036] In one embodiment of this application, temperature data, image streams, waveform data, and production logs acquired during factory operations are used as the production data. In addition, the production data may also include temperature sensor time series, pressure, industrial camera image streams, acoustic vibration signals, and process parameter log text, etc.

[0037] In this embodiment, sensors acquire data such as temperature and pressure during the factory operation process, resulting in time-series sensor data with a sampling frequency on the order of milliseconds. High-resolution video frames are acquired via an industrial camera image stream, with a lower sampling frequency, such as 10 frames per second. Acoustic vibration signals, such as high-frequency vibration waveform data, are acquired using sensors such as oscilloscopes, with a sampling rate on the order of kHz. Process parameter log text, such as parameter adjustment time points and discrete event records of operation commands, is obtained through the data stream acquisition.

[0038] In step S120, the production data is preprocessed to obtain first data, and a multimodal tensor is generated based on the data source and correlation of the first data.

[0039] In one embodiment of this application, the production data is cleaned and standardized to obtain first data, and a multimodal tensor is generated based on the data source that generated or collected the first data and the correlation between the first data.

[0040] As shown in Figure 2, in one embodiment of this application, the production data is preprocessed to obtain first data, and a multimodal tensor is generated based on the data source and correlation of the first data, including:

[0041] S210, perform data cleaning on the production data to obtain first data, and determine the correlation between the first data and the corresponding time vector;

[0042] S220, generate tensor nodes based on the data source corresponding to the first data, and generate tensor hyperedges according to the correlation between the first data and the time vector;

[0043] S230, Generate a multimodal tensor based on the tensor node and the tensor hyperedge.

[0044] In this embodiment, production data is cleaned and comprehensively reviewed to identify and process missing values. Based on data distribution characteristics and business logic, data integrity is ensured through mean imputation, median imputation, mode imputation, or imputation based on algorithmic models. Simultaneously, outliers are accurately located, and thresholds are set based on statistical methods or business rules to correct, remove, or retain outliers, which are then marked for subsequent analysis.

[0045] In addition, the preprocessing process can standardize data formats, regulate the representation of fields such as dates and times, and eliminate data misunderstandings and calculation errors caused by format differences; remove duplicate records by identifying duplicates based on key field combinations and retaining valid information; and correct data entry deviations such as spelling errors and inconsistent expressions, ensuring data consistency in semantics and expression. Through this series of rigorous data cleaning operations, accurate, complete, consistent, and analytically sound primary data is finally obtained, providing a solid and reliable data foundation for subsequent in-depth analysis and decision-making.

[0046] After obtaining the first data, tensor nodes are generated based on the data source corresponding to the first data. The tensor nodes include: v1 representing the temperature node corresponding to the temperature sensor time series, v2 representing the image node corresponding to the industrial camera image stream, v3 representing the vibration node corresponding to the acoustic vibration signal, and v4 representing the log node corresponding to the process parameter log text.

[0047] Based on the attributes of the first data, the relationships between the first data are determined. In this embodiment, the dependencies between multimodal data are defined by the following rules, including: a first time alignment constraint rule, for example, if an image frame corresponds to temperature data in a specific time period, then a hyperedge connection v1 (temperature node) and v2 (image node) are constructed; a spatial association constraint rule, for example, if a vibration signal is related to the location of a device, then a hyperedge connection v3 (vibration node) is constructed with the sensor node at the corresponding location; and a semantic association constraint rule, for example, v4 corresponding to a process parameter adjustment event may be associated with the responses of multiple sensors (v1, v2, and v3).

[0048] Based on the original timestamp of the first data, a vector representation of the original timestamp is generated. Based on the original timestamp and the vector representation, the time vector t′ mapped to hyperbolic space is generated as follows:

[0049] Where t represents the original timestamp, x represents the vector representation of the original timestamp, and ||x|| represents the Euclidean norm of vector x. By projecting the timestamp into a nonlinear space, long-period alignment errors are reduced, and the hierarchy and periodicity of time series are modeled more naturally.

[0050] After generating the association relationships and time vectors, a tensor hyperedge is generated based on the association relationships between the first data and the timestamps: e j =({v j 1, v j2 v jn}, w j ,t′)

[0051] After generating tensor nodes and tensor hyperedges, the tensors are combined to generate multimodal tensors.

[0052] The above process acquires heterogeneous data from multiple sources, including temperature data, image streams, waveform data, and production logs, from sensors. After data cleaning and time alignment, a multimodal tensor is constructed. This structured representation method preserves the spatiotemporal relationships between different modalities, providing a unified data foundation for subsequent analysis. Compared to traditional single-modal analysis, multimodal tensors can more comprehensively reflect the production status and avoid information loss.

[0053] In step S130, the multimodal tensor is decoupled, and the obtained image submodal data and acoustic submodal data are input into the defect recognition network to output defect features.

[0054] After generating the multimodal tensor, the multimodal tensor is decoupled to obtain image submodal data and acoustic submodal data. A defect recognition network is pre-constructed, and the image submodal data and acoustic submodal data are input into the defect recognition network to output defect features. In this embodiment, the defect features are used to represent the defects existing in the current process during industrial production.

[0055] In one embodiment of this application, the multimodal tensor is decoupled, and the resulting image submodal data and acoustic submodal data are input into a defect recognition network to output defect features, including:

[0056] The multimodal tensor is decoupled to separate image submodal data and acoustic submodal data;

[0057] A defect identification network is constructed based on an encoder, a generator, and a discriminator. The defect identification network is trained using mixed data consisting of normal samples and defective samples.

[0058] The image submodal data and acoustic submodal data are respectively input into the defect recognition network. Abnormal features are separated by reconstruction loss and distribution alignment loss, and defect features are output.

[0059] Image and acoustic submodal data are separated from the multimodal tensor. Specifically, image and acoustic signal subdata are separated from the spatiotemporally aligned tensor and standardized. Image data is resized to a uniform size and pixel values ​​are normalized to the [0, 1] interval. Acoustic signals undergo denoising processing, such as wavelet denoising, and are segmented into fixed-length temporal segments. This process ensures compatibility of input formats for different modal data, facilitating subsequent network processing.

[0060] A recognition network consisting of an encoder, a generator, and a discriminator is constructed. The encoder compresses the original data (image or acoustic signal) into low-dimensional latent variables; the generator reconstructs the original data from the latent variables, attempting to restore the input information; and the discriminator determines whether the latent variables conform to a preset prior distribution, such as a standard normal distribution. Optionally, the generator and discriminator share some weight parameters of intermediate layers, such as convolutional kernels or fully connected layers. Sharing parameters reduces model complexity and forces the generator and discriminator to learn consistent feature representations, avoiding feature conflicts.

[0061] The defect identification network is trained using a mixture of normal and defective samples. During this process, the encoder maps the input data to a latent space, generating a latent variable distribution. The latent variables of normal samples cluster tightly in the central region of the prior distribution; the latent variables of defective samples deviate from the central region, becoming marginalized.

[0062] The generator produces reconstructed data based on latent variables. Reconstruction errors are small with normal data but significantly larger with anomalous data. If the input is a defective image, the generator may fail to reproduce the defect details, resulting in a noticeable difference between the reconstructed image and the original image. Anomalies are indirectly identified by the magnitude of the reconstruction error, while the distribution shift of the latent variables directly reflects the anomaly pattern.

[0063] The discriminator learns to distinguish between latent variables generated by the encoder and samples randomly sampled from the prior distribution, ensuring that the distribution of latent variables output by the encoder approximates the prior distribution and guarantees the structural consistency of the latent space. The generator reconstructs the data during adversarial training, retaining key features such as defects while ignoring irrelevant noise. The latent variables retain the core information of the input data, such as defect features, and conform to a standardized distribution, facilitating processing by subsequent modules. Through adversarial training, the model's sensitivity to subtle defects (such as small scratches or low-frequency abnormal vibrations) is significantly improved.

[0064] In this embodiment, intermediate features are generated by a discriminator, and reconstructed data is obtained by a generator based on these intermediate features. Then, a reconstruction loss is generated based on the reconstructed data and the input data. A latent variable distribution is defined by an encoder, and the difference between the two distributions is measured based on the KL divergence between the latent variable distribution and the prior distribution, serving as the distribution alignment loss. Finally, the loss function is determined using the reconstruction loss and the distribution alignment loss: L AAE =||G(D(x))-x|| 2 +KL(q(z|x)||p(z))

[0065] Where x represents the input data, such as an image or acoustic signal; G(D(x)) represents the reconstructed data of the generator based on the intermediate features of the discriminator; q(z|x) represents the latent variable distribution defined by the encoder, usually assumed to be Gaussian; p(z) represents the prior distribution, such as the standard Gaussian distribution; and KL(·||·) represents the KL divergence, which is used to measure the difference between two distributions.

[0066] Subsequently, based on the comparison between the loss function value and a preset threshold, abnormal features are separated to obtain defect features. The latent variable distributions of the image and acoustic signal branches are fused across modally using reconstruction loss and distribution alignment loss. An attention mechanism is used to weight the fusion of the latent variable distributions of the two modalities, highlighting common features related to defects. For example, if an image shows surface cracks within a certain time period, and the acoustic signal exhibits high-frequency vibrations, the fused features will strengthen this abnormal association, thus outputting unified defect features. Through joint training of images and acoustic signals, complex defects that are difficult to detect with a single modality can be identified, such as latent faults that appear normal visually but exhibit abnormal vibrations.

[0067] The above process employs a defect recognition network with an encoder, generator, and discriminator architecture. Through training on normal and defective samples, it learns feature representations to distinguish abnormal patterns. The reconstruction loss ensures the network accurately reconstructs normal samples while introducing larger errors for defective samples; the distribution alignment loss makes the latent features conform to a predetermined distribution, facilitating anomaly detection. Compared to traditional thresholding methods, this adversarial training mechanism can discover more subtle defect patterns and is more robust to noise.

[0068] In step S140, a Bayesian causal graph is constructed based on defect features, physical features representing the physical state of the process in the production data, and environmental features. The adjustment parameters of the production process are then output through the Bayesian causal graph.

[0069] In one embodiment of this application, a Bayesian causal graph is constructed based on defect features, physical features representing the physical state of the process in production data, and environmental features. Defect features characterize abnormal product features, such as surface defect image features and abnormal vibration waveforms. Physical features reflect the physical state of the process, such as temperature distribution and pressure changes. Environmental features represent the characteristics of the external environment and the long-term operating state of the equipment, such as ambient temperature and humidity and equipment aging trends. Then, adjustment parameters for the production process are output through the Bayesian causal graph.

[0070] In one embodiment of this application, a Bayesian causal graph is constructed based on defect features, physical features representing the physical state of the process in production data, and environmental features. Adjustment parameters for the production process are output through the Bayesian causal graph, including:

[0071] Based on defect features, physical features representing the physical state of the process in production data, and environmental features, a Bayesian causal graph is constructed.

[0072] The parameters in the Bayesian causal graph are simulated and adjusted to obtain the influence parameters of the simulated and adjusted parameters on the relationship chains in the Bayesian causal graph.

[0073] Based on the aforementioned influencing parameters, the parameters in the Bayesian causal graph are adjusted to obtain the adjustment parameters for the production process.

[0074] A Bayesian causal graph is constructed based on defect features, physical features, and environmental features. Specifically, the Bayesian causal graph consists of nodes, edges, and parameters. Nodes contain all feature dimensions (such as temperature sensor data and defect type) and process parameters (such as heating power and cooling rate). Edges represent causal relationships, such as an increase in heating power leading to an increase in temperature, which in turn leads to a reduction in defects. Parameters define conditional probability distributions, such as a decrease in defect probability when the temperature increases.

[0075] Next, verify whether the causal relationship between features and parameters holds. For example, when the equipment temperature is known, test whether the ambient humidity is independent of the defect features. If new latent variables are found, such as unmonitored internal wear and tear of the equipment, dynamically add nodes and update the edge structure to ensure that the Bayesian causal graph covers both explicit and implicit factors in real production and avoids missing key causal relationships.

[0076] The parameters in the Bayesian causal graph are simulated and adjusted to obtain the influence parameters of the adjusted parameters on the relationship chains in the Bayesian causal graph. Assume a virtual adjustment is made to a certain parameter, such as increasing the heating power by 5%, while keeping other parameters unchanged. Based on the structural equation of the Bayesian causal graph, simulate how the adjusted parameter affects quality indicators, such as yield rate, through causal relationship chains, and obtain the influence parameters of the adjusted parameter on the causal relationship chains, such as the numerical change in yield rate. This process isolates the causal effect of a single parameter adjustment and eliminates interference from confounding factors such as environmental noise.

[0077] Based on the aforementioned influencing parameters, the marginal impact of each parameter on the quality index is calculated. The parameters in the Bayesian causality diagram are adjusted to obtain the adjustment parameter Δp for the production process. i for:

[0078] in, This indicates that the preset index Y corresponds to the i-th parameter p. i The sensitivity to change, Pa(Y) represents the set of causal parent nodes directly related to the preset index Y in the Bayesian causal graph, and ∫·dY represents the integral operation on Y.

[0079] This method generates parameter adjustment directions and ranges that conform to actual production conditions, ensuring the safety and feasibility of the optimization plan. For example, a 1% increase in heating power is expected to improve the yield rate by 0.3%. If the parameter adjustment has a positive impact on quality indicators, such as an increase in yield rate, it is recommended to increase the parameter; if it has a negative impact, such as a sharp increase in energy consumption, it is recommended to decrease the parameter.

[0080] The Bayesian causal graph constructed through the above process explicitly models the causal relationships between defect characteristics, physical characteristics, and environmental characteristics. By analyzing the impact of parameter adjustments on quality indicators through virtual intervention, key process parameters can be identified. Compared to correlation-based methods, this causal reasoning avoids spurious associations and ensures the scientific validity of adjustment recommendations.

[0081] In step S150, a process adjustment plan is generated based on the adjustment parameters and the production data.

[0082] After obtaining the adjustment parameters and production data, a process adjustment plan is generated based on the current production status and the status that needs to be adjusted.

[0083] In one embodiment of this application, a process adjustment plan is generated based on the adjustment parameters and the production data, including:

[0084] Based on the adjustment parameters and the production data, candidate parameter combinations are generated, and the objective function and constraints are determined according to actual production needs.

[0085] Based on the candidate parameter combinations, diverse initial parameter combinations are derived. According to the objective function and constraints, the optimal parameter combination is searched from the initial parameter combinations.

[0086] A process adjustment scheme is generated based on the optimal parameter combination.

[0087] Based on the adjusted parameters and real-time production data, the correlation and adjustment range between parameters are first analyzed. Combined with equipment limitations (such as maximum temperature and pressure thresholds) and resource constraints (such as energy consumption budget), candidate parameter combinations covering different adjustment directions are generated. These combinations are injected with diversity through algorithms to avoid overly concentrated initial solutions. Simultaneously, multi-objective functions (such as reducing energy consumption, improving yield, and extending equipment life) and hard constraints (such as safety thresholds) are defined according to production needs to construct a mathematical model of the optimization problem, thus defining the solution space and evaluation criteria for subsequent searches.

[0088] Candidate parameters are encoded as quantum superposition states, and a massive number of parameter combinations are explored in parallel using the quantum annealing algorithm. By dynamically adjusting the coupling relationships between parameters, such as the synergistic effect of temperature and pressure, regions sensitive to the objective function are prioritized for searching. During the search process, combinations that violate constraints, such as overheating schemes, are eliminated in real time, and the remaining solutions are non-dominated and sorted to retain a Pareto optimal solution set that is balanced across multiple objectives.

[0089] The optimal parameter combination is decoded to obtain specific process instructions, such as setting the temperature to 480℃ and adjusting the pressure gradient, and these instructions are prioritized, such as prioritizing yield. The execution effect of the solution is simulated using a digital twin module to verify its stability and feasibility in a virtual environment. The verified solution is then converted into control instructions and issued to the production line for execution. Simultaneously, actual operating data is fed back to the optimization system, forming a closed loop of optimization-verification-execution-feedback, continuously improving the adaptability and reliability of process parameters.

[0090] The above process searches for the optimal combination of parameters while considering real-time operating constraints. This process balances multiple conflicting objectives and, compared to single-objective optimization, provides more comprehensive decision support, such as simultaneously providing multiple feasible solutions for high-quality, high-consumption and low-quality, energy-saving approaches.

[0091] In one embodiment of this application, after generating a process adjustment plan based on the adjustment parameters and the production data, the method further includes: forwarding the process adjustment plan to a control terminal and displaying it on the interface of the control terminal.

[0092] In one embodiment of this application, the final generated process adjustment scheme, such as temperature setpoints and pressure parameters, is transmitted in real time to the control terminal of the production line via an industrial communication protocol, and dynamically displayed on the human-machine interface of the terminal in the form of structured tables and trend charts.

[0093] Optionally, the interface will highlight the adjustment range of key parameters, such as heating power +5%. It will also provide an operation confirmation button and an explanation of the reason for the adjustment, such as to improve the yield rate.

[0094] Optionally, the control terminal automatically verifies whether the parameters are within the safe range of the equipment. If the parameters exceed the limit, an alarm is triggered, and the parameters are updated after manual confirmation, ensuring that the adjustment process is safe and controllable.

[0095] Optionally, after execution, the actual effect data of the new parameters, such as the real-time temperature curve, can be synchronously fed back to the interface to form a closed-loop monitoring system.

[0096] The final process adjustment plan is transmitted to the control terminal via a standardized interface, displaying the adjustment content and its basis in a visual format. The system continuously monitors the implementation effect, forming a closed loop of optimization-execution-feedback. This dynamic adjustment mechanism allows process parameters to adapt to changes in the production environment, improving system adaptability compared to fixed parameter settings.

[0097] This application's technical solution acquires production data during factory operations through sensors; preprocesses the production data to obtain first data; generates a multimodal tensor based on the data source and correlation of the first data; decouples the multimodal tensor, inputs the obtained image submodal data and acoustic submodal data into a defect recognition network, and outputs defect features; constructs a Bayesian causal graph based on the defect features, physical features representing the physical state of the process in the production data, and environmental features; outputs adjustment parameters for the production process through the Bayesian causal graph; and generates a process adjustment scheme based on the adjustment parameters and the production data. By constructing a spatiotemporally aligned multimodal tensor through multimodal data fusion, efficient fusion of heterogeneous data is achieved. The Bayesian causal graph reveals the causal relationship between process parameters and quality indicators, avoiding the spurious correlation defects of correlation analysis. Finally, a process adjustment scheme is generated through multi-objective optimization, forming a closed-loop optimization from data perception to decision execution, enabling process parameters to have dynamic adaptive capabilities and improving the accuracy and yield of process optimization.

[0098] The following describes an apparatus embodiment of this application, which can be used to execute the smart factory process optimization method based on multimodal data fusion in the above embodiments of this application. It is understood that the apparatus can be a computer program (including program code) running on a computer device, for example, the apparatus is an application software; the apparatus can be used to execute the corresponding steps in the method provided in the embodiments of this application. For details not disclosed in the apparatus embodiments of this application, please refer to the embodiments of the smart factory process optimization method based on multimodal data fusion described above.

[0099] Figure 3 shows a block diagram of a smart factory process optimization system based on multimodal data fusion according to an embodiment of this application.

[0100] Referring to FIG3, a smart factory process optimization system based on multimodal data fusion according to an embodiment of this application includes:

[0101] Acquisition unit 310 is used to acquire production data during the factory operation process through sensors;

[0102] Tensor unit 320 is used to preprocess the production data to obtain first data, and generate a multimodal tensor based on the data source and correlation of the first data;

[0103] The detection unit 330 is used to decouple the multimodal tensor, input the obtained image submodal data and acoustic submodal data into the defect recognition network, and output defect features;

[0104] The adjustment unit 340 is used to construct a Bayesian causal graph based on defect features, physical features representing the physical state of the process in the production data, and environmental features, and output the adjustment parameters of the production process through the Bayesian causal graph.

[0105] Solution unit 350 is used to generate a process adjustment solution based on the adjustment parameters and the production data.

[0106] In this application, based on the aforementioned scheme, the step of acquiring production data during the factory operation process through sensors includes: acquiring temperature data, image streams, waveform data, and production logs during the factory operation process through sensors, as the production data.

[0107] In this application, based on the aforementioned scheme, the step of preprocessing the production data to obtain first data and generating a multimodal tensor according to the data source and correlation of the first data includes: cleaning the production data to obtain the first data, and determining the correlation between the first data and the corresponding time vector; generating tensor nodes based on the data source corresponding to the first data, and generating tensor hyperedges according to the correlation between the first data and the time vector; and generating a multimodal tensor according to the tensor nodes and the tensor hyperedges.

[0108] In this application, based on the aforementioned scheme, the step of decoupling the multimodal tensor and inputting the obtained image submodal data and acoustic submodal data into the defect recognition network to output defect features includes: decoupling the multimodal tensor to separate image submodal data and acoustic submodal data; constructing a defect recognition network based on an encoder, generator, and discriminator, and training the defect recognition network with mixed data consisting of normal samples and defect samples; inputting the image submodal data and acoustic submodal data into the defect recognition network respectively, separating abnormal features through reconstruction loss and distribution alignment loss, and outputting defect features.

[0109] In this application, based on the aforementioned scheme, the step of constructing a Bayesian causal graph based on defect features, physical features representing the physical state of the process in production data, and environmental features, and outputting adjustment parameters of the production process through the Bayesian causal graph, includes: constructing a Bayesian causal graph based on defect features, physical features representing the physical state of the process in production data, and environmental features; simulating and adjusting the parameters in the Bayesian causal graph to obtain the influence parameters of the simulated and adjusted parameters on the relationship chains in the Bayesian causal graph; and adjusting the parameters in the Bayesian causal graph based on the influence parameters to obtain the adjustment parameters of the production process.

[0110] In this application, based on the aforementioned scheme, the step of generating a process adjustment scheme according to the adjustment parameters and the production data includes: generating candidate parameter combinations based on the adjustment parameters and the production data, and determining the objective function and constraints according to actual production needs; deriving diverse initial parameter combinations based on the candidate parameter combinations, and searching for the optimal parameter combination from the initial parameter combinations according to the objective function and constraints; and generating a process adjustment scheme based on the optimal parameter combination.

[0111] In this application, based on the aforementioned scheme, after generating the process adjustment scheme according to the adjustment parameters and the production data, the method further includes: forwarding the process adjustment scheme to the control terminal and displaying it on the interface of the control terminal.

[0112] This application's technical solution acquires production data during factory operations through sensors; preprocesses the production data to obtain first data; generates a multimodal tensor based on the data source and correlation of the first data; decouples the multimodal tensor, inputs the obtained image submodal data and acoustic submodal data into a defect recognition network, and outputs defect features; constructs a Bayesian causal graph based on the defect features, physical features representing the physical state of the process in the production data, and environmental features; outputs adjustment parameters for the production process through the Bayesian causal graph; and generates a process adjustment scheme based on the adjustment parameters and the production data. By constructing a spatiotemporally aligned multimodal tensor through multimodal data fusion, efficient fusion of heterogeneous data is achieved. The Bayesian causal graph reveals the causal relationship between process parameters and quality indicators, avoiding the spurious correlation defects of correlation analysis. Finally, a process adjustment scheme is generated through multi-objective optimization, forming a closed-loop optimization from data perception to decision execution, enabling process parameters to have dynamic adaptive capabilities and improving the accuracy and yield of process optimization.

[0113] Figure 4 shows a schematic diagram of the structure of a computer system suitable for implementing the electronic device of the present application.

[0114] It should be noted that the computer system of the electronic device in this embodiment is only an example and should not impose any limitations on the function and scope of use of the embodiments of this application.

[0115] In this embodiment, the computer system includes a central processing unit 401, which can perform various appropriate actions and processes based on a program stored in a read-only memory 402 or a program loaded from a storage section 408 into a random access memory 403, such as executing the smart factory process optimization method based on multimodal data fusion described in the above embodiment. The random access memory 403 also stores various programs and data required for system operation. The central processing unit 401, the read-only memory 402, and the random access memory 403 are interconnected via a bus 404. An input / output interface 405 is also connected to the bus 404.

[0116] The following components are connected to the input / output interface 405: an input section 406 including a keyboard, mouse, etc.; an output section 407 including a cathode ray tube (CRT), liquid crystal display (LCD), etc., and speakers, etc.; a storage section 408 including a hard disk, etc.; and a communication section 409 including a network interface card such as a LAN (Local Area Network) card, modem, etc. The communication section 409 performs communication processing via a network such as the Internet. A drive 410 is also connected to the input / output interface 405 as needed. A removable medium 411, such as a disk, optical disk, magneto-optical disk, semiconductor memory, etc., is installed on the drive 410 as needed so that computer programs read from it can be installed into the storage section 408 as needed.

[0117] Specifically, according to embodiments of this application, the processes described above with reference to the flowcharts can be implemented as computer software programs. For example, embodiments of this application include a computer program product comprising a computer program carried on a computer-readable medium, the computer program including a computer program for performing the methods shown in the flowcharts. In such embodiments, the computer program can be downloaded and installed from a network via communication section 409, and / or installed from removable medium 411. When the computer program is executed by central processing unit 401, it performs various functions defined in the system of this application.

[0118] It should be noted that the computer-readable medium shown in the embodiments of this application can be a computer-readable signal medium or a computer-readable storage medium, or any combination of the two. A computer-readable storage medium can be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination thereof. More specific examples of a computer-readable storage medium may include, but are not limited to: an electrical connection having one or more wires, a portable computer disk, a hard disk, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM), flash memory, optical fiber, portable compact disc read-only memory (CD-ROM), optical storage device, magnetic storage device, or any suitable combination thereof. In this application, a computer-readable storage medium can be any tangible medium containing or storing a program that can be used by or in conjunction with an instruction execution system, apparatus, or device. In this application, a computer-readable signal medium can include a data signal propagated in baseband or as part of a carrier wave, carrying a computer-readable computer program. The transmitted data signal can take various forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination thereof. The computer-readable signal medium can also be any computer-readable medium other than a computer-readable storage medium, which can send, propagate, or transmit a program for use by or in connection with an instruction execution system, apparatus, or device. The computer program contained on the computer-readable medium can be transmitted using any suitable medium, including but not limited to wireless, wired, etc., or any suitable combination thereof.

[0119] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of this application. Each block in a flowchart or block diagram may represent a module, segment, or portion of code, which contains one or more executable instructions for implementing a specified logical function. It should also be noted that in some alternative implementations, the functions indicated in the blocks may occur in a different order than those indicated in the drawings. For example, two consecutively indicated blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in a block diagram or flowchart, and combinations of blocks in a block diagram or flowchart, can be implemented using a dedicated hardware-based system that performs the specified function or operation, or using a combination of dedicated hardware and computer instructions.

[0120] The units described in the embodiments of this application can be implemented in software or hardware, and the described units can also be located in a processor. The names of these units do not necessarily limit the specific unit itself.

[0121] According to one aspect of this application, a computer program product or computer program is provided, comprising computer instructions stored in a computer-readable storage medium. A processor of a computer device reads the computer instructions from the computer-readable storage medium and executes the computer instructions, causing the computer device to perform the methods provided in the various alternative implementations described above.

[0122] In another aspect, this application also provides a computer-readable medium, which may be included in the electronic device described in the above embodiments; or it may exist independently and not assembled into the electronic device. The computer-readable medium carries one or more programs, which, when executed by the electronic device, cause the electronic device to implement the smart factory process optimization method based on multimodal data fusion described in the above embodiments.

[0123] It should be noted that although several modules or units for the device used to perform actions have been mentioned in the detailed description above, this division is not mandatory. In fact, according to the embodiments of this application, the features and functions of two or more modules or units described above can be embodied in one module or unit. Conversely, the features and functions of one module or unit described above can be further divided and embodied by multiple modules or units.

[0124] Through the above description of the embodiments, those skilled in the art will readily understand that the exemplary embodiments described herein can be implemented by software or by combining software with necessary hardware. Therefore, the technical solutions according to the embodiments of this application can be embodied in the form of a software product, which can be stored in a non-volatile storage medium (such as a CD-ROM, USB flash drive, external hard drive, etc.) or on a network, including several instructions to cause a computing device (such as a personal computer, server, touch terminal, or network device, etc.) to execute the method according to the embodiments of this application.

[0125] Other embodiments of this application will readily occur to those skilled in the art upon consideration of the specification and practice of the embodiments disclosed herein. This application is intended to cover any variations, uses, or adaptations of this application that follow the general principles of this application and include common knowledge or customary techniques in the art not disclosed herein.

[0126] It should be understood that this application is not limited to the precise structure described above and shown in the accompanying drawings, and various modifications and changes can be made without departing from its scope. The scope of this application is limited only by the appended claims.

Claims

1. A method for optimizing intelligent factory processes based on multimodal data fusion, characterized in that, include: Acquire production data during factory operations using sensors; The production data is preprocessed to obtain first data, and a multimodal tensor is generated based on the data source and correlation of the first data. The multimodal tensor is decoupled, and the resulting image submodal data and acoustic submodal data are input into the defect recognition network to output defect features; Based on defect features, physical features representing the physical state of the process in production data, and environmental features, a Bayesian causal graph is constructed, and the adjustment parameters of the production process are output through the Bayesian causal graph. Based on the adjustment parameters and the production data, a process adjustment plan is generated.

2. The intelligent factory process optimization method based on multimodal data fusion according to claim 1, characterized in that, The acquisition of production data during factory operations via sensors includes: Temperature data, image streams, waveform data, and production logs obtained through sensors during factory operations are used as the production data.

3. The intelligent factory process optimization method based on multimodal data fusion according to claim 1, characterized in that, The step of preprocessing the production data to obtain first data, and generating a multimodal tensor based on the data source and correlation of the first data, includes: The production data is cleaned to obtain first data, and the correlation between the first data and the corresponding time vector are determined. Tensor nodes are generated based on the data source corresponding to the first data, and tensor hyperedges are generated according to the correlation between the first data and the time vector. A multimodal tensor is generated based on the tensor nodes and the tensor hyperedges.

4. The intelligent factory process optimization method based on multimodal data fusion according to claim 1, characterized in that, The process of decoupling the multimodal tensor, inputting the resulting image submodal data and acoustic submodal data into the defect recognition network, and outputting defect features includes: The multimodal tensor is decoupled to separate image submodal data and acoustic submodal data; A defect identification network is constructed based on an encoder, a generator, and a discriminator. The defect identification network is trained using mixed data consisting of normal samples and defective samples. The image submodal data and acoustic submodal data are respectively input into the defect recognition network. Abnormal features are separated by reconstruction loss and distribution alignment loss, and defect features are output.

5. The intelligent factory process optimization method based on multimodal data fusion according to claim 1, characterized in that, Based on defect features, physical features representing the physical state of the process in production data, and environmental features, a Bayesian causal graph is constructed. The adjustment parameters for the production process are output through this Bayesian causal graph, including: Based on defect features, physical features representing the physical state of the process in production data, and environmental features, a Bayesian causal graph is constructed. The parameters in the Bayesian causal graph are simulated and adjusted to obtain the influence parameters of the simulated and adjusted parameters on the relationship chains in the Bayesian causal graph. Based on the aforementioned influencing parameters, the parameters in the Bayesian causal graph are adjusted to obtain the adjustment parameters for the production process.

6. The intelligent factory process optimization method based on multimodal data fusion according to claim 1, characterized in that, The step of generating a process adjustment plan based on the adjustment parameters and the production data includes: Based on the adjustment parameters and the production data, candidate parameter combinations are generated, and the objective function and constraints are determined according to actual production needs. Based on the candidate parameter combinations, diverse initial parameter combinations are derived. According to the objective function and constraints, the optimal parameter combination is searched from the initial parameter combinations. A process adjustment scheme is generated based on the optimal parameter combination.

7. The intelligent factory process optimization method based on multimodal data fusion according to claim 1, characterized in that, After generating the process adjustment plan based on the adjustment parameters and the production data, the process further includes: The process adjustment plan is forwarded to the control terminal and displayed on the control terminal interface.

8. A smart factory process optimization system based on multimodal data fusion, characterized in that, include: The acquisition unit is used to acquire production data during the factory operation process through sensors; Tensor units are used to preprocess the production data to obtain first data, and generate multimodal tensors based on the data source and correlation of the first data; The detection unit is used to decouple the multimodal tensor, input the obtained image submodal data and acoustic submodal data into the defect recognition network, and output defect features; The adjustment unit is used to construct a Bayesian causal graph based on defect features, physical features representing the physical state of the process in the production data, and environmental features, and output the adjustment parameters of the production process through the Bayesian causal graph. The scheme unit is used to generate a process adjustment scheme based on the adjustment parameters and the production data.

9. The intelligent factory process optimization system based on multimodal data fusion according to claim 8, characterized in that, The acquisition of production data during factory operations via sensors includes: Temperature data, image streams, waveform data, and production logs obtained through sensors during factory operations are used as the production data.

10. The intelligent factory process optimization system based on multimodal data fusion according to claim 8, characterized in that, The step of preprocessing the production data to obtain first data, and generating a multimodal tensor based on the data source and correlation of the first data, includes: The production data is cleaned to obtain first data, and the correlation between the first data and the corresponding time vector are determined. Tensor nodes are generated based on the data source corresponding to the first data, and tensor hyperedges are generated according to the correlation between the first data and the time vector. A multimodal tensor is generated based on the tensor nodes and the tensor hyperedges.