Method and system for fast generation of few-shot chinese fonts on basis of diffusion model

Abstract

A method and system for the fast generation of few-shot Chinese fonts on the basis of a diffusion model. The system comprises a low-resolution generative model module and a super-resolution model module, and may further comprise a diffusion model distillation module. The method comprises: constructing a conditional latent diffusion model having a component-aware encoder, i.e., a low-resolution generation model, so as to generate high-quality low-resolution fonts; constructing a super-resolution model based on a cascaded diffusion model, so as to extend a low-resolution generation result to a high-resolution result; and the diffusion model distillation module being further capable of distilling the trained low-resolution generation model and super-resolution model into a one-step generation model, so as to improve the generation efficiency of a glyph image. The method can extract and transfer font styles from limited input glyph image samples, producing glyph image generation results with significant improvements in both style consistency and glyph structure accuracy, and obtaining high-quality vectorized glyphs by means of a vectorization tool.

Description

A Fast Method and System for Few-Shot Chinese Font Generation Based on Diffusion Model Technical Field

[0001] This invention belongs to the fields of computer graphics processing technology and artificial intelligence technology, and relates to Chinese font generation technology. In particular, it relates to a method and system for rapid generation of high-resolution Chinese fonts with few samples based on a diffusion model, which can generate a complete Chinese character library for a given new font with a limited number of input glyph samples. Background Technology

[0002] Font generation refers to the task of generating a complete font library for a given set of input glyphs. Automated font generation reduces the time-consuming and labor-intensive human burden, thus having broad application prospects and attracting widespread attention in academia. Especially for writing systems containing tens of thousands of characters, such as Chinese, Japanese, or Korean, fast and efficient few-sample font generation methods can not only accelerate the design of high-quality commercial font products, but also be applied to application scenarios such as personalized handwriting generation, scene text image editing, and data enhancement for optical character recognition (OCR).

[0003] Existing font generation methods generally fall into two categories: directly generating vector glyphs, or first generating high-quality rasterized images and then vectorizing them into vector glyphs. This study focuses on the second type of method, namely, developing a font generation system that takes a small number of glyph images of the target style as input and outputs high-quality, high-resolution glyph images of the remaining characters in the font library. High-quality vector glyphs are then obtained using existing vectorization tools. Compared to methods that directly generate vector glyphs, although processing high-resolution images involves higher computational complexity and still requires manual intervention during the vectorization process, image modalities with relatively more structured information are easier to process, and there are already a large number of mature and advanced generation models available for use and reference. On the other hand, methods that directly generate vector glyphs can currently only handle simple shapes, which limits their application in font generation for text systems containing glyphs with high topological complexity (such as Chinese characters). However, if sufficiently high-quality image modal generation results can be obtained, then high-quality, high-complexity vector glyph results can be obtained through mature vectorization tools.

[0004] Despite recent advancements in font generation technology, few-shot font generation remains an unsolved and highly challenging task due to the complexity of character structures and the creativity and diversity of font designs. Most previous methods have built upon Generative Adversarial Networks (GANs), training image-to-image translation networks in an adversarial manner. However, GANs are notoriously difficult to train and scale, suffer from mode collapse, and lack diversity, limiting their potential for high-quality generation. On the other hand, in image modalities, diffusion models have proven to produce better results than state-of-the-art GANs and can easily scale to larger datasets. However, the iterative inference process of diffusion models makes them more time-consuming than single-step inference GANs, hindering their application in real-world scenarios. Furthermore, in addition to diffusion models, it is necessary to design how to appropriately integrate prior knowledge of glyphs into the model architecture.

[0005] Diffusion-based generative models have recently achieved unparalleled performance in many image synthesis tasks and other modalities such as video, audio, and text generation. Latent Diffusion Models, a variant of diffusion models that greatly reduces computational costs, have been widely used in various conditional generation tasks, including image editing and text-to-image generation. Cascaded diffusion models, a variant of diffusion models that extends this to high-resolution generation, successively upsample the generated images to higher resolutions and have been successfully applied to various tasks that require high-resolution generation. The literature documents Diff-Font (He, Haibin, et al. "Diff-Font: Diffusion Model for Robust One-Shot Font Generation." arXiv preprint arXiv:2212.05895. 2022.) and FontDiffuser (Yang, Zhenhua, et al. "FontDiffuser: One-Shot Font Generation via Denoising Diffusion with Multi-Scale Content Aggregation and Style Contrastive Learning." arXiv preprint arXiv:2312.12142. 2023.) have attempted to apply diffusion models to font generation tasks and achieved impressive results. While these methods can leverage the generative capabilities of diffusion models to produce clear results, the pre-trained style features or style learning strategies they use often fail when faced with unknown and complex font styles. Furthermore, their pixel-space-based model construction and multi-step sampling limit their practical application and scalability to higher resolutions.

[0006] Summary of the Invention

[0007] To overcome the shortcomings of the existing technologies, this invention proposes a font generation method and system based on a diffusion model, which supports one-step font generation and high-resolution font generation, and achieves high-quality, high-speed, and high-resolution few-sample font generation.

[0008] First, this invention incorporates prior knowledge about component decomposition, which is highly useful for font style learning and transfer. It utilizes an attention mechanism to reuse component-level style information and leverages the powerful generative capabilities of the diffusion model to learn how to effectively transfer diverse and complex font styles to the target glyph. Furthermore, to adapt to smaller input sample sizes in practical applications (e.g., less than 10 samples), this invention introduces a reference sample selection strategy, enabling the model to extract both global and local style information simultaneously. Second, to further achieve higher resolution generation, this invention designs a super-resolution process, allowing the loss of detail style and shape at low resolution to be recovered during the super-resolution process, thereby narrowing the gap between image modalities and vector modalities, thus achieving high-quality, high-speed, and high-resolution few-sample font generation. Finally, to address the slow inference speed of the diffusion model, this invention applies fractional distillation sampling technology from text-to-3D generation techniques, achieving efficient one-step generation with almost no quality loss.

[0009] For ease of explanation, the following terms are defined in this invention:

[0010] Token: The smallest unit of language processed by the attention mechanism.

[0011] Auto-encoder: self-encoder.

[0012] U-Net is a convolutional-based deep learning model architecture proposed in the paper (Ronneberger, Olaf, et al. "U-net: Convolutional networks for biomedical image segmentation." Medical image computing and computer-assisted intervention–MICCAI 2015: 18th international conference, Munich, Germany, October 5-9, 2015, proceedings, part III 18. Springer International Publishing, 2015.).

[0013] Score Distillation Sampling (SDS): A score distillation sampling method proposed in the literature (Poole, Ben, et al. "Dreamfusion: Text-to-3d using 2d diffusion." arXiv preprint arXiv:2209.14988 (2022).).

[0014] The technical solution provided by this invention is as follows:

[0015] A method and system for rapid generation of high-resolution Chinese fonts with few samples based on a diffusion model. The system proposed in this invention comprises three parts: the first part includes a conditional implicit diffusion model with a component-aware encoder, used to generate high-quality low-resolution results (64×64 resolution); the second part includes a conditional super-resolution cascaded diffusion model constructed based on cascaded diffusion model super-resolution technology, used for super-resolution generation of high-resolution glyphs; the third part includes a diffusion model distillation technique, used to distill the trained conditional implicit diffusion model with a component-aware encoder and the conditional super-resolution cascaded diffusion model into a one-step generation model.

[0016] The first part constructs a conditional implicit diffusion model with a component-aware encoder based on the conditional diffusion model and the component-aware encoder, to achieve high-quality low-resolution font generation.

[0017] Based on the core idea of ​​the diffusion model, this invention employs a method of progressively adding noise to real data until the data is transformed into pure noise, and then using a network that learns to gradually transform the pure noise back into real data to estimate the distribution of the real data. Specifically, given a reference image x of character content rendered using a standard font (such as sans-serif), c and k style reference images from the target font. The goal is to generate a glyph image that matches the target font style and character content. This invention uses a component-aware encoder to encode information from the input content and style reference image, ultimately obtaining glyph content and style encoded features. And utilizing the principle of implicit diffusion model, a conditional implicit diffusion model is used. (That is, the constructed low-resolution generative model) estimates the distribution of the target generated image, ultimately achieving the goal of generating the target image based on style features (as conditional input). Let x0 be the glyph target image, and z0 be the implicit representation of the glyph target image encoded into the latent space defined by the autoencoder in the conditional implicit diffusion model. The corresponding diffusion model training loss is as follows: ,in, Let y be Gaussian noise, y be the glyph content and style encoding features, and z be Gaussian noise.t The result of adding noise to z0 to the noise intensity t is, i.e. , and Let be a series of non-zero variables, where t is the noise intensity, taking values ​​between 0 and T, where T=1000. For expectations; The distance is L2.

[0018] Furthermore, in multi-step diffusion model sampling scenarios, classifier-free guidance—a method that guides samples to move towards high-density regions of a conditionally determined distribution—is often used to leverage diversity for higher quality. This invention defines a classifier-free guidance method for two conditional inputs: content and style. The corresponding noise is estimated as follows, where the two coefficients... and The correlation strength between the generated result and the input content and style conditions is controlled separately; expressed as: ,in To be The result of replacing it with a vector of all zeros of the same size, To be The result of replacing it with a vector of all zeros of the same size, This is the noise estimation result without classifier guidance.

[0019] Since font style exists at different structural levels of the glyph image, and different characters share different common style information, ideally, the model should focus on different relevant parts of the style reference at different structural levels of the input content. This invention employs a multi-head cross-attention mechanism to achieve this goal. Specifically, the part-aware encoder consists of a style encoder. A content encoder It consists of a style encoder and a cross-attention module. The style encoder encodes each of the k style reference images into style features. The content encoder encodes the content reference image into content features. Each style feature is represented as a dimension of of Each token represents the content features as a dimension. of Each token is represented as follows: A cross-attention operation is performed on the tokens, resulting in:

[0020] in Let be the projection matrix corresponding to the i-th attention mechanism; y is the glyph content and style encoding features; Attention is the attention mechanism; softmax is a function that maps any real vector to a probability distribution; Q is the query matrix; K is the key matrix; and V is the value matrix. To be Features obtained by flattening To be The features obtained by concatenating and flattening are then processed. In this way, through an attention mechanism, each fine-grained location on the content feature map learns to pay attention to relevant locations on the style feature map, combining related style features to form the final conditional input.

[0021] To guide this learning process, we provide the style encoder with pre-selected style references corresponding to the components and strokes contained in the content characters. This invention defines a set of character sets containing 775 commonly used Chinese characters that cover all Chinese character encoding standards (GB2312 standard) and their corresponding components and strokes as the style reference set, serving as the model training set. For any character in the training set, its corresponding style reference is selected as a character in the style reference set containing that character's corresponding component or stroke, thereby guiding the model to learn component- and stroke-level style extraction and transfer in a targeted manner.

[0022] The second part describes the construction of a conditional super-resolution cascaded diffusion model based on cascaded diffusion model super-resolution technology to achieve high-resolution font generation.

[0023] The motivation for shifting the generation process to higher resolution regions is twofold: first, a large amount of information is lost at low resolution, such as stylistic details or dense strokes, so an upsampling process is needed to recover this lost information; second, upsampling can reduce the gap between rasterized images and vector glyphs, achieving higher quality vector generation results.

[0024] This invention uses a conditional super-resolution cascaded diffusion model to continuously upsample the high-quality low-resolution font results obtained in the first part to higher resolutions, i.e., from 64 to 256, and then from 256 to 1024, thereby generating high-resolution glyph images. This invention does not directly apply other existing image super-resolution models, but uses the same model as in the first part, namely the implicit diffusion model and the part-aware encoder, and makes three improvements to the super-resolution model: 1) This invention inserts the corresponding low-resolution image into the model by replacing the content reference image with the low-resolution image and concatenating it with the input noise; in this way, similar to the effect of the cross-attention module in the style feature conditional input on the low-resolution model, this invention guides the attention at each position of the low-resolution image to the relevant position in the style reference to recover lost details; 2) To perform the noise conditional enhancement operation, which is crucial for bridging the gap between the generated image and the training image, this invention encodes the low-resolution image into the latent space, adds Gaussian noise to it, and then decodes it back into the image space as the input of the enhanced low-resolution image; in addition, this invention adds an extra condition s to the U-Net part of the conditional implicit diffusion model with the part-aware encoder in the first part, namely the noise level added during the noise conditional enhancement process; 3) To address the increased memory consumption caused by high resolution, this invention adds an extra downsampling layer to the style and content encoders in the part-aware encoder. Since the style and content encoders already possess the knowledge to extract and transfer the required font styles, this invention uses the weights of the low-resolution generative model to initialize the super-resolution model.

[0025] The third part focuses on achieving high-speed font generation through single-step sampling based on 3D distillation technology.

[0026] Image diffusion models typically require dozens of sampling steps to produce effective, high-quality output. For the model obtained in the first part, sampling for 10 steps using a first-order SDE sampler can yield sufficiently good results, but 10 steps of inference still cannot achieve fast, real-time generation. This invention proposes a strategy based on the Score Distillation Sampling (SDS) method to distill the conditional implicit diffusion model with a part-aware encoder and the conditional super-resolution cascaded diffusion model into a one-step generative model.

[0027] Score Distillation Sampling (SDS) was originally proposed for text-to-3D generation tasks. It utilizes the knowledge of a pre-trained 2D text-to-image diffusion model to guide a 3D model in rendering realistic 2D images from random angles, thereby enabling the 3D model to learn. This distillation process can also be used to reduce the sampling steps of the model, that is, to replace the 3D model with a single-step sampling diffusion model, and generate more realistic images under the guidance of a multi-step diffusion model.

[0028] The distillation process includes a frozen teacher model. A student model generated in one step, the weights of which are initialized from the teacher model. In the student model, the noise intensity t is fixed at T to ensure that the model starts from pure noise during inference. During training, the student model will adjust the noise level based on the implicit representation to be denoised. Generate samples based on glyph content and style encoding features y Similar to how the SDS method processes 2D images rendered from 3D models, the goal is to optimize parameters. ,make It looks close to samples from the teacher model. To do this, for each noise intensity, the sample needs to be shifted towards the high probability density region implicitly defined by the teacher model, which is equivalent to optimizing the following objective:

[0029] ,

[0030] Among them, L SDS To optimize the objective; For expectations; and D is a series of predefined non-zero variables. KL Let KL divergence be a metric. For the forward process of the diffusion model (i.e.) Marginal distribution, Let w be the implicitly defined marginal distribution of the teacher model at noise intensity t, and w be a constant. Since estimating the noise added to the data samples is equivalent to estimating the score function of the noisy data distribution, the approximate gradient of the above optimization objective can be expressed as: ,in, For gradient operators; L SDS The optimization goal of SDS; For expectations; It is Gaussian noise.

[0031] The noise estimation ( Noise estimation without classifier guidance can be used. During the optimization process, only the weights of the diffusion model are updated, meaning the weights of the part-aware encoder are fixed during training.

[0032] Based on the diffusion model-based rapid generation system for few-sample high-resolution Chinese fonts constructed according to the present invention, the method for rapid generation of few-sample high-resolution Chinese fonts includes the following steps:

[0033] The first step is to build a style reference selection mapping;

[0034] In the specific implementation of the style reference selection mapping, based on the 1032 component classes and 339 stroke classes defined for Chinese characters, a set of characters that can cover all components and strokes contained in Chinese characters in the GB2312 standard is constructed as a style reference set, from which the style reference for model training is selected.

[0035] For any character c in the training set, establish a component-level character reference map and a stroke-level character reference map. The former consists of all characters in the style reference set that contain the components contained in c, and the latter consists of all characters in the style reference set that contain the strokes contained in c.

[0036] The second step is to select the style reference corresponding to each target image during training based on the style reference selection mapping, and train the implicit diffusion model to generate low-resolution glyph image results.

[0037] This step first constructs a conditional implicit diffusion model with a part-aware encoder, which consists of a style encoder. A content encoder It consists of a style encoder and a cross-attention module. The style encoder encodes each of the k style reference images into style features. The content encoder encodes the content reference image into content features. Each style feature is represented as a dimension of of Each token represents the content features as a dimension. of Each token is used, and cross-attention operations are performed.

[0038] We employ classifier-free sampling on the constructed conditional implicit diffusion model with a part-aware encoder and define a classifier-free method for conditional input of multiple style features. We design and train the implicit diffusion model using a loss function, obtaining the trained conditional implicit diffusion model with the part-aware encoder. High-quality low-resolution results are generated using the trained conditional implicit diffusion model with the part-aware encoder.

[0039] The third step is to train the conditional super-resolution cascaded diffusion model using the parameters of the trained low-resolution generative model (i.e., the conditional implicit diffusion model with a component-aware encoder) as the initial parameters.

[0040] Using a conditional super-resolution cascaded diffusion model, the low-resolution results are continuously upsampled to higher resolutions, from 64 to 256, and then from 256 to 1024, thereby generating high-resolution glyph images.

[0041] Using the model from the first step, namely the implicit diffusion model and the part-aware encoder, as initialization parameters, three improvements were made to the conditional implicit diffusion model with the part-aware encoder for super-resolution tasks:

[0042] 1) The corresponding low-resolution image is obtained by replacing the content reference image with the low-resolution image and then combining it with the input noise. The connection method is inserted into the U-Net part of the conditional implicit diffusion model with a component-aware encoder;

[0043] 2) Encode the low-resolution image into the latent space, add Gaussian noise to the implicit representation of the low-resolution image in the latent space, and then decode it back into the image space as the input of the enhanced low-resolution image.

[0044] In addition, an extra condition s is added to the U-Net part of the conditional implicit diffusion model with a component-aware encoder, namely the noise level added during the noise conditional enhancement process.

[0045] 3) An additional downsampling layer was added to the style and content encoders in the part-aware encoder to address the increased memory footprint caused by high resolution.

[0046] The fourth step involves using 3D distillation technology to shorten the conditional implicit diffusion model and conditional super-resolution cascaded diffusion model with component-aware encoders to a single sampling step.

[0047] In practice, diffusion model distillation is performed based on the Score Distillation Sampling (SDS) strategy.

[0048] Let x0 be the glyph target image, and z0 be the implicit representation of the glyph target image encoded into the latent space defined by the autoencoder in the conditional implicit diffusion model. and For a series of predefined non-zero variables. Taking the conditional implicit diffusion model of distillation with a component-aware encoder as an example (the process of the distillation conditional super-resolution cascaded diffusion model can be derived similarly), the distillation process includes a frozen teacher model. (i.e., a conditional implicit diffusion model with a component-aware encoder) and a one-step generated student model, where Let be the implicit representation to be denoised, t be the noise intensity, and y be the glyph content and style encoding features. The weights of the student model are initialized from those of the teacher model. In the student model, the noise intensity t is fixed at the maximum value T supported by the teacher model.

[0049] During the training of the student model, the student model is based on the implicit representation of the object to be denoised. Generate samples based on glyph content and style encoding features y To optimize model parameters , making the sample The sample appears to be similar to that from the teacher model. The following objectives are optimized:

[0050] ,

[0051] Among them, L SDS To optimize the objective; For expectations; and D is a series of predefined non-zero variables. KL Let KL divergence be a metric. For the forward process of the diffusion model (i.e.) Marginal distribution, Let w be the implicitly defined marginal distribution of the teacher model at noise intensity t, where w is a constant. The gradient of the optimization objective is approximated as:

[0052] ,

[0053] in, For gradient operators; L SDS The optimization goal of SDS; For expectations; It is Gaussian noise.

[0054] Noise estimation can be performed using a classifier-free version. During optimization, only the weights of the diffusion model are updated, i.e., the weights of the part-aware encoder are fixed during training.

[0055] By inputting a limited set of glyph samples for a new font, the above steps can be used to quickly generate a high-resolution Chinese font with few samples based on a diffusion model, thus generating a complete Chinese character library for that font.

[0056] Compared with the prior art, the beneficial technical effects of the present invention are as follows:

[0057] This invention provides a rapid method and system for generating high-resolution Chinese fonts with few samples based on a diffusion model. It utilizes an attention mechanism to reuse component-level style information and effectively transfers diverse and complex font styles to the target glyph based on diffusion model learning. Furthermore, this invention introduces a reference sample selection strategy, enabling the model to extract both global and local style information simultaneously. Secondly, this invention applies fractional distillation sampling techniques from text-to-3D generation literature to achieve highly efficient one-step generation with almost no quality loss, addressing the slow inference speed of the diffusion model and improving the speed and efficiency of font generation. Finally, this invention designs a super-resolution process, allowing the recovery of detail style and shape losses at low resolution, thereby narrowing the gap between image modalities and vector modalities and further achieving higher-resolution font generation. Attached Figure Description

[0058] Figure 1 is a flowchart of the font generation method provided by the present invention.

[0059] Figure 2 is a schematic diagram of the reference selection process of the present invention.

[0060] Figure 3 compares the generation results of the present invention with those of existing methods on characters that are not yet seen and characters that are already seen. n_ref is the size of the input style reference sample.

[0061] Figure 4 compares the generation results of this invention with existing methods on unseen fonts and characters. n_ref is the size of the input style reference sample.

[0062] Figure 5 shows the visualization results of the detail style restored by the super-resolution model in this invention.

[0063] Figure 6 shows the vector glyph results obtained by applying a vectorization tool to the high-resolution glyph image results in this invention.

[0064] Figure 7 shows a comparison of the generation results of the single-step model (after distillation) and its teacher model (before distillation) of the present invention on characters that are not seen and characters that are seen.

[0065] Figure 8 shows a comparison of the generation efficiency of the present invention with that of existing methods.

[0066] Figure 9 shows a comparison between the present invention and existing methods when applied to font color and texture style transfer.

[0067] Figure 10 shows the generation results of this invention applied to cross-language generation. Detailed Implementation

[0068] The present invention will be further described below with reference to the accompanying drawings and embodiments, but the scope of the invention is not limited in any way.

[0069] This invention provides a rapid Chinese font generation technology and system based on a diffusion model with few samples and high resolution. It consists of three parts, and the process and network structure are shown in Figure 1. The first part achieves high-quality generation based on a conditional diffusion model and a component-aware encoder (stage A in Figure 1). The second part achieves high-resolution font generation based on a cascaded diffusion model super-resolution technology (stage B1 in Figure 1). The third part achieves high-speed font generation by single-step sampling based on 3D distillation technology (stage B2 in Figure 1).

[0070] The specific implementation includes the following steps:

[0071] 1) Construct a style reference selection mapping, as shown in Figure 2.

[0072] To guide the style learning process at different levels of character forms, this invention provides the style encoder with pre-designed style references corresponding to the components and strokes contained in the content characters. EasyFont (Lian, Zhouhui, et al. "EasyFont: a style learning-based system to easily build your large-scale handwriting fonts." ACM Transactions on Graphics (TOG) 38.1 (2018)) defines m = 1032 component classes and n = 339 fine-grained stroke classes for Chinese characters, and constructs several character sets of different sizes covering different proportions of Chinese characters. We use the OptSet set defined in this paper, which contains 775 commonly used Chinese characters, covering all components and strokes contained in Chinese characters in the GB2312 standard; let this set be the style reference set, from which the style reference for model training is selected. For any character c in the training set, let c contain m c Components Let the i-th component contain Each stroke Since Chinese characters are typically broken down into a maximum of 6 components, the number of style references, k, is set to 6. We establish a component-level character reference map and a stroke-level character reference map, which map each component in character c. Allocate a sub-reference set For component-level character reference mapping, From all components in the style reference set The character composition, with stroke-level character reference mapping, Any style reference set contains The characters are composed of strokes. During the training process, this invention extracts from each... A character is randomly selected from the given set to form the style reference set for character c, and then the remaining k positions are randomly filled. This invention uses these two mappings with probabilities 1-p and p. During training, OptSet is used as the style reference set; during testing, a new mapping is constructed based on the given reference set.

[0073] 2) Select the style reference corresponding to each target image during training based on the style reference selection, train the implicit diffusion model to generate low-resolution glyph image results, and set p to 0.1.

[0074] 3) Train the super-resolution model using the parameters of the trained conditional implicit diffusion model with a component-aware encoder as initial parameters.

[0075] 4) Using the 3D distillation technique described above, the conditional implicit diffusion model with component-aware encoder and the conditional super-resolution cascaded diffusion model are shortened to a single-step sampling, while fixing the guiding coefficient of the classifier-guided sampling. and .

[0076] 5) Use the Image Tracer tool in Adobe Illustrator as a vectorization tool to convert high-resolution glyph images into high-quality vector glyphs.

[0077] During testing, that is, when the user uses the system, the user inputs a style reference image of the target font and the target generated character. The present invention sends the style reference and the target character rendered by the standard font (such as sans-serif) into the system. After passing through the low-resolution generation model, the super-resolution model and the vectorization tool, the vector glyph result is obtained.

[0078] Figure 3 compares the generation results of this invention with existing methods on unseen fonts and seen characters. `n_ref` represents the input style reference sample size. The samples generated by this invention have fine and sharp details while maintaining consistency with reference images of various styles. By using different input sample sizes, the model can extract different levels of style information from the input reference sample set. Figure 4 compares the generation results of this invention with existing methods on unseen fonts and unseen characters. `n_ref` represents the input style reference sample size. This invention can also generate high-quality results on characters not seen during training. Figure 5 shows the visualization results of the detailed style recovered by the super-resolution model in this invention. The model in this invention can extract and recover style details from high-resolution style references. Figure 6 shows the vector glyph results obtained by applying a vectorization tool (Adobe Illustrator) to the high-resolution glyph image results in this invention. Due to the sufficiently high quality of the generated high-resolution glyph image, this invention can obtain high-quality vector glyph results with precise control points. Figure 7 compares the generation results of the single-step model of this invention and its teacher model on unseen fonts and seen characters. The single-step student model obtained by this invention faithfully mimics the output of the multi-step teacher model, while the teacher model fails to generate valid results in the single-step generation setting, proving the effectiveness of the distillation method in this invention. Figure 8 compares the generation efficiency of this invention with existing methods. The single-step model obtained by this invention is significantly faster than other diffusion-based models and comparable to generative adversarial network-based methods. Figure 9 compares this invention with existing methods when applied to glyph color and texture style transfer. This invention performs better in terms of content matching and style consistency across all test styles. Figure 10 shows the generation results of this invention applied to cross-language generation, i.e., using a model trained on Chinese characters to transfer the style of Chinese characters to English and Korean characters.

[0079] The technical solutions of the embodiments of the present invention have been clearly and completely described above with reference to the accompanying drawings. It is understood that the described examples are merely a part of the embodiments of the present invention, and not all of them. All other embodiments obtained by those skilled in the art based on the embodiments of the present invention without creative effort are within the scope of protection of the present invention.

Claims

1. A method for rapid generation of Chinese fonts using a few samples based on a diffusion model, characterized in that, High-quality low-resolution font results are generated by constructing and training a conditional implicit diffusion model (i.e., a low-resolution generation model) with a component-aware encoder; a conditional super-resolution cascaded diffusion model (i.e., a super-resolution model) is then constructed based on cascaded diffusion model super-resolution technology to super-resolution the obtained high-quality low-resolution font results, thereby generating high-resolution vector glyph fonts; the process includes the following steps: The first step is to construct a style reference selection mapping; specifically, based on the components and strokes defined for Chinese characters, a set of characters covering the components and strokes contained in Chinese characters in the Chinese encoding standard is constructed as a style reference set, which serves as the model training set. For any character in the model training set, establish a component-level character reference map and a stroke-level character reference map; wherein, the component-level character reference map consists of all characters in the style reference set that contain the components contained in that character, and the stroke-level character reference map consists of all characters in the style reference set that contain the strokes contained in that character; The second step is to select the style reference corresponding to each target image during training based on the style reference selection mapping, construct a conditional implicit diffusion model with a component-aware encoder, i.e., a low-resolution generation model, and train it to generate low-resolution glyph image results. The component-aware encoder in the low-resolution generative model includes a style encoder, a content encoder, and a cross-attention module; the style encoder encodes all style reference images into style features, and the content encoder encodes content reference images into content features; the cross-attention module is used to represent each style feature and content feature as multiple tokens and then perform a cross-attention operation. The constructed low-resolution generative model is sampled using a classifier-free method with multiple style feature conditions as input. A loss function is designed and used to train the low-resolution generative model, resulting in a well-trained model. High-quality low-resolution font results are generated using this trained model. The sampling process using the classifier-free method on the low-resolution generative model is represented as follows: ， in, The result is the noise estimation result without classifier guidance; and These are the correlation strength coefficients between the generated results and the input content and style conditions, respectively. For reference images of character content, A style reference image derived from the target font; To be The result of replacing it with a vector of all zeros of the same size, To be The result of replacing it with a vector of all zeros of the same size; 3D distillation technology is used to distill the low-resolution generative model and the super-resolution model into a diffusion model, shortening the sampling time to a single step to obtain a one-step generative model. The third step is to use the parameters of the trained low-resolution generation model as initial parameters to train the conditional super-resolution cascade diffusion model, i.e., the super-resolution model; using the trained super-resolution model, the low-resolution font results are continuously upsampled to a higher resolution, thereby generating high-resolution font images. By following the steps above, rapid generation of Chinese fonts based on a diffusion model with few samples can be achieved.

2. The method for rapid generation of few-sample Chinese fonts based on a diffusion model as described in claim 1, characterized in that, In the low-resolution generation model, the style encoder in the part-aware encoder is used to encode all k style reference images into style features, represented as follows: ; The content encoder in the component-aware encoder is used to encode the content reference image into content features, represented as follows: ； Each style feature is represented as a token with multiple dimensions, and content features are represented as tokens with multiple dimensions. A cross-attention operation is then performed, resulting in the following representation: ， Where y represents the glyph content and style encoding features; Attention represents the attention mechanism; Q represents the query matrix; K represents the key matrix; and V represents the value matrix. are the projection matrices corresponding to the i-th attention mechanism; softmax is the probability distribution mapping function; To flatten the content features to obtain the features, The features obtained by connecting and flattening stylistic features.

3. The method for rapid generation of few-sample Chinese fonts based on a diffusion model as described in claim 2, characterized in that, In the second step, the loss function used to train the low-resolution generative model is... Represented as: ,in, Image of the character shape; It is Gaussian noise. For noise intensity, This represents a conditional implicit diffusion model; To be Add noise The corresponding implicit expression; The implicit representation corresponding to the latent space defined by the autoencoder in the conditional implicit diffusion model for encoding glyph target images; Represents expectations; This represents the L2 distance.

4. The method for rapid generation of few-sample Chinese fonts based on a diffusion model as described in claim 1, characterized in that, In the third step, the super-resolution model is trained using the low-resolution generative model, including the following improvements: 1) By using the corresponding low-resolution image as the content reference image and concatenating it with the input noise, it is inserted into the convolutional deep learning model U-Net in the low-resolution generative model; 2) Encode the low-resolution image into the latent space, add Gaussian noise to the implicit representation of the low-resolution image in the latent space, and then decode it back into the image space as the input of the enhanced low-resolution image. 3) Add an additional downsampling layer to the style and content encoders in the part-aware encoder to address the increased memory footprint caused by high resolution.

5. The method for rapid generation of few-sample Chinese fonts based on a diffusion model as described in claim 4, characterized in that, In Improvement 2), additional conditions are added to the convolutional deep learning model U-Net in the low-resolution generative model, namely, adding noise levels during the noise condition enhancement process.

6. The method for rapid generation of few-sample Chinese fonts based on a diffusion model as described in claim 1, characterized in that, The 3D distillation technology specifically employs a strategy based on the Score Distillation Sampling Method (SDS), including: The distillation process includes a frozen teacher model. A student model generated in one step, the weights of which are initialized from the teacher model. Noise intensity in the student model Fixed as T; During training, the student model is based on the implicit representation of the data to be denoised. Generate samples based on glyph content and style encoding features y ; Optimize parameters ,make Same samples as those from the teacher model; optimize objective function L SDS Represented as: ,in, The marginal distribution of the forward process in the diffusion model; The noise intensity is The marginal distribution implicitly defined in the teacher model; It is a constant value; and For predefined non-zero variables; Let KL divergence be a metric. Gradient of the optimization objective Represented as: Noise estimation Noise estimation without classifier guidance is adopted; during the optimization process, only the weights of the diffusion model are updated, that is, the weights of the component-aware encoder are fixed during the training process.

7. A rapid few-sample Chinese font generation system based on a diffusion model, used to implement the method of claim 1, characterized in that, It includes a low-resolution generation model module, a super-resolution model module, and a diffusion model distillation module; among which, The low-resolution generation model module is used to generate low-resolution glyph image results; the super-resolution model module is used to continuously upsample the low-resolution font results to a higher resolution, thereby generating high-resolution vector glyph images. The diffusion model distillation module is used to perform diffusion model distillation on the low-resolution generative model and the super-resolution model to obtain a one-step generative model.

Images