Vision-assisted millimeter wave beam prediction method for low-light environments

Abstract

The present invention relates to a vision-assisted millimeter wave beam prediction method for low-light environments, belonging to the field of wireless communications. Targeting millimeter wave communication systems, the method collects image data of a communication environment under low-light conditions by means of a camera installed at a base station to construct a millimeter wave beam prediction system model, and uses a deep neural network model and a course training policy to learn the collected image data so as to predict an optimal communication beam. The present invention can improve the performance of models in low-light environments and accelerate the convergence speed of network models, thereby improving the accuracy of beam prediction and enhancing the robustness and reliability of millimeter wave communication systems in low-light environments.

Description

A Visual-Assisted Millimeter-Wave Beam Prediction Method for Low-Light Environments Technical Field

[0001] This invention belongs to the field of wireless communication and relates to a visual-assisted millimeter-wave beam prediction method for low-light environments. Background Technology

[0002] Millimeter waves possess characteristics such as high bandwidth and low latency, enabling efficient data transmission and finding wide application in 5G and 6G communication systems. However, millimeter waves are prone to short transmission distances and are easily obstructed, which can significantly reduce the reliability and stability of millimeter wave communication. To ensure the reliability of millimeter wave communication, large-scale antenna arrays are typically deployed to form narrow beams for long-distance wireless communication. However, such large-scale antenna arrays incur enormous beam training overhead and resource waste. To address this issue, sensing data-assisted wireless communication schemes have gained significant attention. These schemes utilize artificial intelligence technology to learn the patterns of changes in sensing data within the communication environment and predict the optimal communication beam.

[0003] Video and image data contain rich information closely related to wireless communication, such as user location, and are therefore widely used in millimeter-wave beam prediction. However, in low-light environments, the image information captured by cameras is significantly reduced, leading to a substantial decrease in the accuracy of vision-based beam prediction. For example, in industrial manufacturing scenarios, limited ambient lighting conditions significantly increase the difficulty of vision-based millimeter-wave prediction, resulting in unstable data transmission. Therefore, improving the accuracy of beam prediction in low-light environments has become a crucial problem that urgently needs to be solved.

[0004] Summary of the Invention

[0005] In view of this, the purpose of this invention is to provide a visual-assisted millimeter-wave beam prediction method for low-light environments. For the problem of visual-assisted millimeter-wave beam prediction in low-light scenarios, a course training method is used to ensure high-accuracy beam prediction in low-light communication environments.

[0006] To achieve the above objectives, the present invention provides the following technical solution:

[0007] A vision-assisted millimeter-wave beam prediction method for low-light environments is proposed. Addressing the increased difficulty of beam prediction due to reduced wireless communication information in image data under low-light communication conditions, a course-based training-based beam prediction method is suggested. Based on the difficulty level of the training samples, difficult (low-light) samples are added to the training set in stages, allowing the prediction model to learn from easy to difficult, thereby improving beam prediction accuracy in low-light environments.

[0008] The method specifically includes the following steps:

[0009] S1: Construct a millimeter-wave wireless communication system, extract features from image data under low-light conditions using a neural network model, optimize beamforming vectors, and select the optimal communication beam from the beam codebook to maximize received signal power and reduce the complexity of beam prediction;

[0010] S2: Collect image data of millimeter-wave wireless communication scenarios and the corresponding optimal beamforming dataset in the codebook, preprocess the collected image data, construct a millimeter-wave beam prediction deep neural network model based on the collected dataset, and define the model loss function and model optimizer to train the model.

[0011] The preprocessing of the acquired image data specifically involves: dividing the acquired image dataset into a low-light image subset and a normal-light image subset; adding the low-light image dataset to the normal-light image training set in stages based on the course training method; using the normal-light image dataset as the original training set in the first stage of training; and uniformly adding low-light images to the training set in subsequent stages; and finally predicting the optimal beam in the entire dataset.

[0012] S3: Input the entire dataset into the trained millimeter-wave beam prediction deep neural network model to predict the optimal beam.

[0013] Furthermore, step S1 specifically includes the following steps:

[0014] S11: Construct a millimeter-wave wireless communication system, including a mobile user equipment equipped with an omnidirectional antenna to receive downlink signals from a base station; and a fixed base station equipped with a camera for capturing real-time environmental images; the base station includes a uniform linear antenna array consisting of M antenna elements, and is beamformed by a beamforming vector w. q Transmit a signal to a mobile user equipment, where w q ∈￡ M×1 Represents the beamforming vector in the codebook, codebook Q is the number of beam vectors in the codebook; the system uses Orthogonal Frequency Division Multiplexing (OFDM) technology to transmit signals through K subcarriers, and the communication channel of each subcarrier is represented as h. k Where k = 1, 2, ..., K;

[0015] S12: Define the beam prediction optimization problem and predict the optimal beamforming vector using a neural network model. To maximize the received signal power on a given subcarrier, the optimization objective formula is:

[0016] The beam prediction objective can be defined as selecting the optimal communication beam from the beam codebook Q to connect and communicate with the user; if This represents an image of a communication scene captured by a camera at the base station. Let the optimal beam predicted by the neural network be represented, then the beam prediction function can be expressed as:

[0017] Where, θ w The parameters represent the optimization parameters of the network model. Since there is a one-to-one correspondence between the beam vector and its index in the codebook, the prediction difficulty can be further reduced by directly predicting the beam index. Therefore, the original formula can be expressed as:

[0018] in, This represents the index of the predicted optimal beam vector.

[0019] Furthermore, in step S2, a deep neural network model for millimeter-wave beam prediction is constructed and trained, specifically including the following steps:

[0020] S21: A base station equipped with a camera captures real-time environmental images in RGB format, containing multiple feature information of the communication environment. The base station uses the captured image information to select from a preset beamcodebook. Select the optimal beamforming vector The beamcodebook contains multiple beamforming vectors w q ∈￡ M×1 To cover the entire communication scenario; the collected dataset includes normal light and low light datasets;

[0021] S22: Construct a millimeter-wave beam prediction neural network model. Use the pre-trained neural network model ResNet50 to perform the beam prediction task. Input the acquired images into the ResNet network model. The ResNet network model processes the image data and extracts deep features of the image through multi-layer convolution and pooling operations.

[0022] S23: After image feature extraction, the model uses a fully connected layer to map the features to the beamforming vector space, and calculates the probability distribution of each beamforming vector using the Softmax function. The model is trained using the cross-entropy loss function, and the optimization objective is to maximize the probability of correctly predicting the beamforming vector. The loss function is expressed as:

[0023] Among them, L cross-entropy Let P(s) be the loss function, n be the number of training samples, and P(s) be the training sample number. n |X n ) is the model in a given image X n In the case of predicting beam vector index s nTo further improve the model's generalization ability, the Adam optimizer is used to dynamically adjust the model parameters, ensuring that the model converges quickly during training.

[0024] S24: Select the beamforming vector with the highest probability as the output result to ensure the best communication quality in the current communication environment and guarantee the effectiveness and reliability of the connection.

[0025] Furthermore, in step S2, the preprocessing of the acquired image data specifically includes the following steps:

[0026] S201: Divide the training dataset into easy samples and hard samples, where easy samples consist of images taken under normal lighting conditions, and hard samples consist of images taken under low light conditions, using D... normal D represents the segmented normal light image dataset. low This represents the partitioned low-light dataset; the sample distributions of these two subsets are P and P, respectively. normal P low ;

[0027] S202: The course training method is adopted, in which the training process starts with image data under normal lighting conditions and gradually introduces images taken in low light environment to improve the beam prediction accuracy of the neural network model in low light environment. By adding low light data to the training set in stages, the network model's ability to learn complex low light samples is increased.

[0028] Specifically, the training process is divided into V stages. In each stage, a low-light dataset that is more difficult to learn is added to the training set. Let λ be the training set. i Let D represent the probability of random sampling added to the training set from low-light samples in the i-th stage; i Let P represent the training set for the i-th stage. i Let D represent the distribution of training set samples in the i-th stage. i and P i It can be represented as: D i =D normal ∪sample(D low ,λ i ) P i ∝P normal +λ i ·P low

[0029] Among them, sample(D) low ,λ i ) represents the random sampled dataset D low λ of the total iIn the new dataset that is formed, the training set in the final stage will contain all the normal light and low light datasets.

[0030] Let λ1 = 0 and λ2,..., This indicates that in the first stage of training, all normal lighting images are used as the initial training set to train the model, and in subsequent stages, low-light samples are uniformly added to the training set. A new training set is formed by adding elements to the training set; finally, the optimal beam is predicted on the entire dataset.

[0031] The beneficial effects of this invention are as follows:

[0032] (1) This invention takes into account the problem of reduced accuracy of millimeter wave beam prediction in low light environment, optimizes the prediction in low light environment, and ensures the stability and reliability of 5G millimeter wave communication.

[0033] (2) The present invention adopts the course training method to improve beam prediction performance. By utilizing the phased learning method of course training, the model can learn more general features from simple samples, which effectively improves the accuracy of model prediction and enables industrial 5G communication to maintain stable and reliable implementation.

[0034] Other advantages, objectives, and features of the invention will be set forth in part in the description which follows, and in part will be apparent to those skilled in the art from the following examination, or may be learned from practice of the invention. The objectives and other advantages of the invention can be realized and obtained through the following description. Attached Figure Description

[0035] To make the objectives, technical solutions, and advantages of the present invention clearer, the preferred embodiments of the present invention will be described in detail below with reference to the accompanying drawings, wherein:

[0036] Figure 1 is a model of the vision-assisted millimeter-wave wireless communication system provided by the present invention;

[0037] Figure 2 is a schematic diagram of deep learning training based on course training according to the present invention;

[0038] Figure 3 is a flowchart of the vision-assisted millimeter-wave beam prediction method based on course training according to the present invention. Detailed Implementation

[0039] The following specific examples illustrate the implementation of the present invention. Those skilled in the art can easily understand other advantages and effects of the present invention from the content disclosed in this specification. The present invention can also be implemented or applied through other different specific embodiments, and various details in this specification can be modified or changed based on different viewpoints and applications without departing from the spirit of the present invention. It should be noted that the illustrations provided in the following embodiments are only schematic representations of the basic concept of the present invention. Unless otherwise specified, the following embodiments and features can be combined with each other.

[0040] Please refer to Figures 1-3. This invention provides a vision-assisted millimeter-wave beam prediction method for low-light environments. For vision-assisted millimeter-wave communication systems, a camera installed at the base station captures real-time low-light images of the communication environment to predict which communication beam from the codebook the base station should select to communicate with the user. A ResNet50 network model is used for beam prediction, and a course-based learning training method is employed to accelerate training speed and improve prediction accuracy, thus inventing a deep learning-based beam prediction scheme.

[0041] Figure 1 shows a model of a vision-assisted millimeter-wave communication system. It includes a fixed base station, a mobile user, and other mobile scattering objects. The base station is equipped with a camera and an antenna array, which has M antennas. The base station captures and collects real-time images of the communication environment, forming a raw dataset. In this system, the base station uses a predefined beam steering codebook. Connect and communicate with users.

[0042] Figure 2 illustrates the deep neural network prediction framework based on course training. The entire training process is divided into V stages, each containing v training rounds. Normal light images are used as the initial training set, and low-light datasets are gradually and randomly added to the training set in subsequent stages. For the beam prediction task, a deep residual network, ResNet50, is used for prediction. In the final classification output layer, the predicted output category is fine-tuned to the number of beam codebooks. When image data is input into the model, the convolutional layers and residual blocks in ResNet50 extract depth features from the image, which are then passed through fully connected layers to generate the predicted output.

[0043] Figure 3 is a flowchart of the vision-assisted millimeter-wave beam prediction method based on course training according to the present invention, which specifically includes the following steps:

[0044] V1~V4: Obtain relevant datasets for beam prediction. Divide the collected datasets into two subsets: low-light image dataset and normal-light image dataset. Use these subsets to initialize model hyperparameters, the number of training stages, and the number of training rounds per stage for course training.

[0045] V4~V10: Build the ResNet50 network model for beam training and define the loss function and optimizer in the network model. Use normal light images as the original training set for the first stage of training. In the subsequent training stages, gradually and evenly add low light image datasets for course training.

[0046] V10~V15: Network training and prediction process. During the training process, the course is implemented in stages. The input image trains the entire network, the loss function is calculated and the network parameters are updated by backpropagation. After the training termination condition is reached, the optimal predicted beam is finally output.

[0047] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention and are not intended to limit it. Although the present invention has been described in detail with reference to preferred embodiments, those skilled in the art should understand that modifications or equivalent substitutions can be made to the technical solutions of the present invention without departing from the spirit and scope of the present invention, and all such modifications or substitutions should be covered within the scope of the claims of the present invention.

Claims

1. A visual-assisted millimeter-wave beam prediction method for low-light environments, characterized in that, The method specifically includes the following steps: S1: Construct a millimeter-wave wireless communication system, extract features from image data under low-light conditions using a neural network model, optimize beamforming vectors, and select the optimal communication beam from the beam codebook to maximize the received signal power; S2: Collect image data of millimeter-wave wireless communication scenarios and the corresponding optimal beamforming dataset in the codebook, preprocess the collected image data, construct a millimeter-wave beam prediction deep neural network model based on the collected dataset, and define the model loss function and model optimizer to train the model. The preprocessing of the acquired image data specifically involves: dividing the acquired image dataset into a low-light image subset and a normal-light image subset; adding the low-light image dataset to the normal-light image training set in stages based on the course training method; using the normal-light image dataset as the original training set in the first stage of training; and uniformly adding low-light images to the training set in subsequent stages; and finally predicting the optimal beam in the entire dataset. S3: Input the entire dataset into the trained millimeter-wave beam prediction deep neural network model to predict the optimal beam.

2. The visual-assisted millimeter-wave beam prediction method according to claim 1, characterized in that, Step S1 specifically includes the following steps: S11: Construct a millimeter-wave wireless communication system, including a mobile user equipment equipped with an omnidirectional antenna to receive downlink signals from a base station; and a fixed base station equipped with a camera for capturing real-time environmental images; the base station includes a uniform linear antenna array consisting of M antenna elements, and is beamformed by a beamforming vector w. q Transmit a signal to a mobile user equipment, where w q ∈￡ M×1 Represents the beamforming vector in the codebook, codebook Q is the number of beam vectors in the codebook; the system uses orthogonal frequency division multiplexing (OFDM) technology to transmit signals through K subcarriers, and the communication channel for each subcarrier is represented as h. k Where k = 1, 2, ..., K; S12: Define the beam prediction optimization problem and predict the optimal beamforming vector using a neural network model. To maximize the received signal power on a given subcarrier, the optimization objective formula is: The beam prediction objective is defined as selecting the optimal communication beam from the beam codebook Q to connect and communicate with the user; if This represents an image of a communication scene captured by a camera at the base station. If the optimal beam predicted by the neural network is represented, then the beam prediction function is expressed as: Where, θ w The parameters represent the optimization parameters of the network model. Since there is a one-to-one correspondence between the beam vector and its index in the codebook, the prediction difficulty can be further reduced by directly predicting the beam index. Therefore, the original formula can be expressed as: in, This represents the index of the predicted optimal beam vector.

3. The visual-assisted millimeter-wave beam prediction method according to claim 2, characterized in that, In step S2, a deep neural network model for millimeter-wave beam prediction is constructed and trained, specifically including the following steps: S21: A base station equipped with a camera captures real-time environmental images in RGB format, containing multiple feature information of the communication environment. The base station uses the captured image information to select from a preset beamcodebook. Select the optimal beamforming vector The beamcodebook contains multiple beamforming vectors w q ∈￡ M×1 To cover the entire communication scenario; the collected dataset includes normal light and low light datasets; S22: Construct a millimeter-wave beam prediction neural network model. Use the pre-trained neural network model ResNet50 to perform the beam prediction task. Input the acquired images into the ResNet network model. The ResNet network model processes the image data and extracts deep features of the image through multi-layer convolution and pooling operations. S23: After image feature extraction, the model uses a fully connected layer to map the features to the beamforming vector space, and calculates the probability distribution of each beamforming vector using the Softmax function. The model is trained using the cross-entropy loss function, and the optimization objective is to maximize the probability of correctly predicting the beamforming vector. The loss function is expressed as: Among them, L cross-entropy Let P(s) be the loss function, n be the number of training samples, and P(s) be the training sample number. n |X n ) is the model in a given image X n In the case of predicting beam vector index s n To further improve the model's generalization ability, the Adam optimizer is used to dynamically adjust the model parameters, ensuring that the model converges quickly during training. S24: Select the beamforming vector with the highest probability as the output result.

4. The visual-assisted millimeter-wave beam prediction method according to claim 2, characterized in that, In step S2, the acquired image data is preprocessed, specifically including the following steps: S201: Divide the training dataset into easy samples and hard samples, where easy samples consist of images taken under normal lighting conditions, and hard samples consist of images taken under low light conditions, using D... normal D represents the segmented normal light image dataset. low This represents the partitioned low-light dataset; the sample distributions of the two subsets are P and P, respectively. normal P low ; S202: The course training method is adopted, in which the training process starts with image data under normal lighting conditions and gradually introduces images taken in low light environment to improve the beam prediction accuracy of the neural network model in low light environment. By adding low light data to the training set in stages, the network model's ability to learn complex low light samples is increased. Specifically, the training process is divided into V stages. In each stage, a low-light dataset that is more difficult to learn is added to the training set. Let λ be the training set. i Let D represent the probability of random sampling added to the training set from low-light samples in the i-th stage; i Let P represent the training set for the i-th stage. i Let D represent the distribution of training set samples in the i-th stage. i and P i Represented as: D i =D normal ∪sample(D low ,λ i ) P i ∝P normal +λ i ·P low Among them, sample(D) low ,λ i ) represents the random sampled dataset D low λ of the total i In the new dataset that is formed, the training set in the final stage will contain all the normal light and low light datasets. Let λ1 = 0 and This indicates that in the first stage of training, all normal lighting images are used as the initial training set to train the model, and in subsequent stages, low-light samples are uniformly added to the training set. A new training set is formed by adding elements to the training set; finally, the optimal beam is predicted on the entire dataset.

Images